So Outside Technology Security CMS
AnQiCMS
Online Demonstration

XML Sitemap and robots.txt配合：2026年抓取预算优化的正确方法

📅 October 26, 2026 👁️ 1

How to use XML Sitemap and robots.txt to optimize crawling budget, the answer is to let the two files handle different segments: the sitemap is responsible for providing the list of addresses worth crawling, and robots.txt is responsible for declaring paths that are not worth crawling. The crawling budget is the amount of crawling that the site can allocate in a unit of time, which is affected by the update rhythm of the site and the server response, so the real operable space is in 'Don't spend the budget on invalid addresses', rather than adding weight numbers to the page.

First, clarify the division of labor between the two.

A sitemap is an enumerable list that tells the crawler 'which pages are here and when they might change'. The robots.txt is a statement of access, telling the crawler 'these paths should not be requested'. The two have different ways of becoming invalid: if a category is missing in the sitemap, those pages can only be passively discovered through internal links; if one extra exclusion is written in robots.txt, it may block the directory that you want to include, while the addresses in the sitemap are still listed, forming a contradiction of 'listed but cannot enter'.

The sitemap address should be written in the robots.txt

It is a common practice to write the location of the sitemap into the robots.txt: Use a line of Sitemap declaration in the file to indicate the map address, so that the crawler can discover the entry along with reading the access declaration, without relying solely on backend submission. The value of this step lies in supplementing a discovery path, especially for new websites or after a domain change, as the probability of passive discovery is already low.

Note a few points: The declared address must be a real accessible final address, avoiding redirection to old domain names; multiple maps (articles, images, categories) can be declared line by line; the robots.txt itself must be directly accessible, and returning an error page is equivalent to invalid declaration.

The three most common wastes in crawling budget

The first category is the repeated crawling of valueless paths: in-site search result pages, lists with filtering parameters, login and backend prefixes. These addresses are numerous but should not be indexed. The correct practice is to exclude them in robots.txt and not put them in the site map.

The second type is when addresses that should not be included mix into the list: paths excluded, draft pages, or historical archives are written into the sitemap. The crawler attempts to access them and is blocked, which is equivalent to wasting the budget on invalid requests. The sitemap should only contain the addresses of formal documents, and content in draft or pending publication status should not appear.

The third type is the change not submitted in time: after the column is revised and the URL structure is adjusted, the old address is neither used for 301 redirect convergence nor updated on the map, causing the crawler to repeatedly traverse the historical structure. Especially easy to miss this step after the pseudo-static rule adjustment, the rule has changed but the map is still the old path, and the two sides do not match.

The recommended landing sequence

Do it in the order of 'first reduce invalid, then supplement valid'. The first step is to sort out the exclusion list, write the search page, parameter page, and background prefix into robots.txt; the second step is to regenerate the site map, confirm that it only contains formal content, and declare the sitemap address in robots.txt; the third step is to actively push the recently added and revised addresses to search engines using link push to shorten the waiting time for discovery; the fourth step is to observe the differences between inclusion and crawling, see which addresses have always been in the list but not caught, and check the server response and internal link depth back.

This action does not need to be done every day, but it needs to be run every time the column structure, URL form, or pagination logic changes, especially for sites that automatically generate site maps, changing the rules once is equivalent to rewriting the entire list.

Common questions

robots.txt has been excluded, does it matter if the address is still in the site map?It matters. If there are entries in the list that are blocked by requests, it will cause the crawling party to waste resources on invalid addresses, and it will also make the inclusion status difficult to explain. Both sides should check with the same standard.

Can the site set its own crawling budget?Cannot be set directly. What can be changed is the exclusion of invalid paths, the cleanliness of the list, and the redirection and promotion of old addresses after revision, which determine how many effective收录 can be obtained with the same crawling volume.

Is it better to have a larger sitemap?No. The value of the map lies in covering the formal content to be included, and putting in archive pages, filter pages, and addresses without inclusion value will only dilute the effective crawling.

Related articles

Large-scale CMS Vulnerability Exploitation Activity: Security Alerts that Website Owners Need to Know in 2026

In July 2026, the Australian Cyber Security Centre reported a large-scale exploitation activity targeting website content management systems, including platforms such as WordPress, Joomla, and Craft CMS. Attackers exploited known vulnerabilities to upload files, remotely execute code, and leave behind Webshells to maintain access. This article summarizes the three characteristics of such activities and a set of emergency response procedures, and explains which built-in capabilities in self-built systems can be directly used to reduce the attack surface.

2026-10-26

The new role of robots.txt in 2026: from search engine to control panel for AI spiders

robots.txt is no longer just a crawling instruction for search engines; it has become a layer of access control for the site to declare which robots can read and the range of reading. This article explains the change that mainstream AI spiders will read robots.txt before crawling, sorts out the differences in case matching and path matching in the protocol, and the order of judgment when there is a conflict in the rules, and gives the on-site configuration methods in cooperation with llms.txt and site maps.

2026-10-26

Among the more than 10,000 new vulnerabilities in a year, only six are core, and 90% are plugins. How should this statistic be used?

The correct usage of ecological vulnerability statistics is to determine the layer of the attack surface, not to evaluate the core quality. The security report shows that in 2025, more than 11,000 new vulnerabilities were added to the WordPress ecosystem, 90% of which came from plugins, 9% from themes, and only six from the core, with a median time to exploitation of only 5 hours. It points to a conclusion: whether the defense line should be inside the system or outside, determines the level of maintenance burden.

2026-10-26

How to determine the minimum memory requirement when choosing a self-hosted CMS

The memory threshold for self-hosted CMS should be based on the runtime stack, not the lowest configuration of the installation page. In the public comparison criteria, Directus and Payload report above 512MB, while Strapi and Ghost recommend above 1GB. The official criteria of AnQiCMS in the GoLang technology stack is that the memory usage is reduced by about 80% compared to PHP-based CMS, and the single machine can carry about 5 million PV. The bottom line should be determined based on the peak concurrent volume.

2026-10-26

Winter CMS SQL injection vulnerability CVE-2026-32593 analysis and protection enlightenment

CVE-2026-32593 affects the Filter component of Winter CMS's backend, attackers can interfere with the value by using the numberrange filtering type, the fixed version is 1.2.13, all previous versions including 1.2.12 are affected. This article analyzes this announcement in terms of attack surface, triggering conditions, and repair scope, and summarizes the three insights it provides for content management system users: that the backend is not a security boundary, that sorting and filtering parameters should be treated as input, and why parameterized queries can reduce such risks.

2026-10-26

GEO market growth data: The impact of AI search traffic on content sites in 2026

Looking at several sets of public data from 2026: The U.S. generative engine optimization market size is expected to be about 3.654 billion U.S. dollars, with a compound annual growth rate of 42.9%, the monthly visits to generative AI platforms have increased by 70% year-on-year, AI overview has appeared in most commercial intent searches, and the zero-click ratio of searches continues to rise. This article explains the three-level implications of these data for content sites and provides executable response orders.

2026-10-26

Webflow vs Wix vs Squarespace: Cross-functional comparison of no-code website building tools in 2026

Compare Webflow, Wix, and Squarespace as hosting website platforms in terms of public pricing and positioning in 2026: entry-level prices are approximately $14, $17, and $16 per month, respectively; the main differences are in editing methods and constraints; also, explain the different trade-offs of the self-built solution in terms of data ownership, long-term costs, and operational responsibilities, helping to choose based on team capabilities rather than feature lists.

2026-10-26

Google Search's Generative AI Optimization Guide: Official Practices in 2026 **

Google released an optimization guide for search generative AI features in May 2026, with the core suggestion being to focus on content prioritizing human needs and basic crawling and indexing conditions, and explicitly not to introduce any special practices for AI features. This article summarizes the three main lines of the guide, clarifies the position of newly added files like llms.txt, and provides technical and content inspection items that can be implemented on the site.

2026-10-26