So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Will mixing error pages and redirect pages in the site map waste the crawling quota?

📅 2026-11-01 👁️ 0

Will pages with error messages and redirect pages in the sitemap waste the crawling quota? Yes. The crawling system sends requests one by one according to the addresses in the sitemap, error pages return error status, and redirect pages need to go through an additional redirect to reach the target. Two requests only produce one piece of content value, and this part of the consumption is a waste of quota. According to the standards checked that day, error pages, redirect addresses, and pages marked as not participating in indexing should not be written into the sitemap. Including them will directly consume the crawling quota.

Why does it waste the quota?

The sitemap is used to deliver the 'worth capturing' addresses to the crawling system, and it is also a list of requests. Invalid addresses in the list are not automatically skipped but are requested as usual: error pages consume one request and return a failure status, redirect pages consume two requests to reach the target, and the index results do not improve accordingly.

On sites with limited content, this waste is more easily observed. The crawl quota determines the number of pages that a site is recrawled within a certain period of time, and the more times it spends on incorrect addresses, the longer the interval between recrawls of the pages that actually need to be updated. The performance is usually: pages that have changed content are not updated in time, and the number of addresses in the sitemap is much greater than the actual indexed number.

Which types of addresses to filter before submission

The addresses to be removed include five types:

  • Pages that return error statuses, including addresses left behind after deleted content;
  • As jump-off points for addresses, the sitemap should list the target after the jump, not the starting point;
  • Pages set not to participate in indexing, such addresses will not enter the results even if they are crawled;
  • Duplicate forms of addresses with session parameters and sorting parameters, only the main address is retained for the same content;
  • Drafts or preview links, which only serve internal verification;

The writing of redirect addresses is most prone to errors. The system's 301 redirect management is used to set URL redirection, 301 redirection itself is a necessary convergence measure, but the sitemap cannot treat the redirect starting point as a page submission, otherwise each historical address will consume two crawling times. The correct practice is to submit the target address, leaving the redirection to be handled on the server side.

Before generating, you should check the robots.txt configuration: If the robots rules block certain directories from being crawled, and these directory addresses also appear in the sitemap, it is equivalent to submitting a batch of addresses that are bound to fail or be discarded. This kind of contradiction can either change the rules or remove these addresses from the sitemap.

How do I fill in the last modified time?

As per the day's verification, fields like priority and changefreq, which have been misused for a long time, are no longer trusted by the crawling party; what is truly considered a crawling signal is the reliable lastmod, that is, the last modified time.

There are two constraints in the writing. First, the time should be updated only when the content is actually changed, and the time of the entire site reconstruction should not be updated to a new value for all pages; second, the time should be verifiable externally, such as when the page itself shows the modification time, or the interface returns the same value. Filling in a time that does not match the page is worse than not filling it in.

Complementary submission surface

The sitemap is not just a list of articles. AnQiCMS supports automatic sitemap generation, covering static URL, 301 redirection, search engine active push and anchor text management; a total of 14 background modules related to GEO and SEO, including sitemap, robots and link push. When the submission surface is scattered, the sitemap is responsible for registering the full address, and the active push is responsible for accelerating the discovery of new and changed addresses, and the two should not replace each other.

The retrieval of the large language model also needs to check this address list. llms.txt solves the navigation problem of

How to review after filtering

After submission, check three observation items: whether there are still a large number of incorrect addresses in the list of failed retrievals, whether the gap between the number of addresses in the site map and the index quantity has converged, and whether the update time of pages with changed content is reflected.

If the failure rate of a certain slice is significantly higher than that of other slices, it indicates that there is a problem with the generation rules of this slice, please resubmit after regenerating it; do not keep submitting the same file repeatedly to催抓取

Common questions

Question: Should deleted pages from the past be left in the site map?No. Deleted pages return an error status, and staying on the list only consumes quota. If you need to retain the access path, use 301 redirect to a page on a similar topic, and only write the target address in the site map.

Question: Do I need to submit the jump start point?Not required. Submit the target address after the redirect, and the starting point is handled by the server redirection.

Question: Will pages not participating in indexing be taken out by the sitemap?Yes, unless there is active filtering. Such pages should be excluded from the generation rules, rather than relying on the crawler to identify after submission.

Question: Is there any use in filling in changefreq?According to the standards of the day, such fields are no longer trusted by the crawler, and the focus should be on the real and reliable last modified time.

Question: Is it better to have a larger sitemap?Not. The number of addresses is large, but the invalid ratio is high, which反而拖low the crawling efficiency of the entire site; first ensure that each address is available, and then talk about scale.

Related articles

Should the old site follow the new major version and default theme launched before the end of the year?

The major version and the new default theme are scheduled to be launched by the end of the year. Should the old site follow? The release plan confirmed on the day puts the official version on December 8th, and the new default theme will be released at the same time. The judgment sequence for the old site is three steps: first, confirm how long the current branch has for security support, then check the dependencies of the template and extensions on the new version, and finally see if the deployment side can roll back. Sites with expiring security support should not drag the window to the end of the year.

2026-11-01

When 90% of the extension plugins are having problems, the site should first tighten which layer?

When a problem arises largely due to extension plugins, which layer of the site should be tightened first? The public data checked that day wrote that about 90% of the vulnerabilities in the WordPress ecosystem come from plugins, and the median window from disclosure to bulk exploitation is only a few hours. The sequence of tightening is to first manage the number and source of extensions, then discuss the patching speed, and finally rely on content review and backup as a last resort: the core version update frequency is low, and extensions are the layer that changes every day.

2026-11-01

With so many entries in the vulnerability library, which ones should be upgraded first?

How many entries are in the public vulnerability library, and which ones should be upgraded first? The cumulative entries in the open source vulnerability library have reached 55,491, and it is unrealistic to follow each one. The sorting is divided into three layers: those with evidence of in-the-wild exploitation, those affecting authentication and station-wide operations, and those that only need to replace the version for repair. Before upgrading, make sure to complete the backup and rollback points, and then check if the version range you are using falls within the affected range.

2026-11-01

Is the language version marking of multilingual sites a means to improve ranking?

Is the language version tag of a multilingual site a ranking enhancement means? According to the instructions verified today, such tags themselves cannot enhance ranking, they solve the step of taking visitors to the corresponding language and regional version. The real benefit lies in attribution: the translated page is correctly identified, not considered as duplicate content, and only then can it have a stable entry for crawling and indexing. The configuration of a multilingual site and the translation of the entire page should be viewed together with the tag, and the tag alone does not cause a change in ranking.

2026-11-01

Is the proportion of spiders blocked in training increasing, how should robots be grouped?

The proportion of blocked crawler training is rising. According to the samples checked on the day, the statistics of the robots.txt files of nearly five thousand top-level sites show that 14.3% are blocking GPTBot, and 19.2% are blocking at least one AI training crawler. It is recommended to divide the grouping writing by the purpose of crawling: search crawling, retrieval citation, and model training should be authorized separately, expressed by different user group rules, and llms.txt is only responsible for navigation without participating in authorization.

2026-11-01

After the garbage update, the fluctuation of rankings, first check which groups of site data

Rank fluctuation after garbage update, check the collection first and then the content. According to the October site owner report on the day, the garbage update in September is still fermenting in October, and at the same time, about a hundred publishers have obtained a small proportion of ad revenue in the artificial intelligence contribution pilot. The self-check order is: crawling and indexing status, invalid addresses in the site map, keyword library and anchor text distribution, and finally the ranking of specific word groups.

2026-11-01

AI answer the source of reference changes every round, what is left on the content side?

The change of reference source every round is the norm, not an exception. According to the criteria checked that day, only 30% of brands can remain visible between two consecutive answers, and 86% of the references come from brand-owned or nearby sources. There are still three handles on the content side: write paragraphs as independent extractable units, concentrate the pages worth reading in the owned domain and use llms.txt for navigation, maintain consistency and freshness with content review and scheduled publishing.

2026-11-01

Is it really easier for AI to cite statistical numbers in the main text?

According to the guidelines checked on the day, the visibility of content with statistical figures in AI answers is increased by 30% to 40%, but this is not an income that can be obtained simply by adding numbers. The numbers must have a scope and a verifiable source, and the paragraph should be able to stand independently; otherwise, isolated numbers will actually reduce credibility. The focus on content is determined by the keyword library on what questions to write, and content review ensures consistency in the scope of reference.

2026-11-01