Two days after the change of robots rules, AI spiders are still crawling, how to explain this caching issue?

After the robots rule changed two days later, the AI crawler is still crawling. How should we explain and deal with this caching issue? The most likely explanation is that the crawling party is still using its own cached set of rules. Most AI crawlers check the authorization files of the site before crawling, but they cache it for performance reasons, usually reading it again every 24 to 48 hours. That is to say, the effective time of the rule change is 'the next read', not 'the next request', and the two-day observation window just falls on the edge of the caching cycle.

The rule change does not mean that the crawler will immediately follow the new rule.

It is necessary to clarify the nature of the authorization document: it is a voluntary agreement requested by the crawling party, and there is no mandatory force on the site side. Caching is an inherent cost of this agreement - the crawling party needs to access a large number of sites within a very short period of time, and it is impossible to read the rules again for each request.

Therefore, the judgment of 'Changed but not effective' must include a waiting window. The order of judgment should be: first confirm that your file has indeed been updated (access with a random query string address to exclude interference from intermediate layer caching), then look at the time of the last reading of the rule file by the corresponding crawler in the capture log, and finally determine if it is still acting according to the old rules. If there is an intermediate reverse proxy or caching layer, it needs to be confirmed separately whether that layer has cached the rule file itself. This situation often manifests as the rules have been corrected on the source site, but the external read is still the old content.

First, see what kind of crawling it is in the log.

Not all abnormal requests belong to the same category. Log verification should be divided into at least three piles: search engine main crawler, AI training and retrieval type crawler, and batch collection disguised as normal user agent. The handling methods of the three categories are completely different.

The first category affects inclusion, the handling method is to cautiously adjust the scope of the rules; the second category is the subject of this discussion, the handling method depends on the site's judgment of the reference value; the third category does not comply with the rule document at all, and it is meaningless to talk about agreements with it, it must be blocked with access control and content layer means. If the high-frequency requests in the log carry uncommon proxy identifiers, are concentrated on list pages and detail pages, and have an abnormal uniformity in access depth, it is more likely to be the third category, and even if the rules are changed ten times, there will be no change.

In addition to the robots.txt file, do we need a second gate?

To. The authorization document solves the issue of 'whether to cooperate', and the second gate solves the issue of 'what to do if not cooperating'. Each gate manages a section:

The content layer uses anti-collection interference codes. It acts on the content that has been taken away, leaving recognizable traces during copying and bulk collection, which belongs to post-event protection and will not change whether the crawler enters or not.

The rule layer uses content review and sensitive word filtering, as well as keyword replacement. It manages the entry point of content generated within the site, such as comments, forms, and user submitted fields. If these are bulk-written, the page quality decline caused by crawling anomalies is often misinterpreted as the rules not taking effect.

When a hard interception is truly needed, it falls on access control: limit the frequency of exceptions, and authenticate interfaces that do not need to be publicly accessible. These are all executed at the application layer and have nothing to do with the authorization file.

Write the effective period into the change order.

The most practical action in dealing with such problems is to turn it into a verifiable process rather than repeatedly changing the rules. The change order should clearly state four items: which rule was changed, what the purpose is, how long the expected effective window is (set to a maximum of 24 to 48 hours), and what data is used to confirm effectiveness (corresponding to the request time of the crawler reading the rule file and the subsequent change in the amount of crawling).

AnQiCMS configures the authorization file as an independent setting within the site, after modification, it is still recommended to observe the crawling distribution according to the above cycle; there are also active actions such as link push and sitemap in the same site, which solve the issue of whether they want to come, and the authorization file solves the issue of what can be crawled, which is not the same thing, do not mix them when scheduling.

Common questions

Should we actively notify the crawler after changing the rules?The mainstream crawling parties do not provide a channel for the rules to take effect immediately. The actual practice is to submit the updated sitemap and continue to observe the crawling distribution.

How to judge whether the cache is the main cause?Whether the same crawler still requests paths allowed by the old rules and prohibited by the new rules after the rule change, and compare it with the time it last read the authorization file, if both match, it belongs to the cache cycle.

Is the authorization file useful for training crawlers?It belongs to an agreement, whether to comply with it or not is decided by the crawling party; the choice between training and retrieval types is different, and it needs to be judged separately by the log distribution, not just by the change of a rule.

How long should you change means if there is no change?If there is still no change after more than two cache cycles, it should not continue to change the rules, but to do access control and frequency restriction.