When AI crawlers come to grab sites, who does robots.txt actually block?
The robots.txt blocks AI spiders that report their identity and follow instructions properly, it cannot block those that do not read the rules and directly crawl.Based on a sample study opened on October 27, 2026, reviewed against the original text: In 285 site crawling control samples, 261 of the robots rules could be read, with 35% at least blocking an AI training class crawler, and only 21% at least blocking a retrieval class crawler.That means most sites haven't actually passed the robots expression interception intent, rather than not being able to intercept. Understanding this sentence is key to separating the two different configurations of 'crawling control' and 'search visibility'.
What does the interception ratio in the sample mean
The denominators of the two ratios are readable rules, not the total number of sites. When cited, they cannot be mixed. They answer the question 'How many sites have done this', not 'How useful it has been done'.
| Criteria | Value | Meaning |
|---|---|---|
| Rules are readable | 261 pieces | Robots configuration that can be normally accessed in the sample |
| Blocks at least one training type spider | 35% | Clearly prohibits crawling for training purposes |
| Block at least one search crawler | 21% | Prohibited the crawling identifier used to answer the search |
The gap itself has practical reasons: the identifiers of training crawlers are more concentrated, and the site is easy to cut off all at once; search crawlers often share paths with search engine crawling, with a high risk of collateral damage, and webmasters are more hesitant.
Training crawlers and search crawlers should be treated separately
Before writing rules, clarify the purpose. If you only want to avoid content being used for model training, restrict it according to training class identifiers; if you hope that AI search can still cite the content, don't block a group of retrieval class identifiers. Conversely, mixing the two in a single rule usually results in either wasting training protection or losing the opportunity to be cited. The order of judgment is: first confirm which identifiers will affect the visibility you truly want, and then decide which group to block.
The scope of action for robots configuration
robots control crawling only, not indexing or ranking. Indexing relies on sitemaps and active submission to deliver new links to search engines: AnQiCMS (AnQi Content Management System) automatically generates Sitemap in the background and places the links for Baidu and Bing in independent modules, with separate management of robots configuration, and site-side verification should be performed separately.Similarly, llms.txt and such documents for large models are not within the scope of robots, they address readability rather than crawling permissions.In the background GEO and SEO related modules, these items are independent entries, do not replace each other when configuring.
Another boundary: the robots file is a public file, anyone can read your path agreements, so it is not access control. The real permissions should stay on the server.
The implementation sequence of the site self-inspection
It is recommended to follow four steps. The first step is to capture your own robots file and confirm that it is accessible and the syntax has not been destroyed by template variables. The second step is to list the identifiers of the crawlers to be distinguished and write a list of allowed and prohibited items for the training and retrieval groups. The third step is to confirm that the sitemap has been generated and that the link push has been successfully acknowledged, not just by looking at the background switch being turned on. The fourth step is to review the logs one to two weeks later and check whether the change in the amount of crawling has fallen within the expected categories; if it is found that a certain type of crawler is still arriving, it means that it has not followed the robots, and at this point, it is necessary to switch to server-side measures instead of adding more lines to the file.
Common questions
Ask: Can Disallow in robots.txt completely block AI crawling?Answer: Only blocks the crawlers that comply with the rules. Only 35% and 21% of the sites respectively intercepted the training and retrieval classes, and the majority of the remaining ones did not express intentions, rather than expressing but failing to do so.
Ask: Will blocking crawlers affect search rankings?Answer: Robots control crawling, not ranking rules. The restriction of crawling may affect discovery and inclusion, so it is necessary to confirm the sitemap and active push separately.
Ask: Should we arbitrarily disable all new crawler identifiers?Answer: First, determine whether it brings you引用value. There is often an intersection between retrieval class identification and search engine crawling, and it is easy to mistakenly injure the whole group.