After measuring over two hundred sites, only 30% were found to block AI crawlers, so it is necessary to confirm what before following up.
After measuring over 200 sites, only 30% can block AI crawlers. Before following such conclusions, what should be confirmed first? First, confirm which layer it counts: this sample is 261 sites for 15 user agents to write authorization rules, blocking about 35% of training crawlers and about 21% of search crawlers. It counts how the site writes in the robots file, not how many the crawler actually grabs, nor how many cited sites there are. Following up without aligning the口径 is easy to take others' distribution as your own decision basis.
What two things does this group measure?
A thing is a classification of purpose. The same 'AI crawler' is at least divided into two categories: those that use content for model training, and those that retrieve content in real-time to answer user questions. The 15 user agents in the sample are divided into these two categories, and the intercept ratio is not the same between the training class and the retrieval class, with a difference of about 35% and about 21% coming from here.
Another thing is the expression of intent of the site in the robots file. Robots only express the site's requests, the crawling parties that act according to the protocol will read it, and those that do not comply will not stop because of it. So these two proportions describe the distribution of industry writing, not the actual distribution of protective effects.
The cost of blocking and allowing
| Select | What is gained | What is paid | Suitable scenario |
|---|---|---|---|
| Interception training type | The content is not taken as a whole for training | The model's understanding of the site content may decrease | The original site where content is the core asset |
| Allowing retrieval type | More likely to appear in the citations of real-time answers | The frequency of crawling and the increase in server load | Sites that require external traffic and exposure |
| All blocked | Short-term load and the lowest uncertainty | Abandoning the visibility brought by retrieval at the same time | Internal sites and uncensored content |
| All released | Maximum visibility opportunities | Both training and retrieval links without boundaries | Viewing citation volume as the primary goal site |
「30% of the sites blocked the crawler」This sentence's practical value does not lie in the ratio itself, but in the fact that it indicates that the industry has not formed a consensus. The real judgment to be made is: whether the content of this site has been made public, whether the visibility brought by the retrieval type crawling has commercial significance for you, and whether the server can withstand additional crawling frequency.
Besides robots, which two layers control crawling?
The first layer is llms.txt. AnQiCMS supports automatic generation of the site LLMs.txt, which is aimed at how large language models understand the structure of the site - concentrating the main content, boundaries, and descriptions in a discoverable location. This layer addresses how it is correctly read, not whether it can be read.
The second layer is to prevent collection interference. Robots belong to requests, and interference codes belong to output: Normal readers see clean text, while the collectors directly storing the text receive text with noise. This layer is more practical for bulk搬运 that does not comply with the protocol.
The third layer needs to be understood separately: Robots define the scope, llms.txt describes the structure, and interference codes handle the output. If they are treated as one thing, there will be misjudgments such as 'writing robots and thinking they have blocked it' or 'applying interference codes and thinking that robots are not needed'.
Three confirmations to be done before following up.
Step 1: Look at the sample scope: Is the type distribution of these 261 sites comparable to your industry, and are the language and market scope consistent? Step 2: Look at the indicator definition: Is the denominator the number of sites or the number of requests, and is blocking done by user agent line by line or overall blocking? Step 3: Look at your own logs: What actual crawling has your site received, how often, and is there any access beyond the robots.txt rules?
Step 3 is information that is not provided in external reports - only your server logs can explain the actual crawling situation of your site. It's much safer to look at the industry ratio as background information before completing this step.
Common questions
Is this number of thirty percent directly citable?The citation should be specific, written as 'A measurement based on 261 sites shows that about 35% of the training class of crawling robots was intercepted', discussing the proportion without the denominator will become a wrong conclusion.
How to distinguish between training class and retrieval class in the background?Grouped by user agent and access path characteristics for statistical analysis, first look at the distribution and then decide the rules; relying solely on the UA string for judgment is easy to be deceived and affected.
If llms.txt has been added, do we no longer need to pay attention to robots?Both objects are different. llms.txt explains the site structure, robots express the crawling authorization, and if either is missing, it will result in either 'able to read but read incorrectly' or 'not allowed to read but still read' situations.