Is the proportion of spiders blocked in training increasing, how should robots be grouped?
Is the proportion of trained bots being blocked increasing? How should robots be grouped? According to the samples verified that day, the robots.txt of the leading stations already has 14.3% blocking GPTBot, 19.2% blocking at least one AI training class bot, and blocking training class crawling is becoming a common choice. The grouping idea is to divide three categories by crawling purpose: search crawling, retrieval citation, and model training, with each category writing user group rules instead of a global Disallow to solve all problems.
Let's look at the sample proportion first.
This statistics comes from a sample of robots.txt from nearly five thousand sites in the top one million sites of Tranco: 14.3% block GPTBot, and 19.2% block at least one training crawler.
The difference in the two numbers shows the attitude: there are fewer sites that directly block specific training crawlers, and more sites that block 'at least one', indicating that most sites deal with crawlers individually according to the list, rather than a blanket approach. For content sites, this distribution is more reasonable than overall blocking - the benefits of training use and retrieval use are different, and the authorization should also be different.
The purpose of crawling should be divided into three categories
The purpose of search crawling is to build indexes, the benefit is natural traffic, and almost all content sites allow it.
The purpose of retrieval is real-time or near real-time retrieval, the answer will include source links, the benefit is that it can be cited and引流，usually allowed, but it is necessary to control the directory range, avoiding opening low-value addresses such as preview pages, search result pages.
The purpose of model training is to absorb the text into parameters, the benefits are not directly visible, and the cost is reflected in bandwidth and content reuse, therefore, this type of need to express a stand-alone position. After separating the three types of use, the rules have explanatory power, and subsequent adjustments only affect the target category.
How to write the grouping rule
In terms of writing, divide according to user groups, each group only declares a position of use, avoiding mixing different uses in the same paragraph. Suggested order:
| User group | Purpose | Suggested position | Supplementary constraints |
|---|---|---|---|
| General search class | Establishing an index | Full-site release, excluding background and preview paths | The site map only submits the addresses within the release |
| Search and summary class | Reference answer | Allow public content directory, limit parameter page | Text paragraphs are self-contained, easy to extract |
| Training class | Absorb into the model | Explicitly allow or deny according to site strategy | Maintain consistency with the open range of llms.txt |
| Data collection and batch download class | Batch transfer | Limit frequency and screen list traversal path | Cooperate with anti-crawling interference code |
Three common errors worth avoiding: writing one global rule and expecting it to differentiate between uses; using path masking instead of user groups, resulting in both search and crawling being blocked; frequently changing lists without keeping records, making it impossible to explain the traffic changes on a certain day.
The Robots.txt configuration is a maintainable item in the background, the AnQiCMS GEO and SEO related modules total 14, including robots, llms.txt, anti-crawling interference codes, etc., the configuration is centralized, making it convenient to align the list with the open range at one time.
Division of labor with llms.txt
robots solve authorization: can it grab. llms.txt solve navigation: which pages to read. Both cannot be substituted for each other.
Only allow without navigation, the model has to find the focus in the whole site by itself; only write navigation but prohibit crawling in robots, it is equal to giving a list that cannot be read. AnQiCMS supports automatic generation of site LLMs.txt, convenient for large language models to understand and index website content, and should be configured to list pages exactly within the range allowed by robots.
If the site has multiple language versions open at the same time, the authorization and navigation of the language directory should also correspond one to one, to avoid a language version being blocked in robots.txt but appearing in the recommended list in llms.txt.
How to review after grouping
Three observations: whether the access frequency of each user group in the crawl logs has changed as expected, whether the references and redirections from the answer engine are still present, and whether the proportion of bandwidth and invalid requests has decreased.
Screening training type web crawlers usually first see a reduction in invalid batch requests, followed by changes in indexing and citation surfaces. If the index volume slips, prioritize checking whether the path writing has mistakenly injured search type crawling, rather than rolling back the entire grouping strategy.
Common questions
Question: Will screening training web crawlers affect the AI citation?It affects training use, but does not mean losing citation. Whether the citation depends on the retrieval type crawling being able to read the page, this type should be kept open.
Question: Should we write wildcards to screen all new web crawlers at once?Wildcard writing will mistakenly injure unknown purpose crawling, and the list changes frequently. It is more controllable to declare each known user group individually.
Ask: Is there still a need for robots.txt after llms.txt is written?Yes. The former is content navigation, and the latter is access authorization; their functions are different.
How often should the list be reviewed?Decided by the emergence of new crawlers and changes in traffic structure, usually reviewed immediately after discovering abnormal batch requests or site-wide revisions.