Does the AI crawler have dozens of user agents, and should training and search classes be separated?

The AI crawler has dozens of user agents, but it's not as good to remember them one by one by name as to divide them by purpose: training, search, and user-initiated classes. The reason for separating them is that the benefits of the three are completely different - training class absorbs your content into the model, and the site does not get backflow; search and user-initiated classes will directly bring visitors back to the site. The list only solves 'what names are there', and grouping solves 'which should be placed, which should be blocked'.

What user agents do training and search classes refer to?

After grouping by purpose, common user agents can be classified as follows:

A 2026 visibility priority list provides 14 relevant user agent names, and another monthly updated list has already listed 35 AI spiders. The difference in number comes from the scope of collection: the former only selects tokens directly related to visibility, while the latter counts the crawling programs of each product line. So, 'dozens' is the norm, but what needs to be dealt with is the category, not all names.

The judgment of the training category can be referred to in the official description: The content collected by GPTBot may be used to train its generative basic model. This is exactly why it is suitable to be separated from the retrieval class.

How to only block training in robots.txt but not indexing

The expressiveness of robots.txt is grouped by user agent, so the separate blocking style is: a separate Disallow group is set up for training, and indexing and user-initiated classes are allowed separately. There are two points to note when writing.

First, list the token names one by one. Writing a general group name like 'AI Spider' is not effective, the rules only recognize specific user agent names. Second, programs with the same name but different purposes should be checked separately, for example, search classes and training extension classes are two different names, narrowing one does not mean narrowing the other.

For sites targeting visibility, the common suggestion is to relax restrictions on search and user-initiated classes, while imposing limitations on training classes, which is exactly opposite to 'block all AI'. This is not a matter of stance but of indicators: if you want cited and returning visitors, blocking the traffic-carrying crawling at the same time is equivalent to cutting off the source of visibility.

Why would writing one Disallow statement affect the citation?

A Disallow covering all user agents is a typical consequence of a粗暴做法: training, retrieval, and user-initiated classes are all rejected, the site is no longer used for training, but it will also not appear in the answers. The meaning of writing rules in groups is to separate these three categories and decide, so that 'limit training' and 'keep visible' can coexist.

Another common mistake is to only modify robots while ignoring other layers of access control. robots.txt expresses the intention to be crawled, while server-side throttling and blocking is another set of mechanisms. When the two sides are not consistent, the actual visit will not match the expectation.

AnQiCMS backend modifies robots and generates llms.txt operation sequence

Both have different tasks: robots.txt controls whether access is allowed, llms.txt controls which navigation the large language model should read. The AnQiCMS (AnQi content management system) backend provides Robots.txt configuration and supports automatic generation of the site's llms.txt, making it convenient for large language models to understand and index the site's content, the GEO-related backend modules and the SEO modules are in the same configuration group.

The recommended order of operation is as follows: first, organize the user agent list by group, then modify the robots configuration, confirm that the generated results are consistent with expectations, then check whether the content entry pointed to in llms.txt is the page you want to be referenced, and finally, over a period of time, look at the actual user agents visiting the server logs to verify whether the rules are effective as expected.

Common questions

Is it convenient to block all AI spiders directly?

Convenient but at a clear cost: while training classes are blocked, search classes and user initiated classes are also rejected, leading to a decline in the visibility and return traffic of the site. The cost of grouped handling is only to write a few more rules.

How often is the list updated?

A review once a quarter is enough. The focus is not to copy all new names into the rules, but to determine which category the new names belong to. After categorizing, adding names is just a line added to the same group.

How long will it take to see the effect after changing robots?

Changes on the crawling side are usually reflected within a few days, but the log should be the standard. The verification method is to count the visit volume of various user agents before and after the rule change, rather than judging by impression.

Can llms.txt replace robots.txt?

Cannot. llms.txt is a navigation suggestion and does not constitute access control; to truly limit a type of crawling, it still needs to be expressed simultaneously in the robots configuration and on the server side.