When blocking training crawlers, first identify the identity of the robot, and control access and reading paths.
When configuring AI crawler strategies, access control is handled by robots.txt, and reading paths are handled by llms.txt, which cannot be substituted for each other. A more common pitfall is confusing identifiers: GPTBot is not equal to OAI-SearchBot, ClaudeBot is not equal to Claude-SearchBot, the same prefix of search and training purposes are two uses, and the error group will also close the entry referred to by the answer engine.
Two files handle different matters
A decision matrix for AI crawler access control splits this into two sentences: robots.txt controls access, llms.txt controls navigation.The former determines whether a certain identifier can capture a path, which belongs to the permission layer; the latter provides a structured reading order to the model after it has entered the page, which belongs to the navigation layer.The same material has also counted the number of mainstream identifiers: there are eight main AI crawlers with different calling behaviors, so the rules should be written by function groups instead of using a wildcard to reject the entire user agent string.If you only want to use it for training purposes and close the retrieval type at the same time, it is equivalent to giving up the qualification to appear in generative answers.
Write rules by grouping by purpose
| Purpose | Common identifiers | Which part should be changed | Typical mistake method |
|---|---|---|---|
| Model Training | GPTBot, ClaudeBot | Grouping rules for robots.txt | Use wildcard to block all identifiers |
| Search and answer generation | OAI-SearchBot, Claude-SearchBot | robots.txt is placed separately | Block with training type in the same group |
| Reading order of pages entered | All released identifiers | Description of llms.txt structure | Thought it could replace access control |
| Traditional search discovery | Search engine crawler | Sitemap and link push | Place the discovery in the configuration file |
The configuration should be verifiable by the log
Check the log first: Count the distribution of requests by identifier, confirm that the search type is still accessing and the training type has stopped, and then see if the rejected ones are on the expected path. The internal AnQiCMS also provides robots.txt configuration and llms.txt auto-generation, the permission layer and navigation layer are maintained in the same backend to avoid each saying its own rules in two places;The generation of the site map is also here, the discovery process does not need to be assembled separately. It should be noted that the identification is declared by the crawling party, following instructions is an industry convention rather than a mandatory constraint, the meaning of configuration is to clearly express the intention and leave a traceable record, it does not constitute a technical isolation, and important content still needs to be protected by access restrictions and authorization.
Cooperation between gateway layer and log layer
The rules are written in the site, but the execution occurs in two places. The CDN or edge gateway's spider management often takes effect before the site, the configuration within the site only takes effect on its own directly connected crawling, when there is a discrepancy between the two sides, the one that hits first is taken as the standard, therefore, after changing the rules, it is necessary to check the response from both the edge and the source site separately.At least four columns should be left on the log side: time, user agent, request path, and status code. Grouping statistics by identifier is meaningful; just looking at the total number of visits does not show structural changes.If the site has enabled caching, you also need to confirm that the rule file has not been cached beyond the expiration time, and that changes have not taken effect, a large proportion of which is actually just the cache has not been refreshed.
Common questions
Question: Can only write llms.txt block training?Answer: No. It does not participate in access judgment, blocking training relies on grouping rejection in robots.txt and corresponding strategies on the edge gateway.
Question: Should case sensitivity be distinguished in the rules?Answer: The matching of user agent groups usually does not distinguish between uppercase and lowercase, path matching does, and if both allow and deny are written in the same file with the same conditions, it is usually taken as allowed first. Write the rules based on this assumption and check each item with the log.
Question: How long will it take to see the effect after the change?Answer: It depends on the cycle of the other party's re-fetching and caching, and the judgment is based on the change in the distribution of requests in the log, rather than whether the file itself has taken effect.