AI crawling is divided into three categories: retrieval, training, and agency. Each site needs to decide on its own which category to choose.

After AI acquisition is divided into retrieval, training, and agency categories, the site must independently decide which category to allow, this is the most practical boundary line in current access control.Retrieval-based crawling replaces search engines and Q&A products in reading pages that can be cited, only if it is allowed can it have the opportunity to appear in the answer; Training-based crawling reads text for model training, usually does not bring immediate traffic to the site;The agent class captures the user's visit this time, reading the page needed for the current answer. In the past, these were handled with a single switch, and the site's desire to tighten training would also block the search and agent from the outside, causing the visibility of inclusion and citation to decline simultaneously, while these two things should not have been tied together to decide.

What do the three types of crawling give to the site?

The content form of the three types of extraction is similar, but the returns to the site are very different. The retrieval type reads column pages, text pages, and explanation pages, producing indexes and source references; the training type reads a large amount of corpus, producing knowledge within the model that the site side cannot see directly; the proxy type is triggered by the user's current question, reads less but has a clear intention, often directly corresponding to a visit or a conversion. Mixing these three types into one column for judgment is likely to lead to conclusions such as 'AI extraction has no value' or 'AI extraction must be fully open.'

What will a switch control for the three types harm?

The problem with a single switch is the mixed use. The existing statistics口径 has categorized a significant part of the verified traffic as mixed use collection, where the same user agent may be used for both retrieval reading and training batch downloads. When tightening the entire segment, the site wants to block a category that is not blocked cleanly, and the category that wants to be retained is blocked first. After the event, from the report, it can only be seen that the overall collection volume has decreased, and it is not clear which category has been mistakenly injured. More麻烦 is that this kind of injury often lags: the decline in inclusion takes several weeks to reflect on natural traffic, while the site's training intercept statistics show that it has taken effect.

Decide the order of passing judgment according to the category

First set the default value by category, then adjust the data, the sequence is roughly like this:

Ask a question first when judging: Will this type of citation bring visitors? If so, let it pass first and optimize the landing page; if not, tighten this type separately, do not affect the first two types.

Where is the backend robots configuration and llms.txt generation?

On the operational level, the robots file expresses the intention to access, telling the crawling party which paths can be read and which cannot, it is about whether it can enter; llms.txt expresses the reading path, telling the large language model what content the site has and where to start reading, it is about how to go after entering. Both are not substitute relationships, and only one of them will leave a gap. Both of these have independent entries in the AnQiCMS backend: Robots.txt is used to set the crawling rules, and the site LLMs.txt is automatically generated by the system, and there is no need to write the file manually after modification.

After the modification, don't omit the verification: first check the access log to confirm which type actually visited, and then check the inclusion and reference reports to confirm that the category released is not affected. It's easy to confuse 'configuration生效' with 'intention achieved' when only changing the switch in the background and not seeing the evidence log.

Common questions

Only blocking the training class, will it affect the search inclusion?As long as the release rules of the search class have not changed, the index reading is not affected. The risk lies in the part of traffic with mixed purposes, after tightening, you need to monitor the crawling log for a period of time to confirm that the blocked objects are indeed for training purposes.

Can llms.txt replace the robots file?Cannot. One explains the content and structure of the site, and the other explains the access permission, with different scopes. Sites that have already placed robots.txt are still worth generating llms.txt, so that the model can read in the order the site hopes; vice versa as well.

The site has no AI visitors, should we just shut it down completely?Shutting down completely saves bandwidth and also cuts off the path that appears in the source of the answer. A more prudent approach is to initially allow search classes, and strictly control training classes according to business judgment, and observe for a period of time before deciding.

How long will it take to see the effect after changing the category switch?Changes in search classes usually require waiting for one round of crawling and indexing, followed by the synchronization of report standards; in the short term, first check the log to see if 'whether it has entered according to the new rules', and leave the judgment of rankings and traffic for subsequent periods.