Large sites limit AI crawlers in robots.txt, should small and medium-sized sites follow suit?

Most large publishers have already restricted AI crawlers in robots.txt, should small and medium-sized sites follow suit and block them? It is not recommended to copy directly.In the analysis of third-party large publishers, 70 of them explicitly limit at least one configured AI crawler, with more than fifty completely banning at least one——but this is a choice made by those with a large content inventory and are in talks with the model side about authorization.The visibility of small websites mainly comes from being searched and cited. Turning off the crawling channel also turns off the opportunity to be found, and the revenue structure is completely different. Whether to follow up depends on whether you are clear about what you gain and what you lose.

Firstly, distinguish between two types of AI crawlers

In the same robots.txt, 'AI crawler' actually includes two types of crawling with different purposes.

A type of service for search and retrieval: the model verifies and summarizes answers to questions, this kind of scraping brings visibility and is also the entry point for subsequent references. Another type of service for batch training: large-scale extraction of the entire site's content for model training, which does not directly return short-term visibility to the site.

The benefits and costs of screening mainly come from the distinction between the second category. The problem is that whether the identification is可信, and whether the other party uses the declared purposes, is not under your control, so the division made by the agent identification is a

robots.txt controls crawling, llms.txt controls navigation.

The division of labor between the two files is often confused. The Robots.txt configuration writes permissions and prohibitions, affecting whether and which directories can be accessed; llms.txt is the site overview, describing content grouping and reading path, affecting how the site is understood after being allowed in.

The two are not substitute relationships. Placing llms.txt does not mean allowing training to crawl, and blocking a proxy does not mean the overview file is ineffective. These two layers within the site are automatically generated and accommodated by the robots configuration and overview file, and the changes need to be verified separately.

What is blocked and what is lost

Small and medium-sized sites generally fall on the second line: need to be searched, quoted, but do not want the whole site to be bulk collected.

The practice of collecting boundaries by category

A more refined approach than cutting with an agent is to classify by content type. The four steps to be taken: first, confirm which columns are public acquisition content (articles, help, product descriptions), and which should not be caught (back-end paths, preview addresses, internal materials, pages with user states); then write prohibitive rules for the parts that should not be caught; then check whether the entire site directory has been disabled while forgetting to separately release the acquisition columns - after disabling the directory, it usually does not take effect to separately release the sub-paths; finally, check the actual抓取 records instead of just looking at the file writing.

A detail that is easily overlooked: the crawling boundary has been changed, and the list and overview files need to be synchronized. The Sitemap should not contain addresses that have been prohibited from crawling; otherwise, it contradicts the rule of not crawling while passing the addresses. The entry groups in the overview file also need to be adjusted accordingly.

Three items to check after the changes

First, check if there is no abnormal decline in search inclusion, compare the crawling status of the main pages within the site with the previous status; second, check if the target column can still be crawled, the most common mistake in rule changes is to ban normal columns along with the prohibited ones; third, check the submission results of the list, and confirm after the site map is re-generated that it does not contain prohibited addresses.

AnQiCMS has corresponding configuration items for these three things: The Robots.txt configuration is responsible for generating and maintaining crawling rules, the site map is automatically generated to ensure that the list is consistent with the actual addresses, llms.txt is automatically generated internally, and the column structure is updated all at once after the changes are made.

Common questions

What will happen if you copy the rules of the big station?The traffic of the big station does not depend entirely on being cited, while the small station depends on it. The same list of banned items is an authorization strategy for the former, and may be a loss of visibility for the latter.

Will it still be caught after blocking?robots.txt is not access control, it only restricts the crawling ends that comply with the protocol. To truly limit, it depends on permission, authentication, or rate control.

Does putting llms.txt in equal to welcoming AI crawling?No. The overview file does not grant crawling permission, whether you can enter is still determined by the robots configuration, and both should be considered together.

How often should the crawling records be reviewed?It is recommended to review according to the change of rules, and add regular inspections at fixed periods. The actual effect and writing of the rules are often inconsistent, only the crawling records can prove.