More than 30% of the sampled sites block training crawlers, while only 20% block retrieval crawlers.

In the sample, more than 30% of the sites block training crawlers, while only 20% block retrieval crawlers. These two numbers indicate that the content site does not make a binary choice between 'whether to be read by AI' but rather makes a scene-based selection: cutting off training crawling with no return and retaining retrieval crawling that brings references. A publicly collected dataset from October 2026 used 15 user identifiers to test over two hundred sites, with 35% blocking at least one training crawler in the samples that can be read normally, and 21% blocking at least one retrieval crawler. The difference is fourteen percentage points, and the direction is very clear.

What do these ratios refer to?

The purpose of training crawlers is to batchly take away positive**corpus, and the site does not get any return from such crawling, so the motivation to intercept is strong. The purpose of indexing crawlers is to include pages into the index, and later may provide source links in the answers, and intercepting them means giving up the opportunity to be cited.

The ratio is counted according to 'at least intercept one', so it reflects the existence rate of the strategy, not the intercepting strength. If a site intercepts one from the training crawler, it is also counted as 35%; this also explains why the denominator of the two groups is the same batch of readable sites, and the difference in results can reach more than 10%.

Pay attention to the sample aperture: out of more than two hundred test sites, more than two hundred and sixty can be read normally, and the rest are timeouts, jumps, or unresponsive to identifiers. When reading, you cannot treat the two proportions as the distribution of the entire station.

Why is the interception rate one segment lower?

The main difference comes from the cost-benefit structure, not technical limitations. Blocking training crawlers almost does not cause any loss: it does not bring citations, does not bring visits, and the site is less one way of crawling, which saves bandwidth. Blocking search crawlers requires a visible cost - giving up the source exposure in the answers.

The second level is the difficulty of recognition. The same robots rules need to distinguish between two types of identifiers, requiring two sets of path strategies; while a large number of crawls do not follow robots, rules written are not equal to being executed. Therefore, many sites choose to allow only the clearly rewarding path, and the rest use the default strategy to observe first, rather than a full封锁.

The third level is the difference in content assets. Original reports and tool documents yield benefits directly after being cited, while image libraries, aggregate pages, and directory pages are more afraid of being taken away entirely. These two types of sites have different intercepting tendencies to begin with, and gaps naturally appear after mixed statistics.

Which layer does the robots configuration manage?

The robots manage the 'statement'. AnQiCMS supports robots.txt configuration, which can include paths that should not be crawled: admin directories, search result pages, and duplicate pages with parameters. These have no indexing value and are easily bulk traversed.

Be aware of the boundary of the declaration layer: it is a cooperative rule that only applies to identifiers that follow the protocol. It has no binding force on the crawling program that does not read the rules. Additionally, banning crawling does not mean it is not cited - even if the banned crawling rule is added to the indexed page later, it may still appear in the results for a period of time. Therefore, changes to the rules should be coordinated with the collection check, not just completed after writing.

Which layer does the interference code manage?

It manages the interference code that 'can it still be used after obtaining it'. AnQiCMS is built-in anti-crawling interference code, inserts content that has no effect on normal display and is harmful to bulk copying into the page, allowing the collector to take away is incomplete text, while maintaining the integrity of human reading. This layer complements the relationship with robots: one declares boundaries on the front end, and the other protects the quality of the finished product in the data.

Content collection is also an industry fact - automatically collecting content from other websites is a common practice, and the speed of site removal is often faster than the speed of indexing. Therefore, only doing indexing optimization without content collection protection is equivalent to putting the finished product on a shelf without supervision.

For sites that use llms.txt, there is another consideration: LLMs class files are the entry points prepared for model reading, and the rules should be consistent with the content protection strategy. If the paths blocked by robots in llms.txt are listed as recommended sources, the two sets of statements will be contradictory. Determine the priority first before writing the file.

How does the content station determine strategy

Converge in three steps:

Firstly, classify. Make a list of pages that are expected to be referenced and pages that should only be digested internally, to avoid using whole-site level rules to handle page-level differences.

Then, layer. Robots handle path-level declarations, interference codes handle high-value text, and sensitive content uses VIP grouping and permission control instead of relying on no-indexing.

The final review. After adjusting the strategy, we need to observe the changes in both inclusion and citation, the performance of the retrieval type being mistakenly blocked is that the cited users disappear, and the performance of the training type not being blocked is that there is no abnormality in the traffic report, but the content appears elsewhere. Both require checking the logs rather than just the reports.

Two-level division comparison table

Common questions

Can this number be used as an industry benchmark?No. It is a实测 distribution of a single sample size, the site type and regional composition will affect the results, it is more stable to use it as a reference direction than as a benchmark.

Will blocking the training crawler affect the search inclusion?There is a risk. The names of the identifiers of the two types of identifiers are similar, it is easy to mistakenly block the search class if the rules are written broadly, after the change, it needs to be confirmed separately whether the search class crawling is still going on.

Will interference code affect normal user copying?There will be an impact, but it will only affect bulk copying of entire paragraphs. When copying segments manually, the entire text is visible, but it does not show up in reports, and manual page inspection is required.