So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The proportion difference of AI crawlers in different industries is very large, who should we compare with?

📅 2026-11-01 👁️ 0

The proportion difference of AI crawler across different industries is very large, who should we compare with? We should compare with the baseline of the same industry and purpose, and not take the average value of the whole site as a reference. The measurement of a document covering 1,103 sites on the day shows that only 18.9% of the sites explicitly block at least one major AI crawler; however, when broken down by industry, the blocking ratio of news and media sites is 61.0%, and for development tool sites it is only 2.9%. Applying the same average value to these two industries will lead to completely opposite conclusions.

Why should the blocking rate be analyzed by industry?

Another issue with the average value of the entire site is that it mixes two completely different strategies. The position of the publishing industry comes from content as an asset: the text is taken in bulk for model training, which is equivalent to giving one's own production costs to the other party, and therefore the interception ratio is naturally high. Another analysis of large publishing institutions on the same day gave the strength of this industry, with sixty out of seventy surveyed publishing institutions explicitly restricting at least one type of AI crawler that has been configured, and fifty-seven of them are completely blocked.

Development tools and document sites are opposite. Their income comes from being searched and cited, and the appearance of their own interface descriptions and tutorials in the answers is effective traffic, so the interception rate is very low. Determine your position first, whether you are closer to one side: selling content, or attracting customers through content.

Training and retrieval should be separated.

The premise of grouping is to know the division of labor between two types of crawlers. The criteria on the day of verification were written very clearly: training and retrieval are now separated into separate crawling agents, which can block GPTBot, Google-Extended, and ClaudeBot used for training, while retaining the qualification for AI retrieval citations. Writing both in the same rule block is the most common source of misjudgment - originally intended to reject corpus collection, but as a result, the citation position in the answer was also lost.

Intention The desired effect Handling of robots.txt Judgment basis
Rejecting corpus collection Not for training Separately blocking training agent entries Is the text your sellable asset?
Keep the cited content Enter the answer source Allow search agents to pass Whether traffic is brought by answer exposure?
Protect user data Do not capture interaction with the background Precisely block the path prefix Whether it contains login, forms, and private content?
Observe unknown agents Take a good look before deciding Conservative release of public paths Whether there is UA and request volume records

Be aware that there is also a hierarchy within the publishing industry: complete blocking and partial restriction are two positions, and the choices of institutions of different scales in the same industry are also different. When benchmarking, choose the level that is close to your content form, not just the industry name.

What if you choose the wrong benchmark

The three most common mistakes.

The first type is to take the practice of large stations as a baseline. The blocking of leading publishing institutions relies on brand and bargaining power, and they can withstand the results of being omitted by AI answers; small and medium-sized sites rely on answer exposure to bring visits, and copying the blocking is equivalent to giving up traceable traffic.

The second type is to treat the average of the entire site as its own industry's number. The above two differences show that such misreading is easy to occur: the interception rate of less than 20% of the entire site, when applied to media sites, will underestimate the risk, and when applied to development tool sites, it will overestimate.

The third type is to only look at the interception without looking at the output. Blocking the crawling agent does not mean blocking the collection, and the content of the page can still be bulkily taken away in other ways.

How to determine the strategy of this site

Determining the strategy requires two steps: one for the access layer and one for the content layer, with each layer handling its own affairs.

The access layer uses robots.txt to define boundaries: group by family, separate training and search classes, and accurately block background paths and result pages. After changing the rules, check the logs to confirm that the families allowed are consistent with expectations.

The content layer manages whether the collected content can be used directly after collection. Anti-crawling interference codes are inserted into the text to be unobtrusive to individuals and effective for bulk copying,配合sensitive word and content review, so that the text taken away does not retain its original form. The significance of this layer lies in: robots.txt can only take effect on agents that comply with the protocol, while interference codes face those who do not comply with the protocol.

Review it once at the end of the quarter. The industry baseline and the agent list are changing, and the grouping that was reasonable last year may not be suitable this year.

Common questions

There is no concept of traffic budget, do you need to manage the crawler?Yes. Web scraping requests also consume server resources, and unrestricted batch requests will occupy the response of real users.

Is it the safest to block all of them?No. Blocking search agents will also lose the opportunity to be cited, and collection will not stop because of this, but will become an untraceable, uncontrolled part.

Will the interference code affect the reading of real people?Proper configuration will not affect it. Its goal is to distort the entire copied text, while the presentation layer remains the same for normal access.

How often should the rules be reviewed?It is recommended to do so quarterly. The renaming and addition of agents are frequent, and too long an interval will cause the rules to deviate from the actual distribution.

Related articles

How many times is the brand mentioned in AI answers, and how to measure this indicator?

How many times is the brand mentioned in AI answers, which can be measured by three indicators: visibility, average position, and sentiment tendency: Visibility is the proportion of questions mentioned, average position shows which part of the answer it appears in, and sentiment tendency determines whether the mention is positive or negative. This article explains how to conduct a large-scale measurement by fixing the question set and engine range, conducting periodic retests, and explaining how llms.txt and the background GEO module receive the measurement results.

2026-11-01

The natural flow decreases week by week, when should we go back and modify the old article?

The average speed of content attenuation is cumulative on a weekly basis. The statistics checked on the day show that the article loses about 0.12% of the traffic each week, and it is halved after about fifty weeks. This article gives the judgment line for triggering the revision of the article: if the natural traffic of a single page drops by more than 20% in 90 days, it is considered a signal, and explains how to arrange the review order, why it is necessary to use the time factor to make the update time visible after the revision is completed, as well as the use of the keyword library in the topic review.

2026-11-01

How much difference is there in the pass rate of mobile performance on mainstream website building platforms?

In the data checked on the day, the difference in the pass rate of mobile performance between the high and low positions is about thirty percentage points. Under the same criteria, the mobile end of the entire station is about nine percentage points lower than the desktop end. This article breaks down the gap into two parts: platform layer and site layer, explains that platform selection determines the baseline, and page optimization determines whether you can reach the top of the platform, and gives the sequence of acceleration actions.

2026-11-01

AI crawlers are not just the leading few, should the budget for crawling be redistributed?

The flow of AI crawlers is no longer concentrated in just a few major ones, the proportion of the part outside the head has increased to about five times the original within a year. This article uses the current day's verified crawling distribution data to explain why assigning robots.txt to just a few old faces is no longer enough, and provides the order of redistributing crawling budgets by crawler group, as well as the division of labor between robots.txt and llms.txt, and the verification method after the modification.

2026-11-01

Can llms.txt replace robots.txt to manage crawling?

llms.txt cannot replace robots.txt in terms of crawling. The division of labor confirmed on the day is: robots.txt determines whether it can be crawled and which paths can be accessed, while llms.txt is about what to look at first after the model comes in. This article explains the levels of solutions provided by the two files, the order of priority when configured simultaneously, and how to verify whether the changes have taken effect.

2026-11-01

Self-hosted content systems need to allocate a few hours for maintenance each month.

The maintenance hours of self-hosted content systems mainly fall on two things: patches and backups. The operation口径 on the day of verification is that critical to high-risk level patches come once every six to eight weeks, with obvious labor costs in addition to the computing power bill. This article divides the monthly maintenance of the small station into two segments: cannot be saved and can be saved, and gives the method of working hours budgeting, and explains which segment can be saved by the technology stack and deployment method.

2026-11-01

Each website pays separately for the website building tool, how to amortize the cost for multiple sites

Hosted website building tools usually charge a subscription fee per site, and the cost increases linearly as the number of sites increases; the comparison criteria I checked that day is that open-source content systems can use the same installation pool of resources. This article explains the cost structure of a multi-brand matrix, as well as how to group permissions and high-risk operations at the site level after multiple sites are brought into one backend, to avoid thinning out costs while also thinning out risks.

2026-11-01

The platform is growing fast but needs to be locked, where is the migration cost calculated?

The platform's rapid growth does not mean that the content can be taken away. The two sets of data verified on the day show that Wix grew by 32.6% in 2026, accounting for 4.3% of global sites, while the typical cost of rebuilding a small website that leaves a hosting website platform is between six thousand and eighteen thousand dollars. This article breaks down the migration cost into three areas: content structure, historical addresses, and dynamic features, and explains the feasible approach to leaving a path in advance.

2026-11-01