So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The proportion of AI crawler interception changed between two months, is it changed by the default value of the platform or the webmaster?

📅 2026-11-01 👁️ 0

The domain names in the same batch have changed the intercept rate of AI crawlers in two months, is it the platform default value changed or the webmaster changed the configuration himself? In most cases, it is the former. The method of distinction is not guessing, but looking at the direction of change: the same batch of samples are measured once every two months, with the platform default value going online in the middle, and the displacement is concentrated on the side of training and proxy types, while the retrieval type did not move much. If the webmaster changes his own robots file one by one, the proportion of several manufacturers should vary randomly, and will not present such a synchronous rhythm.

What happened between the two measurements?

2026-10-30 The original text opened for verification states: This AI crawler blocking index was measured twice in July and September, covering 10,894 domains. The change that went online between the two measurements was described very clearly: Starting from September 15, 2026, new domains connected will receive new default values, and crawlers classified as training and proxy types will be blocked by default on advertising pages, while retrieval types will continue to be allowed.

The scope of the default value is declared, covering only new domains added, and the proportion of the whole batch of samples still moved. The reasons are twofold: first, the new domain enters the sample with a new default value, which is a structural shift; second, some webmasters follow the same template in bulk to change the configuration, which is the part that webmasters are truly responsible for. When looking at your own site, the two things should be calculated separately.

Align the口径 before comparing

There are two different apertures in the same statistical data. At the site level, the domain blocking GPTBot accounts for 7.9%, and blocking any AI crawler accounts for 10.3%; in the news release industry, these apertures are raised to 50.5% and 56.4% respectively. When comparing the difference over two months, it is necessary to lock the same aperture: taking the later value of 'blocking any one' and comparing it with the earlier value of 'blocking only one', will result in an increase that does not exist.

The industry is stratified similarly. The content itself is a high stock in the industry, and the overall average does not have an explanatory power for it; conversely, using the numbers of the news industry to measure your own product documentation site will treat normal conditions as anomalies. The composition of the industry in the sample changes, and the average will also change, which has nothing to do with the decision of a single site owner.

Why should training and retrieval be separated

The uses of the two types of crawlers are different, as are their costs. Training-type collection is a one-time bulk data acquisition, and blocking it reduces an opportunity for model referencing and also reduces potential referral traffic; search and real-time retrieval types often bring real users, and blocking them is equivalent to shutting off the entry from generative answers. Combining the two into an 'AI crawler' switch, the numbers will fluctuate with the platform's default values, and webmasters will find it difficult to determine what they have actually turned off.

In a self-built program, a more convenient layered approach is to assign three types of uses to configuration: the robots file is written to clarify handling by user groups, llms.txt determines whether the content is aimed at large language model organizations, and GEO-related visibility standards are confirmed separately. Each layer handles a segment, and changes can be attributed accordingly.

The three configurations that the webmaster really needs to adjust.

One is the control file itself: grouped by user group, give handling to training classes, proxy classes, and retrieval classes, do not block the entire section. One is the classification switch on the platform side: the new default value only manages new接入 domain names, old domain names need to confirm the current effective value themselves, rather than assuming it has already taken effect. There is also one at the internal exit: the visibility of the site entrance, llms.txt, and the text page should be consistent, do not appear the index entrance is blocked while the text is allowed to pass, or vice versa.

After changing it, don't just look at the configuration text, but also look at the real return. Record the request volume and return status of three types of proxies separately, and move one layer at a time. If you change three layers at the same time, the result cannot be attributed, and it will still be guessing next time you encounter the same displacement.

Common questions

Q: Has someone changed the configuration if the intercept ratio has increased? A: Not necessarily. New domains entering the sample with the default value of the platform can also cause overall displacement.

Q: Is it enough to only change the robots file? A: Not enough. Robots is a polite guidance, and mandatory interception occurs at the platform or rule level, and both places need to be confirmed together for the current effective value.

Ask: Should the retrieval class be stopped together? Answer: First, see if the requests it brings have entered the station and converted. Most content sites will allow the retrieval class separately, only tightening the training class.

Related articles

After a website is attacked, it is necessary to restore the backup, should the vulnerability be patched first or the data be restored first?

The order of disposal is to first cut off the entry that can be reused again, install the affected programs and extensions, and then recover from a known good backup. Restoring data will bring back the time points with backdoors, and the repeated infections are mostly caused by this. After recovery, it is also necessary to check the accounts, scheduled tasks, and modified files, and confirm whether the deleted content still stays in the recycle bin.

2026-11-01

Should the new article be submitted to the search engine actively, and which companies does this protocol cover?

The proactive submission resolves the section of

2026-11-01

Where is the difference in construction costs between static pages and dynamic sites, and who has higher post-maintenance costs?

The price range of static sites during the construction period is lower, while dynamic sites are expensive in terms of backend, database, and security patches; during the operation period, the situation is reversed, where the content update of static sites depends on technical intervention, and dynamic sites require continuous upgrades and backup work. Sites with frequent updates are more suitable for content systems with backend and scheduled publishing, separating the update actions from the development time.

2026-11-01

After the redesign changed the domain name, do we need to keep the redirection of the old address active?

The redirect after changing the domain name is not an obligation forever, but addresses that have received external backlinks or have stable traffic should be retained for a long time, usually at least one year before evaluation; after removing, the old address will directly return an error page, and the backlinks and weight will be cut off accordingly. The cleaning sequence is to do address mapping first, then cut the traffic, and finally batch replace the internal links within the site to avoid diluting the signal of multi-level redirect chains.

2026-11-01

Why would the proportion of different AI crawlers intercepted by the same site differ by a few points?

The same batch of sites intercepts AI crawlers from different manufacturers, and the intercept ratio differs by several points, mostly due to the rules being written according to the user group name, while each proxy's identifier is not just one: omitting one will result in a difference in the ratio. When checking, first distinguish between the three meanings of no file, empty file, and release, and then rewrite the rules by grouping for use, and finally verify the returned status of each one by using the site map entry and real crawling records instead of just looking at the configuration text.

2026-11-01

Is there any advertisement on the page, and will it change whether the AI crawler can read the content?

Whether there is advertising on the page will indeed change whether the AI crawler can read the content: the new platform default value treats whether to display advertising as a judgment condition, the pages with advertising are intercepted by default for training and proxy types, and the search type continues to pass. This rule is not in the robots file, but occurs on the platform side, so there may be two sets of visibility in the same site with advertising templates and without advertising templates, and it should be checked separately according to the template.

2026-11-01

The new default intercept value only manages new domains, old domains need to be moved manually.

The new default blocking value on the platform only applies to newly connected domains, while the old domains already on the platform need to manually fetch and control this layer: first confirm the current effective values for training, proxy, and search categories separately, then merge them with the existing robots rules on the site, and finally verify with the real return status. The default value will not automatically overwrite existing configurations, assuming it has already taken effect is the most common judgment error.

2026-11-01

Why do we need to tighten the registration entry first and then patch the multi-site user registration open?

The reason why a multi-site with open user registration needs to tighten the registration entry first and then patch it is because the triggering conditions of this kind of risk are written together: registration is open, and ordinary users can create new sites. Closing the registration during the patch window period and restricting the permission to create new sites can directly cut off the attack entry; after patching, it is also necessary to check back on the sites that have been created and to synchronize the backup and permission review.

2026-11-01