So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The sites that block training crawlers are almost no longer cited, can this data be implemented directly?

📅 2026-11-01 👁️ 0

It has been found that the domains blocked by the training crawler almost cannot obtain citations. Can this conclusion be directly followed? It cannot be followed directly, but the method can be followed. The original reading of this research is: On the blocked domains, the average number of citations is close to zero, while on the domains not blocked, the average number of citations is two orders of magnitude higher. The gap itself is credible, but the problem is that this number answers 'the blocked domains and the unblocked domains look different', not 'the citations will drop after blocking.'

What did the original numbers say?

The study sample consists of one thousand five hundred and eighty domains, of which approximately one-tenth are explicitly blocked in the robots configuration from training class crawling. When comparing the number of references of these two groups of domains, the value of the blocking group is close to zero, while the value of the unblocked group is significantly higher.

This is a cross-sectional data: it records the differences in the status of two groups of domains at the same time point, without tracking any changes in the domain's blocking settings before and after the change. To support the conclusion 'changed will fall', it requires a comparison of the same batch of domains before and after the change, but such data does not exist in this study.

Why can't we do it directly?

Exclude domains captured during training, as they naturally have obvious industry biases. News, publishing, and media sites are excluded more often, as their content is time-sensitive and has low structuring, and the way they are cited by the model is different from that of product documents and tutorial sites; conversely, the proportion of tools and documents in the domains not excluded is higher, and this type of content is inherently the kind that the model loves to quote.

That is to say, the two things of 'interception' and 'less cited' may be determined by a third factor at the same time, namely, the type of site and the form of content. Treating it as a cause-and-effect relationship will lead to two types of misjudgments: one is to relax all defenses in order to cite, and the other is to lose control over collection and training purposes because the defense is removed.

Three pitfalls of the sample aperture

The first pit is the denominator. The ratio obtained from counting by domain name and by page is different, and the composition of the head domain name and the long tail domain name is also different; the difference in the readings of two studies at the same time is several times, mostly from the denominator rather than the conclusion.

The second pit is a mix of categories. Training, retrieval, and user-initiated crawling are three different purposes. If the research only counts one user agent, what is read is actually just the situation of this chain, and it cannot be generalized to the entire category. The specific identifiers written one by one in the robots configuration are for verification, and should be judged item by item according to the name, not generalized by category.

The third pit is the scope of citation. Being cited once is different from being cited repeatedly, and being cited by summary is different from being linked. Comparing the average number of citations between two groups of domain names flattens the long tail distribution, and the group that is close to zero is likely to have a large number of samples that are actually zero.

The smallest experiment on your own site

Before following the steps, first conduct a small-scale validation on your own site. The method is to select a group of pages with similar content forms, keep the blocking status of one group unchanged, and release the other group as needed. Observe for more than two weeks, and compare three things: the actual visits of various crawlers in the logs, the number of times the site is mentioned in the answers, and the changes in actual traffic volume brought about by this.

The change should leave a对照与回退路径：Interception rules are written in the robots configuration and can be adjusted item by item, llms.txt only affects the way the model reads the site structure, both should be changed separately to determine which layer caused the change. Do not change the content form at the same time during the observation period, otherwise the conclusion will be affected by two variables.

Common questions

Is this study completely unreliable?The可信的部分 is the difference in the state of two groups of domain names in the sample. What is unreliable is to directly explain the difference as the result of interception, which belongs to inference rather than measurement.

Should I default to opening it up?Determine according to the purpose. Sites that are intended to be searched by AI should not be blocked, as retrieval and crawling initiated by users are usually not blocked; sites that are not intended to be used for model training can be restricted only for training class identifiers.

How long should the observation be?At least two weeks, and it should cover one configuration生效 period and one crawling cycle. Fluctuations below a week are easily affected by single scheduling or temporary caching, and stable differences cannot be read.

Related articles

Pages with ads are blocked by default for training crawler, what configuration line should the new domain check?

The newly connected domain blocks training and proxy crawlers by default on the page displaying ads, while search crawlers are still allowed. The verification should be done in three layers: which domains and page types it affects, the user agent names written in the configuration, and the actual visits and rejection results in the access logs. This article also explains the division of labor between access control, site map, and llms.txt.

2026-11-01

The big version at the end of the year comes with a new default theme, and before the upgrade, the test site should be scheduled according to which nodes.

The big version at the end of the year is released with the new default theme, the testing station schedule needs to be reversed according to the official nodes: in the test version phase, only compatibility detection is done, in the candidate version phase, the regression of four types of surfaces is completed, and on the day of the formal version, it is grayed out first and then fully released. This article gives the five-step order on the day of the upgrade, three options for the rollback path, and the check items for the pseudo-static rules and backup.

2026-11-01

The graphic captcha is bypassed by session replay, do we need to add another layer of form anti-spam?

The validation module did not completely invalidate the opened validation session, and the submitted forms may be replayed. This article explains where the replay bypass occurs, what honey pot fields and human verification services block, and provides three sets of acceptance comparisons after going online, as well as the position of sensitive word filtering and content review at the bottom layer.

2026-11-01

The module vulnerability announcement was exploited in the wild on the same day, and if there is an extension of the same type installed on the site, check which step first.

Module security announcement marking the use in the wild, the troubleshooting sequence is to first confirm whether the module is installed, then compare the affected version range, and finally check the business data according to the suggestions in the announcement. This article takes the access bypass announcement of the e-commerce module as an example, and gives the cross-checking method of the extended list and the three checks after the upgrade, and explains the role of the default closed operation at the whole site level in the troubleshooting.

2026-11-01

The adoption rate statistics of two llms.txt files differ by more than a factor of two, so which one should be used to allocate work?

The adoption rate of llms.txt is the same, but the readings from different studies range from ten percent to nearly fifty percent, the gap mainly comes from the denominator rather than the conclusion. This article decomposes several statistics according to the sample size and what they can answer, and gives three indicators that are more important than the adoption rate: whether it has been read, whether the content is consistent with the site map, and whether the maintenance cost can be automated.

2026-11-01

Only about forty percent of the pages cited by AI overview are in the top ten, should the external links be adjusted?

A study covering over 800,000 search result pages shows that the proportion of pages cited by AI overview and ranked in the top ten has dropped from over 70% to about 40% in a year. This article explains why this set of data is not enough to overthrow the external link plan, what aspects of the external link side do not need to be changed, and the two things that can be done first, such as the keyword library, anchor text, and paragraph structure within the site.

2026-11-01

Two weeks after the update, the rankings are still fluctuating. Should we change the content in this round?

Completion of push does not equal stable ranking. This round of garbage update took two weeks to complete, and the official also indicated that recovery may take a month. This article gives the order of handling within two weeks after completion: first, fix the baseline and the timeline of self-change, and then judge whether to change the content according to three types of signals. How to minimize the scope of changes when batch adjustments are needed.

2026-11-01

Exporting the entire old station into a package and then importing it, will the attachments fall outside?

The packaged file exported from the entire station usually contains media files, whether the attachments will fall outside when imported across systems depends on whether the path and fields can correspond. This article compares the three import routes of packaging, table, and compressed package, respectively, what they take away, explains where the upload volume limit is stuck at each level, and gives the three acceptance and rollback preparations after migration.

2026-11-01