So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The training crawler was blocked in robots, why can't the content still be retrieved

📅 2026-11-01 👁️ 0

Why can't I still get usable content on the crawling side even though the training crawler has been blocked in robots.txt? Because these two questions belong to different layers: the former asks whether I allow you to crawl, and the latter asks what you have really caught. In the actual test of a batch of more than 2,300 sites, only 3.6% of the text blocks training crawlers, while 18.8% cannot get usable content when directly accessing the homepage as a real visitor. The opposite direction of the numbers indicates that the gap lies in execution and availability, not in the rule text.

Why can't the two percentages be interchanged?

The proportion blocked is low, and the proportion of inaccessible content is high, these two things have different causes and cannot be inferred from each other.

The proportion blocked is low, which may be because most sites have not changed the configuration for new visitor types, or the change only takes effect in text. The proportion of inaccessible content is higher, and the causes are more complex: the page depends on script rendering, the address jump level is too deep, the returned status code is not normal success, and login is required to view the main text, all of which will lead to

The most common misjudgment when mixing the two is: thinking that you have already done protection, so you believe that the external party cannot read the content; or conversely, seeing that someone can read, you think that the rules are wrong. Both of these conclusions should be verified by the scraping behavior itself, rather than by the configuration text.

Three situations where the content can still be accessed after blocking

The first one is the lack of identity verification. The rules match by visitor name, but the actual visit can claim any name. If the server only determines by name, changing the name can bypass it. The real blocking effect should be based on the source of the request, frequency, and behavior characteristics, rather than just the name field.

The second type is that the same content has multiple entries. List pages, interface addresses, old domains, and content distribution node caches may all return the same text, but the rules only cover one path. In this case, one entry is blocked, and the other entries can still be accessed.

The third type is the difference in the rendering layer. The way to grab pages used is different from the browser, and when the script is not executed, the returned is the skeleton content, looking like it has 'gotten' but the text is empty. This kind of problem is most easily ignored when verifying, because it returns a normal status code.

Self-check in the order of four types of signals

The order should be from outside to inside, checking the four types of signals in turn.

Read the rules text: whether the target directory is written in full, whether the name matches is consistent with the real visitor, and which one is to be followed when there is a conflict between permit and block. At the text level, the AnQiCMS robots configuration can be checked directly in the background, and it must be confirmed that the online file and the background setting are the same after the change.

Check the entry list: List the site map, interface address, old domain redirection, and cache nodes, and confirm that they are constrained by the same set of rules. The third step is to check the return quality: Use your own identity to directly access the homepage and several inner pages, and check if the text is complete, if scripts are needed for it to appear, and if the status code is stable. The fourth step is to check the behavior characteristics: If the same address is visited by a large number of different sources within a short period of time, it indicates that the rules have not truly taken effect, and restrictions need to be added on the server side.

Two layers of protection need to be matched within the site

In addition to the verification of the crawling layer, there are two layers within the site that are related to the discernibility of the content.

The first layer is sensitive word filtering and content review. When the content returned by the public interface is consistent with the front-end text, any missing layer may allow low-quality content into the crawling results; configure the review rules well and avoid mixing placeholder text only for the front-end in the main text.

The second layer is for anti-crawling and interference codes. Its purpose is not to replace robots, but to make bulk搬运 lose value in content quality: the copied pages are accompanied by meaningless characters, and it is not smooth to read when republished. This layer needs to be configured according to visitor types to avoid disturbing normal crawling as well, otherwise the proportion of

After the two-layer configuration, it is still necessary to retest the four types of signals mentioned earlier, because the new problems most easily introduced by changing the configuration are rule conflicts and entry omissions.

Common questions

Can changing robots prevent training crawling?Not reliable. The name can be forged, and the same content may be taken from other entries, robots are more suitable as a declaration layer,配合 service-side source and behavior restrictions.

To what extent should the release side be verified?At least verify to the extent of “full text visible”. Normal status code does not mean content is available, the page retrieval method used for crawling often does not execute scripts, and it needs to be confirmed that the response body contains a complete paragraph.

How often should retesting be done?It is more reasonable to trigger based on configuration changes: retest after changing rules, adding new entries, changing domain names, or caching; usually perform a full-scale check once a quarter.

Related articles

The adoption rate statistics of the two llms.txt files are very different, which decision口径 should be adopted?

Two studies that were checked on the same day are very different: The adoption rate is 8.3% when statistics are taken from the top ten thousand domain names, and it reaches 49.2% when statistics are taken from more than two thousand sites. The difference does not come from who made a mistake in the calculation, but from the denominator range and the method of judgment. When making decisions, do not take the average of the two numbers, but first see which category your site falls into among the samples, and then confirm whether the口径 can be used by checking the three-step verification method.

2026-11-01

Most llms.txt files have never been requested, and is it still worth maintaining them?

The analysis of logs from more than 100,000 sites shows that 97% of the llms.txt files have never been requested, with reading concentrated in a few sites. This result does not mean that the file is useless, but rather that it does not solve the two issues of being discovered and intercepted. Whether to continue maintaining it should be judged by looking at whether there are corresponding requests in the access logs of your own site, rather than by looking at whether the file has been uploaded.

2026-11-01

Only a little over 30% of the pages referenced in the AI overview come from the top ten, so should we still pay attention to the ranking?

A large-scale study shows that only about 30% of the pages referenced in AI overviews are also in the top ten natural search results, while this proportion was about 75% a year ago. The decline in overlap does not mean that the ranking has become ineffective: the ranking is still the entry point, but is no longer a condition for reference. The focus should be on which issues the reference hits, whether paragraphs can be extracted separately, and whether the keyword library and anchor text can stably identify the content.

2026-11-01

The garbage station update in September took nearly fourteen days to end, when is it suitable for review?

This round of garbage content update in September started in the late month and was marked completed in the early October, lasting for more than thirteen days. Before the update was completed, the ranking reading was the intermediate state within the rolling period, which cannot be taken as a conclusion; the time suitable for review should meet two conditions: the official panel is marked completed, and the data after the completion day returns to a stable interval. The waiting period should first check basic items such as robots and site maps.

2026-11-01

Which type of spam submissions do the honeytrap field and captcha each block, can only one be left

The honey pot field blocks script submissions that do not parse styles, there is no visible challenge for real visitors; the scoring captcha is scored according to the possibility of robots, at the cost of mistakenly harming some real users. The two methods do not face each other, and whether only one can stand depends on the composition of garbage: when the amount of scripts is mainly used, the honey pot is enough, but when there are submissions with changed addresses and natural content, only the honey pot is not enough.

2026-11-01

After the site is hacked, should the cleaning costs and loss of business be calculated together?

The cost of cleaning after being hacked is quoted per occurrence, the loss of business is accumulated over time, and the two are different in terms of scope but must be calculated together: reporting only the cleaning fee will underestimate the cost of the event, as it does not include the recovery of search visibility, order loss, and subsequent reinforcement hours. The order of handling is to isolate first, clean next, and then check the entry, and after recovery, use backups and recycle bins to verify whether it has returned to a clean state.

2026-11-01

The domestic website building system announced the repair of remote screenshot risks, what should the site without an upgrade do first.

PbootCMS has strengthened the server-side request forgery protection for the remote capture feature of the editor in V3.2.25, added connection timeout, size limit, re-binding protection, and internal network address blocking, and supplemented the configuration of trusted proxies. For sites that cannot be upgraded temporarily, they should first collect background permissions and server out-of-network, then make a backup, clean up the cache after the upgrade, and check the pseudo-static rules, rather than just wait for automatic updates.

2026-11-01

Most of the plugin vulnerability weekly reports can be triggered without logging in, how to filter out the ones in use.

The plugin vulnerability weekly report adds hundreds of new items in a week, among which a considerable number can be triggered without login. To filter out the components in use, the sequence is to first establish a component ledger, then filter by whether in use and whether login-free, which can reach two levels of filtering, and finally sort by severity. This article provides the screening method and disposal steps from the weekly report to the personal list.

2026-11-01