So Outside Technology Security CMS
AnQiCMS
Online Demonstration

To what extent has the number of times the site blocking the training crawler has been cited by AI dropped?

📅 2026-11-01 👁️ 0

What level will the number of times a site is referenced by an AI engine drop after blocking training crawlers in robots? The research conducted that day covered more than a thousand domain names, providing two sets of numbers: one is the blocking rate, which is counted separately for different crawlers, with the highest level approaching 18%, another level around 15%, and another level around 14%; the other is the drop in citations, the average number of citations per page dropped from 0.417 to 0.003 after blocking the sites of retrieval crawlers, a gap of two orders of magnitude. On another engine in the same sample, the gap is almost halved rather than zeroed out.

The interception rate explains the distribution of behavior

These proportions read out what 'everyone is blocking what', not 'how much content is blocked'. They are listed by the name of the crawler, indicating that the site owner's configuration is differentiated—the same site may block crawlers for training purposes but allow crawlers for search purposes.

This is the premise for the establishment of hierarchical configuration. If the entire site is cut off at once, not only training purposes will be blocked, but also the reading channels on the search and summary sides will be closed simultaneously, with the cost directly reflected in the number of citations. The figures reviewed on the day are the consequences of this path.

How to read the citation gap

The scope of the number of citations is the average number of times a page is named in the engine's answers, unrelated to traffic. A difference of two orders of magnitude means: in the samples where the crawler is blocked, the vast majority of pages did not appear in the citation list at all, rather than 'appearing less.'

It is important to note the direction of relevance. In the sites where the crawler is blocked, there is already a batch that does not want the content to be crawled (such as news and publishing), and the low citations may be the result of blocking or the result of the content type. This number is suitable for estimating 'how big the difference between blocking and not blocking is', but not for estimating 'how much I will lose if I block it.'

Categorize by page type

Page Type Crawl suggestions Reason
Product pages, interfaces, and user manuals Pass The answer-type reference most likely to be extracted
Blog content and help center Pass The main share of the brand mentioned
Drafts, archives, and low-value lists Block Crawled without profit, may also be misquoted
User data and background path Block It should not be open to any reading party

The robots configuration manages who reads and which paths, this level needs to be checked item by item according to the classification above, not a global rule.

Interception and anti-crawling are not the same thing

The rules in robots.txt apply to the crawling stage, whether they are followed depends on the crawler itself; anti-crawling interference codes act on the copying stage, targeting the behavior of copying the entire content of the page. Both do not solve the same problem. Treating the interception of crawlers as content protection, it is usually not well done on both ends: those who want to cite are kept out, and those who want to copy the content can still get it by changing the entrance.

Similarly, llms.txt belongs to a layer of 'How to understand the site structure', which does not conflict with the crawling license. If it is decided to allow retrieval crawlers, this layer can be complete with coordination.

Common questions

Would blocking only training classes also affect the citation?It depends on the specific classification of the crawler's purpose, training and retrieval are often different identifiers, and they need to be configured separately by name in order to only block one side.

Has it been blocked, what is the cost of changing it back?The configuration change itself is quick, but there is a lag in being re-read and referenced, so it is necessary to observe for several weeks after the change, rather than looking at the conclusion on the day.

Would分级放行 be copied by the entire site?Copy risk depends on anti-crawling and permission control handling, and it is clearer to separate the crawling permission.

Related articles

After the release of LLMs.txt, ninety percent of the files were not requested, is this number looking at the access log?

Yes, these extremely low transaction rate numbers come from the site's own access logs: first, filter out the domains that have placed files, and then check the server logs for corresponding crawling requests. The two sets of criteria reviewed that day need to be read separately, one is the adoption rate, which measures how many domains have placed files; the other is the read rate, which measures whether there are requests after placement. The two cannot be mixed into one sentence.

2026-11-01

The total number of vulnerability libraries is increasing day by day, should the webmaster pay attention to the total amount or the increase?

The total number of responses is about whether the ecosystem is expanding or contracting, and the new responses are about whether there are any entries related to you today, both cannot be mutually exclusive. The executable approach is to filter the newly added entries using the component list, leaving only those related to this site and versions within the affected range, and then arrange them in order of whether they can be triggered anonymously. The list check can be exported in bulk via an interface, and there is no need to manually flip through pages.

2026-11-01

After AI capabilities entered CMS, why does the background also need to be managed as an attack surface?

The AI ability makes the background go from 'viewing content' to 'being able to act': one conversation can write libraries, change configurations, and adjust interfaces, making the login state and interface credentials high-value targets. The management method is to tighten three types of login-free interfaces separately, set exposure ranges according to ability domains, retain operation approvals, and pay attention to security changes at the version level. The two types of high-risk issues fixed in the current version exactly illustrate that the input on the backend side and the login tickets should be treated as attack surfaces.

2026-11-01

When connecting a large model, should it be done using a third-party module or the built-in capability, and where do the risks differ?

The risks of the two routes differ in who holds the key and permissions, and whether the entry point of the capability is controlled. The third-party module packages the model key, write permissions, and call links into the extension, and needs to be upgraded in bulk once the upstream issues an announcement; the built-in capability converges the entry point within the program, and can set the exposure range by domain and perform operation approval by round. The selection should consider the density of announcements and the boundary of permissions, not just whether this feature exists.

2026-11-01

AI referral sessions only account for 1% but have higher conversion rates, which two indicators should you believe first

Believe in the conversion rate first, but use it to test the reception ability, not directly as an expansion basis. The review on the day was: across more than ten thousand domain names, AI recommendation on average only accounts for about 1% of the total sessions, but the conversion rate of recommended visitors is much higher than that of natural search sources. The share index fluctuates greatly and is affected by the entry; the quality index is more suitable for long-term baseline, ensuring that a small number of highly interested visitors do not lose interest first.

2026-11-01

About one in ten sites that are directly rejected by the crawler, should we check robots or service configuration first?

A nearly 10% of the sites return a denial status code to the crawler, the troubleshooting sequence is to first check the server configuration rather than change the robots file: robots is just a polite guideline, a denial code means the intercept occurs at the service layer. Locate by grouping status codes for rules, firewall, or pseudo-static, and verify layer by layer, changing only one layer at a time, so that the robots file is changed but the denial code remains the same.

2026-11-01

The quality indicators of the same batch of sites differ by 30%, how much are hosting and website construction methods accounted for

The real user experience passing rate of different website building platforms can vary by 30%, and the platform data verified that day showed that the hosted website building platform is generally higher than the free website building program, and the full station baseline is only equal to the worst mainstream platform. On attribution, the weight of the website building method and the theme category is often greater than that of the hosting category, and it is better to measure the real user data first and then decide which side to move, so as not to waste the budget on the layer with small returns.

2026-11-01

Shared hosting costs as little as a few dollars per month, when is it necessary to switch to a virtual private server

Shared hosting's lowest-tier monthly fee is just a few dollars, do you need to switch to a dedicated virtual server without looking at the number of website pages, but rather at three resource curves: whether the response time is extended during peak hours, whether the resident memory is at the top, and whether the concurrent peak is queuing. The domain annual fee and hosting monthly fee should be calculated separately; when the program-side memory usage is low, the same type of load can maintain a lower tier.

2026-11-01