So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Only a few of the AI crawler requests are for real-time answers for users, is it worthwhile to block the entire site?

📅 2026-11-01 👁️ 0

In the requests of AI crawlers, only a few are for real-time retrieval of content for users, is it cost-effective to block all AI agents on the entire site? Not cost-effective. Blocking the entire site saves the cost of bulk collection, but the cost is to shut down a small part of the traffic with real users. Grouping by purpose is more stable than a one-cut approach: tighten training, allow retrieval and real-time retrieval, and then control collection with content layer means, each with a landing point for three purposes.

Firstly, separate the three types of use

The first type is training material collection, one-time bulk collection of corpus, almost no return. The second type is retrieval index, organizing pages into citable indexes, and subsequent visits will be accompanied by source links. The third type is to retrieve for users in real time, where users ask questions in generative products, and the program captures this page content on the spot. This type of request behind it is a visitor with a clear and real intention.

When the three types are mixed into an 'AI Spider' switch, any adjustment of the default value will change all three types at the same time, and webmasters cannot see what they have turned off. Grouping is the premise of all subsequent judgments.

The proportion of the day's口径

This distribution of usage on October 30, 2026, as verified by opening the original text, is: Training accounts for 47.62% of all AI crawler requests, search types doubled to 8.3%, while the part taken care of for users in real-time is only 3.66%.

The correct way to read this set of numbers is not 'Only 3.66% so it doesn't matter.' It indicates the structure: training is the majority, blocking it can indeed reduce nearly half of the material pressure; real-time retrieval is a small part, but it corresponds to incoming users, and the value is not calculated linearly based on the number of requests. The judgment of blocking or allowing should be based on usage rather than quantity.

Purpose Percentage for the day What is gained by stopping What is given up by stopping
Training to take material 47.62% Reduce the pressure on bulk material collection and server Less opportunity for the model to be cited
Retrieve index 8.3% Content does not enter the index that can be cited The referral traffic from generative answers is cut off
retrieve in real time 3.66% Unable to read this page immediately Turn off a high-intention entry exit

How to group by usage

The most convenient way to group robots files by user group is to write the training class identifiers in one segment, the search class in one segment, and the real-time retrieval class in one segment, each segment to be handled separately. Pay attention that the same group of agents often have multiple identifiers, and only writing one of them may result in 'changed but not effective'.

Grouping outside also requires handling the mandatory layer. Robots are polite guidelines, relying on the voluntary compliance of others; indeed, it is necessary to limit bulk collection, using interference codes and other content layer means for better control, which allows the collected content to be marked with meaningless tags, making it more visible to normal users than directly denying access. The goal of anti-crawling is to reduce the entire site from being moved, not to cut off all machine access, and the two goals correspond to two sets of means.

What to measure after blocking and releasing.

After tightening the training class, observe the changes in server pressure and the amount of crawling; after allowing retrieval and real-time retrieval, observe whether the generative recommendations in the station source have increased. Each set of indicators corresponds to a decision, and when mixed together, it is difficult to determine which layer is at play.

Change one layer at a time: first change the grouping rules and test once; then adjust the content layer methods and test again. If two layers are changed at the same time, the next round will still rely on guessing when encountering the same problem.

Common questions

Question: If real-time retrieval only occupies a small proportion, will it have a big impact if it is turned off? Answer: The impact is not based on the proportion but on the intention. The requests behind this part are users who are looking for answers, and turning it off is equivalent to giving up this type of entry.

Ask: Can robots alone stop the training class? Answer: Cannot guarantee. Guidance relies on voluntary compliance, and forced handling requires cooperation with the interference codes of the rule and content layers.

How can you judge if the grouping of your own station is missing anything? Answer: Check the request volume and return status item by item according to the manufacturer. When one is obviously deviating from the others, it is usually because its identifier has not been written into the rules.

Related articles

Paragraphs with statistical numbers are more likely to be selected by AI, and which step of content scheduling needs to be changed?

Paragraphs with statistical numbers and quotes have higher visibility in AI answers, the adjustment needed in content scheduling is the paragraph writing style: one paragraph should answer only one question, key numbers and sources should appear in the same paragraph, and the title should be written according to the user's question. When the output volume is large, consistency is maintained by keyword libraries and automatic tags, not by manual editing and monitoring article by article.

2026-11-01

When promoting lightweight programs with less than 400 KB of code, what three factors should be considered when selecting?

Promoting lightweight programs with less than 400 KB of code, when selecting, in addition to the repository size, you also need to look at three items: the runtime resident memory and single-machine bearing capacity, the data table structure and the scope of functional coverage, and the deployment form and recovery cost. The code size only indicates the size of the release package, and when the functional gap needs to be filled with extensions, the saved volume will become a new risk area and maintenance hours.

2026-11-01

Nine out of ten new vulnerability entries in a year are due to extensions rather than the core, how can the list prove itself?

In the tens of thousands of newly discovered vulnerabilities, ninety percent are in extension components rather than core programs. To ensure that the site's component list is clean, rely on three sets of accounts: what components are installed, the version of each component in use, and the source and maintenance status of each component. First straighten out the list before talking about patching ahead of time. The built-in SQL injection and XSS protection only covers the input layer and cannot replace component version verification.

2026-11-01

There are only branches of script languages for security repairs, so the operator needs to pay attention to which two dates?

After the script language branch enters the stage of only receiving security fixes, the site operator needs to keep an eye on two dates: the date of cessation of active support and the date of cessation of security fixes. The former means that functional defects will no longer be repaired, and the latter means that even critical security issues will no longer have patches. The period between the two dates is the migration window, and it is usually too late to start when the second date is approaching.

2026-11-01

Sites that have not written a robots file, is the crawler considered to allow or to block

When a site does not have a robots file, the crawler treats it as an allow rather than a block: the three states of missing file, empty file, and all allowed have the same semantics, all indicating that there is no guidance. Nearly two out of ten sites sampled that day did not have this file. When making up for it, write a clear list of the entire site range, background path, and site map entry, and verify it with crawling records after writing, not just by looking at whether the file exists.

2026-11-01

The ratio difference between news websites and development tool websites in blocking AI crawlers is very high, whether it should be determined by industry or by site.

The proportion of AI crawlers blocked on news stations is much higher than that on development tool stations, the difference comes from the relationship between content and income: the industry tends to block content as inventory, and to release content as a lead funnel. The industry average can only be used as a reference, not as a conclusion for this station. A more stable approach for self-built stations like Anqi CMS is to determine strategies at the column granularity, first judge whether this column is inventory or a funnel, and then configure and verify the entry sources separately.

2026-11-01

The pass rate of mobile experience indicators is nearly 10 percentage points lower than that of desktop, where does the difference lie?

The mobile experience index pass rate is nearly 10% lower than that of the desktop, the gap is concentrated in the loading link of the real network and mid-low configuration devices, and images and third-party scripts consume the most. What the program level can make up is the operating margin: memory usage reduced by about 80%, and the single machine can carry about 5 million PV. These official estimates determine whether the response is stable under high concurrency, change one and measure one, do not use laboratory data instead of real user data.

2026-11-01

The click-through rate of the AI overview results page is significantly lower, how much can the title and description save

The click-through rate on the result page with AI overview is significantly lower, the relevant range verified on the day is an average click-through rate low by 58%, but correlation does not equal causation: high-intent queries are more likely to appear in the overview. Rewriting the title and description can save the part that is quoted and clicked on, the direction is to change from keyword stacking to an independent extractable conclusion sentence style, and use a keyword library to ensure the consistency of the whole site.

2026-11-01