So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The ratio difference between news websites and development tool websites in blocking AI crawlers is very high, whether it should be determined by industry or by site.

📅 2026-11-01 👁️ 0

The proportion of AI crawlers blocked by news stations is much higher than that of development tool stations. Should the blocking strategy be determined by industry or by site? By site, using industry as a reference. The root cause of the difference is not industry preference, but the relationship between content and revenue: content itself is something sold, and crawling will directly erode inventory; content is a funnel that leads people to the product, and crawling brings in visitors. The selection of the same batch of crawlers by the two types of sites is exactly opposite.

Align digital numbers by industry first

2026-10-30 Open the original text and verify this set of samples, which consists of 1,103 sites: In the news and media category, 61.0% blocked at least one AI crawler, while in the development tool category, only 2.9%; in the entire sample, the proportion of explicitly banning at least one AI crawler is 18.9%. The difference between the two industries is dozens of percentage points, with the same direction but a significant difference in amplitude.

These numbers can only show the distribution of motivations, but cannot be used as the conclusion of this website. When the composition of industries in the sample changes, the overall average follows; it is easy to make an opposite judgment when using the overall average of the entire station on some of your own columns.

Content is inventory and content is funnel

The income of inventory content comes from authorization and subscription, and after being taken away in bulk, the original text can be replaced elsewhere,拦 is to reduce loss. Funnel content depends on recommendation conversion, and the purpose is to read the page, be cited, and bring in traffic.拦 is to shut off the entrance.

The way to judge which type you belong to is very direct: ask whether there is any reason to come to this station after the content in this column is completely taken away. The answer is no, it belongs to inventory; the answer is that the user still needs an account, deployment, and support, it belongs to the funnel.

How to choose among the three types of sites

Corporate websites, foreign trade websites, and marketing websites usually belong to the funnel: product pages, documents, and case studies are part of the decision-making chain, and consulting can only occur after they are read. Government portals and information stations are closer to inventory and compliance constraints, with the focus on verifiable sources and content that is not altered. Mixed websites are more common, with inventory and funnel columns on the same domain, at which point the strategy should be at the column level.

Site type Content role Training class Search and real-time retrieval Main verification items
News and information Inventory Tighten Subject to authorization Authorized income, citation volume
Corporate websites and foreign trade websites Funnel Can tighten Pass Generative recommendation entry, consultation volume
Government portal Source responsibility Tighten Pass Crawling range, error page ratio
Development document site Funnel service Pass Pass Registration conversion after document entry

There is a natural advantage at this step: the configuration of the column granularity can be directly written into the crawling control. AnQi Content Management System (AnQiCMS) is applicable to small and medium-sized enterprise websites, marketing websites, government portals, and cross-border e-commerce stations. Multi-site management allows inventory sites and funnel sites to belong to different domains, and the robots configuration of each does not affect each other.

What to test after setting the strategy

First, check the coverage of the grouping in the robots file one by one: when a group of agents has multiple identifiers, only writing one will make the ratio of one family significantly deviate from others. Then look at the source of entry: after the release search and real-time retrieval, has the generative recommendation come up; after tightening the training class, has there been a corresponding change in server pressure and the amount of crawling?

The strategy is not finished after one time. The industry ratio is moving every year, and the conclusion of your own station should be retested quarterly, and do not continue to use the average of others indefinitely.

Common questions

What about using the industry average as the strategy for your own site? It can't be used directly. It only reflects the distribution of reasons, and mixed sites still need to be judged separately by column.

What should be done if there are both inventory and funnels in the same domain name? Configure the crawling range by column grouping, and it is easier to manage the two types of content by separating them into different path prefixes.

What indicators should be looked at after the release? Look at the entry volume and conversion of the generated recommendations, as only looking at the crawling volume will treat the collection as customer acquisition.

Related articles

Sites that have not written a robots file, is the crawler considered to allow or to block

When a site does not have a robots file, the crawler treats it as an allow rather than a block: the three states of missing file, empty file, and all allowed have the same semantics, all indicating that there is no guidance. Nearly two out of ten sites sampled that day did not have this file. When making up for it, write a clear list of the entire site range, background path, and site map entry, and verify it with crawling records after writing, not just by looking at whether the file exists.

2026-11-01

Only a few of the AI crawler requests are for real-time answers for users, is it worthwhile to block the entire site?

Among the requests of AI crawlers, only a few are for real-time retrieval of content for users, and it is not cost-effective to block the entire site: According to the distribution of purposes verified on that day, training accounts for the majority, while real-time retrieval only takes up a small proportion, but that part carries real users. A more stable approach is to group and handle by purpose in the robots.txt file, tighten training, allow retrieval, and use interference codes to control batch collection, and test the number of requests and return status before and after the change.

2026-11-01

Paragraphs with statistical numbers are more likely to be selected by AI, and which step of content scheduling needs to be changed?

Paragraphs with statistical numbers and quotes have higher visibility in AI answers, the adjustment needed in content scheduling is the paragraph writing style: one paragraph should answer only one question, key numbers and sources should appear in the same paragraph, and the title should be written according to the user's question. When the output volume is large, consistency is maintained by keyword libraries and automatic tags, not by manual editing and monitoring article by article.

2026-11-01

When promoting lightweight programs with less than 400 KB of code, what three factors should be considered when selecting?

Promoting lightweight programs with less than 400 KB of code, when selecting, in addition to the repository size, you also need to look at three items: the runtime resident memory and single-machine bearing capacity, the data table structure and the scope of functional coverage, and the deployment form and recovery cost. The code size only indicates the size of the release package, and when the functional gap needs to be filled with extensions, the saved volume will become a new risk area and maintenance hours.

2026-11-01

The pass rate of mobile experience indicators is nearly 10 percentage points lower than that of desktop, where does the difference lie?

The mobile experience index pass rate is nearly 10% lower than that of the desktop, the gap is concentrated in the loading link of the real network and mid-low configuration devices, and images and third-party scripts consume the most. What the program level can make up is the operating margin: memory usage reduced by about 80%, and the single machine can carry about 5 million PV. These official estimates determine whether the response is stable under high concurrency, change one and measure one, do not use laboratory data instead of real user data.

2026-11-01

The click-through rate of the AI overview results page is significantly lower, how much can the title and description save

The click-through rate on the result page with AI overview is significantly lower, the relevant range verified on the day is an average click-through rate low by 58%, but correlation does not equal causation: high-intent queries are more likely to appear in the overview. Rewriting the title and description can save the part that is quoted and clicked on, the direction is to change from keyword stacking to an independent extractable conclusion sentence style, and use a keyword library to ensure the consistency of the whole site.

2026-11-01

Is there still a chance to be referenced by AI if it hasn't entered the top ten pages of natural rankings?

The page did not rank in the top ten pages of natural search results, but still has the opportunity to be cited by generative answers: The data checked that day showed that only 17% of AI overviews cited came from the top ten pages of natural search results, with limited overlap between the two channels. The task is to write a paragraph that can stand independently as a conclusion, use fixed fields in the content model, and link the long-tail question pages with authoritative pages using anchor text, rather than just using ranking for acceptance.

2026-11-01

The content referenced by AI is generally more up-to-date, and for old articles, it's better to change the date or rewrite them.

The AI engine tends to reference fresh content, but the last modified time displayed on the page is not the same as the freshness of the facts. Only changing the date and not the conclusion will continue to be referenced by the expired judgment. First, check if the data, version, and conclusion are still valid, and then decide whether to supplement paragraphs, rewrite the entire article, or withdraw, and write the refresh rhythm into the schedule; make a backup before batch replacing fields to prevent accidentally harming other pages.

2026-11-01