So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Pages with ads are blocked by default for training crawler, what configuration line should the new domain check?

📅 2026-11-01 👁️ 0

Cloudflare's new domain name defaults to blocking training and proxy crawlers on pages with ads, what configuration should the webmaster check? It is the default value that determines which layer it acts on: officially, from September 15th, new domains added will block training and proxy crawlers by default on pages displaying ads, while search crawlers are still allowed. That is to say, this configuration line manages the case where the three conditions of 'new domain + page with ads + training and proxy crawlers' are all met, not a site-wide cut-off.

The default value controls which domains

There are two thresholds for the scope of application. One is domain ownership: this default value is for new domains connected, domains already in use before this date will not be covered by the same rule. The other is page type: the criterion is whether the page displays ads, not whether the entire site displays ads.

According to this scale, when verifying, the site should be divided into three categories for separate processing: landing pages for advertising placements, content pages without advertisements, and intermediate pages that are neither search-oriented nor transaction-oriented. The conclusions of the three types of pages are different, and it is easy to make a vague judgment such as 'the site has been blocked' or 'the site has not been blocked' when mixed together.

View the scope by page type

Page Type Training class Agent class Search type What needs to be verified
Pages displaying advertisements Default block Default block Pass Whether you want to be cited by AI search
Content pages without advertisements By site settings By site settings Pass Whether the site-level settings cover here
Intermediate and aggregation pages Based on whether the page has advertisements Based on whether the page has advertisements Pass Is it mistakenly judged as an ad page

Blocking training class is not equal to blocking retrieval class, this is the most easily misunderstood point. From the perspective of the industry as a whole, the proportion of sites that clearly block retrieval spiders is very low, and the proportion of restricting training class crawling is closer to a little more than one; confusing the two is easy to overdo the defense in configuration.

Check the two ways to confirm whether the crawling is blocked

Check the configuration file itself. robots.txt is the access control layer, the training class and the retrieval class often correspond to different user agent names, check them by name rather than by category, because the identification methods of each family in the same category are not uniform. After changing the configuration, confirm that the actual output of the server is consistent with the expected output, and some access layers may rewrite or merge rules.

Check the access log on the second path. The log aggregates by user agent, showing which types of visitors are actually present: whether retrieval types have come, whether training types have been rejected, and whether the rejection is due to return status or explicitly prohibited in the configuration. The reason to rely on the log is straightforward; the configuration text expresses intent, while the log records results; when they are inconsistent, the impact is judged based on the results.

Configuration division and site-side implementation

This layer of division can be remembered as: robots configuration determines which requests can come in, the site map determines the path pages are discovered, and llms.txt determines how the site structure is read after the model comes in. The three are not mutually exclusive, the default intercept change is the first item, and the latter two still need to be ensured to be effective.

AnQiCMS's site-side configuration collects these items in the background: Robots.txt can be configured directly in the background, Sitemap is automatically generated and updated with content, and the site also supports the automatic generation of llms.txt, which is convenient for large language models to understand the site content structure. After changing the default values, review the site map and llms.txt together, which is closer to the actual crawling results than changing robots alone.

Common questions

The site doesn't have any ads, does it mean there's no need to check?It still needs to be confirmed whether the page is judged to contain advertising. The template's conditionally loaded promotional units and event landing pages may trigger the judgment. When checking, sample the results according to the specific page URL.

Which item should be changed to be referenced by AI search?First check if the search category is allowed, and then see if the content structure can be read. Blocking the training category does not affect the search category crawling, but a chaotic page structure and the lack of self-contained paragraphs will still reduce the probability of reference.

How long will it take to see the effect after changing the configuration?Based on the cycle of crawling logs. Configuration changes take effect immediately, and the time for the crawler to re-read depends on each scheduler, usually it is more reliable to observe log aggregation on a daily basis.

Related articles

The big version at the end of the year comes with a new default theme, and before the upgrade, the test site should be scheduled according to which nodes.

The big version at the end of the year is released with the new default theme, the testing station schedule needs to be reversed according to the official nodes: in the test version phase, only compatibility detection is done, in the candidate version phase, the regression of four types of surfaces is completed, and on the day of the formal version, it is grayed out first and then fully released. This article gives the five-step order on the day of the upgrade, three options for the rollback path, and the check items for the pseudo-static rules and backup.

2026-11-01

The graphic captcha is bypassed by session replay, do we need to add another layer of form anti-spam?

The validation module did not completely invalidate the opened validation session, and the submitted forms may be replayed. This article explains where the replay bypass occurs, what honey pot fields and human verification services block, and provides three sets of acceptance comparisons after going online, as well as the position of sensitive word filtering and content review at the bottom layer.

2026-11-01

The module vulnerability announcement was exploited in the wild on the same day, and if there is an extension of the same type installed on the site, check which step first.

Module security announcement marking the use in the wild, the troubleshooting sequence is to first confirm whether the module is installed, then compare the affected version range, and finally check the business data according to the suggestions in the announcement. This article takes the access bypass announcement of the e-commerce module as an example, and gives the cross-checking method of the extended list and the three checks after the upgrade, and explains the role of the default closed operation at the whole site level in the troubleshooting.

2026-11-01

The site did not make a redirect mapping table, so why did the natural traffic drop by two or three tenths?

The main reason for the migration of the natural traffic in the revision is that the old addresses have not been matched one by one to the new addresses: external links and indexed pages pointing to invalid addresses, with the weight and requests falling empty. It is common to lose 20 to 30% of natural traffic when there is no clean redirection plan; the对照表 should list the old addresses to the new addresses one by one, chain redirection controlled within a few jumps, and old redirections retained for a long time.

2026-11-01

The sites that block training crawlers are almost no longer cited, can this data be implemented directly?

A study covering thousands of domain names shows that almost no references can be obtained by blocking the domains captured by training class. This article explains why this set of readings cannot be directly applied: sample scope, industry composition, and category mixing will all contaminate the results, and a minimum experimental design for small-scale comparative observations on your own site is given.

2026-11-01

The adoption rate statistics of two llms.txt files differ by more than a factor of two, so which one should be used to allocate work?

The adoption rate of llms.txt is the same, but the readings from different studies range from ten percent to nearly fifty percent, the gap mainly comes from the denominator rather than the conclusion. This article decomposes several statistics according to the sample size and what they can answer, and gives three indicators that are more important than the adoption rate: whether it has been read, whether the content is consistent with the site map, and whether the maintenance cost can be automated.

2026-11-01

Only about forty percent of the pages cited by AI overview are in the top ten, should the external links be adjusted?

A study covering over 800,000 search result pages shows that the proportion of pages cited by AI overview and ranked in the top ten has dropped from over 70% to about 40% in a year. This article explains why this set of data is not enough to overthrow the external link plan, what aspects of the external link side do not need to be changed, and the two things that can be done first, such as the keyword library, anchor text, and paragraph structure within the site.

2026-11-01

Two weeks after the update, the rankings are still fluctuating. Should we change the content in this round?

Completion of push does not equal stable ranking. This round of garbage update took two weeks to complete, and the official also indicated that recovery may take a month. This article gives the order of handling within two weeks after completion: first, fix the baseline and the timeline of self-change, and then judge whether to change the content according to three types of signals. How to minimize the scope of changes when batch adjustments are needed.

2026-11-01