After Cloudflare splits the AI crawler blocking into three tiers, why might it block Googlebot as well?

Cloudflare categorizes AI crawlers into three categories for different purposes: Search, Agent, and Training. There is a possibility that blocking the training category may also block Googlebot. Observations indicate that this situation has been in effect since September 15, 2026, due to the same crawler identifier potentially taking on multiple roles such as search and training, and it is not possible to block only one of them during category blocking.

What does each of the three categories manage?

The three levels correspond to the purpose of access rather than the company name. The Search level is for building indexes and crawling content, blocking it means giving up visibility in this search product. The Agent level is for the intelligent body traffic that fetches pages for users in real time, this type of access usually brings back references, blocking it means excluding yourself from the answer sources. The Training level is for crawling content for model training, which is the level that content providers are most willing to block.

The meaning of tiered management is to divide the 'one-size-fits-all AI blocking' into selectable actions, allowing the site to retain search and block training. The problem is that the tiers are not one-to-one with the actual crawler.

Where does the cascading block occur?

Cascading occurs at the 'label and usage do not match' level. Googlebot's main purpose is to search for indexes, but it also appears in training-related crawling behavior. When a crawler is classified into multiple purposes, it may be blocked as a whole according to the Training tier blocking rules, and the search side crawling will be affected accordingly.

Performance is usually gradual: first, you see a decrease in the crawling volume of some paths in the logs, and after a few days, the index and traffic change. By the time you notice, it's already past the troubleshooting window. This kind of impact is not an error, but 'less', so it's harder to detect.

Before turning on, do three steps of verification

First step, check the baseline rules of the site itself. The robots.txt configuration is the first gate on the site side, the crawler looks here first; first confirm the existing allow and block rules, and then decide what to add at the CDN layer to avoid contradictions between the two layers of rules. If there are also crawling settings separated by category or subdomain within the site, they should also be checked together.

Step two, establish a comparable baseline for the crawling log. Before enabling, record a period of crawling data, including the request volume and requested paths distinguished by the crawler identifier; after enabling, compare daily, focusing on the changes in the quantity of retrieval identifiers, rather than just looking at the total amount. At the same time, check the coverage and inclusion reports on the search engine side, as they can expose problems earlier.

Step three, retain the rollback path. Changes should be made in small steps: first, close the least impactful section, observe for several days, and then decide whether to tighten further; ensure that the rules can be restored to the previous configuration at any time without the need to rebuild. In the short term after enabling, any inclusion or traffic anomalies should be suspected to be new intercepts rather than content changes.

What else can be done on the site side?

Keep the control granularity as much as possible on your own side: Maintain the budget and priority through sitemap and push mechanism, actively push new content through Baidu and Bing to accelerate indexing, and reduce dependence on a single crawler channel. When it is necessary to observe for a long time, include the access situation of different purpose crawlers in the regular inspection, so that changes such as 'less traffic' can be seen in time, rather than waiting until the monthly report.'

Common questions

Question: Since there is a possibility of injury, isn't it better not to open the intercept?Answer: It depends on your tolerance for training volume. The more stable order is to first open the small-impact档, observe the retrieval side data, and then decide whether to open the training档, rather than turning everything off or on all at once.

Question: How can you determine if Googlebot is truly affected?Answer: Compare the crawling request volume before and after opening, and check the inclusion and coverage reports in the search engine console; if both places show a decrease, it is basically confirmed to be related to the blocking rules.

Question: Which one takes precedence, robots.txt or CDN layer rules?Answer: Spiders usually read robots.txt first, but whether to comply is up to the spider itself; the blocking at the CDN layer is an actual network block, with stronger effect. The two layers must be consistent, otherwise there will be a contradictory state such as 'allowed but blocked' or 'forbidden but accessible'.