So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Will AI crawlers comply with robots.txt, and how many sites have already blocked them according to actual testing?

📅 2026-11-01 👁️ 0

Does the AI crawler comply with robots.txt, how many sites have blocked them? The measured proportion from the public dataset is: among 261 readable sites, covering 15 types of crawlers, 91 (35%) have blocked at least one type of training crawler, and 56 (21%) have blocked at least one type of retrieval crawler. This number indicates two things - the sites configured with robots.txt are still a minority, and it cannot be assumed that 'writing rules equals blocking.'

What does robots.txt actually control?

robots.txt declares the crawl permissions, not the access control. It tells compliant crawlers which paths not to crawl, but it does not prevent anyone from directly requesting a URL. Placing backend addresses, preview links, or non-public documents on the site and hoping to use a single Disallow line to protect it is the most common misuse: crawling is suppressed, but access is still possible.

Its second limitation is the

What can and cannot be explained by the actual caliber

The above set of numbers statistics is the site's defense configuration ratio, covering both training and retrieval crawlers, with the sample being the limited range of readable sites. It supports the conclusion that 'most sites have not made any distinction for AI crawlers', but does not support the conclusion that 'a certain model will definitely or definitely not comply'.

The compliance situation itself is divided into different types of web crawlers: those aimed at retrieval and require returning real-time links to users are usually more motivated to comply with crawling instructions; pure training collection crawlers are more uncertain about compliance as they do not need to maintain long-term access relationships. Treating these two types as one category for configuration can easily lead to incorrect access decisions.

Web crawler intent Whether it depends on long-term access Configuration suggestions
Retrieval and answer reference Dependency, loses source if blocked Generally comply with crawling instructions
Model training collection Weak dependency Explicitly list if needed to block
Differenceless bulk collection None Directly block and implement frequency control

分工 of robots.txt and llms.txt

The problems solved by both are different. robots.txt handles whether to crawl and which paths not to crawl; llms.txt handles how to understand the site when the model is already on your site—it is more like a site description written for a large model, listing what the site is, where the main content entry is located.

AnQiCMS supports automatic generation of the site's llms.txt, and also provides backend Robots.txt configuration capabilities. Both files do not require manual maintenance. It should be noted that they cannot be substituted for each other: writing only llms.txt will not prevent any crawling; writing only robots.txt will not make the model more accurately describe your site.

How to change and verify the backend configuration?

Configure robots rules by writing in the order of 'allow first, then restrict', to avoid a broad rule blocking the desired entries. After modification, it is necessary to verify three things: the file can be accessed directly through a public address, the effective content is the version just saved (note that the cache layer may still be returning the old file), and the performance of the critical path meets expectations.

Verify that evidence is left. The access frequency of different crawlers can be seen in the log, which is the most direct method to determine whether the rules have been executed. The security layer should also be considered: AnQiCMS built-in JWT authentication, content sensitive word filtering and anti-crawling interference code, used to prevent SQL injection and XSS attacks, blocking crawling does not equal site security, the two things are often confused.

Common questions

Question: Will blocking AI crawlers also result in a loss of search traffic?Answer: Distinguish types. Blocking search spider may prevent the answer engine from referencing your page; it will not directly affect normal search inclusion for training collection. Listing rules by type is more stable than a one-size-fits-all approach.

Question: How long does it take for the changes to robots.txt to take effect?Answer: It depends on the time of the next crawl of the corresponding crawler, the file itself can be accessed immediately, but indexing and crawling behavior will not change immediately. Allow an observation window before judging the effect.

Question: How do I know which crawlers have visited?Answer: Check the statistics of independent user agents in the access log, compare the frequency of access with the crawling path. It is easy to miss collectors not on the list of a single tool's report.

Related articles

Is it a cloud server or hosted website building, how to calculate the first-year deployment bill for the official website of a small and medium-sized enterprise?

The deployment invoice for the first year should be calculated based on three items: host, domain and certificate, and operation and maintenance time, rather than just the monthly fee. The cost of building a website on the day it was confirmed shows that the shared host is $3 to $15 per month, and hosting a website combines the host and maintenance into the subscription. This article provides two cost structures for two routes and explains how to choose between the Baota, aaPanel, and Docker deployment methods for AnQiCMS.

2026-11-01

Which part of the content does AI pay more attention to when citing a page, and should the answer in GEO writing be compressed in the first 200 words?

The answer unit of the generative engine is a paragraph instead of a whole page, so for GEO writing, the conclusions that can be directly cited should be placed at the beginning of the text. This article explains the magnitude of this change using the AI search traffic volume of the day, and provides the style of the first 200 words, the review position of AI automatic writing, and the role of llms.txt in the link.

2026-11-01

For large-scale utilization activities of content management systems, what are the first three things small and medium-sized enterprises should do?

The massive exploitation activity bulletin for content management systems named multiple common platforms, with the attack focusing on unauthorized uploads and remote code execution. Small and medium-sized enterprises have limited resources, so they should do three things first: incorporate patches into a fixed schedule, prepare recoverable trusted backups, and replace the entire station with sensitive word filtering to clear residues and unify the external expression, rather than doing expensive reinforcement procurement first.

2026-11-01

What does the field whitelist and parameter reflection repair in the PbootCMS update log mean?

PbootCMS update log shows three types of fixes continuously: Lack of whitelist constraints on backend user fields, page parameters reflected into HTML cache causing code execution, and ORM changed to parameter binding. They point to the same conclusion - most content management system accidents do not come from complex algorithms, but from the lack of full input constraints and output escaping at the most basic level.

2026-11-01

What order should be followed after the website is found to be planted with backdoor scripts, for cleaning, rollback and reinforcement

After discovering the backdoor script, the order of handling is not to delete the file first. The security agency's notice gives two principles: immediately patch the affected platforms and extensions, and recover from a known good backup. This article gives a list of executable steps according to four steps: confirming the extent of the intrusion, patching first, rolling back according to a trusted backup, and reinforcing after the event, and explains why local cleaning keeps failing repeatedly.

2026-11-01

How should static site generators and dynamic CMS be divided, where is the boundary of the hybrid architecture?

Static site generator solves "The published pages need to be fast and save", dynamic CMS solves "Collaboration, authorization and structured content". This article outlines the differences in the operation models of the two types of tools, gives three ways to staticize marketing pages and dynamicize core stations, and explains what pseudo-static rules and custom content models undertake in the hybrid architecture.

2026-11-01

The importance of brand names and entity mentions in AI search is increasing, should the keyword strategy change?

The answer engine relies more on entities and independently citable paragraphs, but this does not mean giving up the keyword library. The statistics on the day show that 86.7% of commercial intent searches appear with an AI summary layer, and the share of generative traffic is also being redistributed. This article provides the division of labor between the keyword library and entity expression, as well as the role of AI Chat, MCP, and llms.txt in maintaining consistency in external statements.

2026-11-01

Where does the difference in running costs lie between Go and PHP on the same server that runs a content management system?

Running a content management system on the same server, the cost difference between Go and PHP mainly comes from the process model rather than the language itself. This article explains the differences between the resident process model and the request-rebuilding model in terms of memory, concurrency, and cold start, and provides the resource calculation steps for selection, and explains how to understand the official statement that memory is reduced by about 80% and single-machine about 5 million PV.

2026-11-01