First look at the crawling log to decide who to block, and find out which paths the proxy is accessing.
How do you decide which AI agents to block before, how to use the crawl logs first to see which paths they are visiting? The answer is to first count and then draw conclusions: group the server crawl logs by agent identifier and access path, count which pages each agent has read, and then go back to robots to decide who to block and who to allow. The llms.txt placed on most sites almost has no traffic, and the real traffic is concentrated in only a few paths. Copying others' blocking list often blocks the wrong objects and wastes bandwidth; writing rules based on the real distribution, changes are smaller, and it is easier to roll back and review errors.
First look at the logs and then decide on the strategy
Interception is a bandwidth and security decision, the premise is to know who is reading and what is being read, to draw conclusions without logs is equal to setting rules based on impression. The division of labor between robots and llms.txt is different: robots control whether compliant agents can crawl and which paths they can crawl, which is a universally observed entry statement.llms.txt is a summary for the large model's site, listing the pages recommended to read first, affecting only the models willing to cooperate. Both complement each other and should not be written randomly based on feelings.First, see the real distribution in the log, and then decide which robots to place and which to block, and whether llms.txt should continue to point to these paths, the order cannot be reversed.AnQiCMS provides robots configuration and llms.txt auto-generation, but whether to allow it still depends on the log data.
The volume of visits within the day's aperture
The real number of visits often goes against intuition, as the two measurements reviewed on the day show this. Ahrefs analyzed the server logs and real-time traffic of about one hundred and thirty thousand domain names and found that 97% of the llms.txt files had no access in May 2026, not a single byte was taken.The top performers are concentrated in training and coding agents, ahead of various AI search and assistant agents. SE Ranking only saw 10.13% of the nearly three million domain samples with llms.txt, and statistical analysis and machine learning both show that llms.txt has no visible impact on the frequency of domain references by large models.In summary: The placement rate is low, and the actual access is even lower, blocking off an entire category of agents for just a few barely read entries, the profit does not outweigh the cost.
The minimum aperture of the statistics
From logs to rules, there is a group statistics that can stand on its own, with four dimensions of minimum aperture. First, group by proxy identifier, and see how much each self-named one read; then group by access path, and find out the few entry points that are repeatedly read; next, look at the category of response results, distinguishing between success, rejection, and throttling, to determine whether a certain proxy has already been blocked; finally, look at the byte range of each transmission, to estimate the bandwidth consumed by various proxies. It is only through the intersection of four dimensions that we can see who is reading, where they are reading, and how deep they are reading, and on this basis, determine the selection of robots and llms.txt.
| Statistical dimension | Record content | Purpose |
|---|---|---|
| Proxy identifier | The name and version string claimed by the crawling party | Distinguish between training class, code class, and assistant class |
| Access path | The requested page or file location | Find the few entry points that are read centrally |
| Response result | Return categories for success, rejection, or throttling | Determine if a certain type of proxy has been blocked |
| Transmission bytes | The size category of a single response | Estimate the bandwidth consumed by various agents |
The four dimensions correspond to the four steps mentioned above, and if a column is missing, return based on intuition
Interface and backend are handled separately
The front-end article page can be controlled declaratively with robots and llms.txt, the interface path and back-end entry should be handled separately. In terms of security, the back-end login page and management interface should not appear in the list given to the model, and any proxy probing is unwelcome. It is advisable to do access control and throttling on the server side rather than relying solely on robots.On the cost side, the interface is quickly consuming bandwidth due to high-frequency batch reading, and thresholds should be set according to the proxy identifier, triggering rate limiting and returning recognizable error categories before checking the log.robots are just suggestions for unruly agents, they can't stop them from being stubborn, the defense line of interface and background must be on the server side.
Common questions
Q: Should we modify robots.txt first or check the crawling log first? A: Check the log first, count the real distribution of agents and paths, and then decide who to block and who to allow, don't copy the list.
Q: llms.txt has been placed but almost no one reads it, should it be deleted? A: Placing is not equal to referencing, check the log to see if any agents have read it, and then decide whether to keep, converge entries, or remove.
Ask: Should the back-end and interface also be handed over to robots to block? Answer: It cannot rely solely on robots, the interface and back-end should rely on the access control and throttling of the service end, robots cannot block the hard grasp.
Ask: How to judge whether a certain proxy is worth allowing through? Answer: According to the statistics of four dimensions, look at which paths, byte levels and frequency it reads, and weigh the cost of bandwidth and exposure to make a decision.