AI crawlers are not just the leading few, should the budget for crawling be redistributed?
AI crawlers are not just the big players at the top, do we need to redistribute the budget for crawling? The answer is: we need to redistribute the traffic structure of this year all over again. The industry statistics checked on the day show that the share of AI crawling agents outside the top has increased from 2.34% a year ago to 12.28% in Q3 2026, which is about five times higher;There are 67 entries of crawlers collected in the same statistics. That is to say, only write robots.txt for the old faces such as Googlebot, Bingbot, and the remaining long-tail agents are either all allowed or all blocked, neither of which is the strategy you originally intended.
Where does the share outside the top fall to several types of agents
Based on the distribution verified today, Googlebot accounts for 23.80%, Meta-ExternalAgent accounts for 13.28%, ClaudeBot accounts for 13.23%, with the remaining being dozens of agents with low request volume but scattered sources.These long-tail agents share the following common characteristics: small number of requests per time, frequent updates of UA names, mixed usage, including those used for search enhancement and training corpus collection.In the batch statistics, there are still 7.9% of AI crawling requests that directly encountered 403 Forbidden, indicating that a large number of site interdictions are blocked entirely rather than differentiated by purpose.
The meaning of grouping is here: it is unrealistic to list the 67 UAs one by one, grouping by purpose is sustainable. Common methods are search type, agent type, and training type, the UAs in the same group are processed with the same rule block, and new UAs only need to decide which group to belong to.
How to arrange the grouping rules of robots.txt
robots.txt controls access permissions, the order of the rules determines the scope of effectiveness. It is recommended to organize in the following order, first closing and then opening up:
| Crawler Group | The desired effect | Processing in robots.txt | Note |
|---|---|---|---|
| Search type | Hope to be cited | Allow all sites to pass, only blocking the background and result pages | This type of proxy brings traceable traffic |
| Agent class | Allow reading the text but not interacting | Allow document path, block form and interface path | High request volume but single purpose |
| Training class | Decide by business | Block rules are unified | The strategy is clear after separating from search-related |
| Unknown UA | Default conservative | Only allow confirmed public paths | Avoid adding new proxies and default to full release |
There are three easy pitfalls to watch out for when writing: make sure to write the family name and UA in one go, any omission will fall back to the default rule; the background path should be precise to the prefix, do not use wildcards to scan the entire site; after making changes, check the priority between rule blocks again to avoid the same UA being covered by two rules.
Which part of robots.txt and llms.txt manages
Both are not substitute relationships. robots.txt determines whether and which paths the spider can enter; llms.txt determines what the model sees first after entering, which is more like a site index prepared for large language models. Enabling llms.txt automatic generation within the site is to make it easier for the allowed content to be correctly understood, but it does not change the crawling permissions themselves.
Therefore, adjust the order of budget allocation for crawling: first, clarify the boundaries of the family on robots.txt, and then decide which plates to list in llms.txt. If you block all training classes first and then pile the content list in llms.txt, it is equivalent to guiding agents that cannot come in.
How to verify after the modification
Verify in three steps. First, check if the rules are hit, use logs to count requests and status codes by user agent family over a period of time, and confirm that no desired family is mistakenly injured. Second, check if the robots configuration in the background is consistent with the online version, to avoid template files overriding manual rules. Third, review the collection and citation status of the pages that have been crawled, as the effect of reallocating the crawling budget usually needs to run for a full statistical period to judge, and don't just look at the data of the day the change was made.
Common questions
Can long-tail crawlers be added one by one to the robots.txt?Can be added, but not sustainable. The list checked today already has 67 entries, and UA will be renamed with the proxy iteration. Group by family and leave a conservative fallback rule for unknown UAs, which is easier to maintain than naming each one individually.
Will it affect the AI reference if the training class is blocked?It will be confusing. Search and training are different purpose proxies, and writing them in the same rule block will result in the situation where you want to reference but both are blocked. Group first, then decide the selection of training classes according to business.
Is 403 a good way to intercept?It is usually not. A large number of requests directly returning 403 indicates that blocking is completely sealed, lacking differentiation by path and purpose. The goal of crawling budget is to spend it where it should be spent, not to reject all unexpected requests.
How often should it be redistributed?It is recommended to review the traffic structure quarterly. The share outside the head will increase from 2% to two digits within a year, and if not checked for half a year, the rules will obviously lag behind the actual proxy distribution.