The new role of robots.txt in 2026: from search engine to control panel for AI spiders

What new role does 'robots.txt' play in the era of AI crawlers? In one sentence: It has transformed from a crawling instruction only for search engines into an access control panel that publicly declares 'which robots can read and which paths they can read.' The participants in the crawling process are no longer just a few search engines, but various large model robots will also read the pages, and this document remains the most comprehensive declaration method that the site can present. Below, let's look at it from the aspects of 'original function - new constraints - changes in matching rules - division of labor with llms.txt - how to configure on the backend.'

What problem did it originally solve

The earliest robots.txt only answered one question: what paths can the crawler request? The site uses User-agent to specify robots, uses Disallow to exclude directories that do not want to be crawled, and also coordinates with the sitemap to hand over the addresses to be included, so that new content can enter the index as soon as possible.This action inside the site is usually a fixed three things: robots configuration, sitemap automatic generation, and link push to actively push new addresses to search engines.robots.txt controls 'can enter', sitemap controls 'which door numbers', the two duties do not overlap.

What constraints have been added in the AI crawler era

The change lies in reading this file more objects. Mainstream AI crawlers read robots.txt before grabbing the page, reputable robots will comply with the instructions in the file, such as grabber programs identified by product tokens such as GPTBot, Google-Extended.This leads to two direct conclusions: First, the cost of Disallow writing incorrectly is magnified, originally just missing one search inclusion, now it also loses the visibility on the model side.The second is to limit the capture of a certain type, adding explanations on the page is not useful, you need to write rules according to the product token in robots.txt so that the rules can be read by machines.

It needs to be clarified: whether to comply or not depends on whether the crawling party is willing. This document is a declaration rather than a mandatory barrier. The truly needed to be protected back-end paths, internal documents, still need to be blocked by permissions and authentication, not just rely on one line of Disallow.

The easiest place to make a mistake in the matching rules

The crawling protocol (RFC 9309) specifies two separate rules for case sensitivity: when searching for a User-agent's corresponding rule set, it should match in a case-insensitive manner; while determining if the path itself matches, it should be case-sensitive. In other words, /Blog/ and /blog/ are two different rules, and when the server distinguishes between case, one of them may not be effective at all.

The order should be understood according to the protocol when there is a rule conflict: if the same address is covered by both allow and disallow and the degree of match is similar, priority should be given to allow. Common debugging methods include writing broad directory exclusions at the beginning, followed by specific paths that need to be allowed, comparing them one by one to determine which one takes effect ultimately, rather than guessing by intuition.

The division of labor between llms.txt and robots.txt

Two files are often compared together, but they manage different levels. robots.txt decides 'whether to crawl', llms.txt decides 'how to read after crawling' — it uses a plain text index to explain the site structure, key content, and boundaries, allowing the large language model to first understand the site context within a limited context.AnQiCMS supports automatic generation of site llms.txt, also supports robots.txt configuration and sitemap output, the three configurations should be checked together: directories excluded by robots.txt, there is no point in writing them into llms.txt either.Pages that only appear in llms.txt but are not included in the sitemap, the coverage is still missing.

It is recommended to follow this order when configuring the background.

First, look at the exclusion list: Include addresses such as backend, search results page, and shopping cart, which have no value for collection, in Disallow, and confirm that there is no injury to normal content directories. Then look at the whitelist: Keep the columns that need to be read by search engines and models open, and declare the address of the site map in robots.txt, so that the crawling can automatically find the entry. Finally, look at the push: Push the new content actively to the search engine using links to avoid only waiting to be discovered passively. After making changes, do a regression check - directly request this file and confirm that the latest content is returned rather than a cache or error page.

Common questions

Block AI crawlers but not search engines, how should it be written?Group rules by product token, for example, only set Disallow for the corresponding crawl token, and keep the search engine group as is. After changing, check whether the grouping really hits, note that the matching of grouping names does not distinguish between uppercase and lowercase, while path matching does.

Will the content excluded by robots.txt not be referenced by the model?Reduce the probability, but it does not mean to cut off. This document relies on the voluntary compliance of the crawling party, and other entries (aggregation stations, third-party mirrors) are still being read. To control visibility, it still needs to be coordinated with permission and content offline.

Can llms.txt replace robots.txt?No. The former is the reading index for the model, and the latter is the access statement, which serve different layers and need to be configured and checked against each other.