So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Most llms.txt files have never been requested, and is it still worth maintaining them?

📅 2026-11-01 👁️ 0

Most llms.txt files are never requested. Is it still worth maintaining them? They are worth keeping, but the reason for maintenance needs to be changed: this file does not bring references on its own, its significance only exists when corresponding requests appear in the logs. Therefore, the correct question for whether to maintain or not is not whether the file is placed or not, but whether there are any readers of it in the access logs of your own site.

How are zero requests measured?

This analysis looks at server logs instead of page content, the sample is more than ten thousand sites, and the statistics are the number of times files are requested. The conclusion is that ninety-seven percent of the files have never been requested once, and a few sites that generate reads are concentrated in a very narrow batch, among which the common visitors are the crawling behaviors of several known robots.

This number is easy to read in reverse. It says that most of the files placed on the shelf are not taken, not that files that have been taken have no effect. It is useful to look at the two issues separately: whether they have been read, which is a distribution issue; and whether they have been cited after being read, which is a content matching issue.

In the same batch of log studies, there is also a consistent signal: the overlap between the AI overview reference sources and the top ten natural results has dropped significantly, indicating that the citation does not rely on 'location' nor on 'an additional file'.

Why does the crawler not read along the file?

There are three explanations that can be self-checked on the site side.

First is the entry sequence. Crawling usually discovers pages from site maps and existing links, with only a few behaviors proactively testing this file from the root directory, and most crawling paths will not read it first.

The scope of the file's role. It describes the site structure and content summary for navigation, but does not change whether it can be crawled. Whether it can be crawled is determined by the robots rules, which are different levels of configuration. Writing one does not replace the other.

The outdatedness brought by maintenance costs. The content has changed, the columns have been rearranged, but the paths and summaries in the file still retain the old ones. Readers will not adjust according to this even if they obtain them. Once such files are inconsistent with the actual situation of the site, their role turns negative.

In what cases is maintenance meaningful

The basis of judgment lies in the logs and can be handled in three categories.

In the log, reading is continuously appearing: it is worth maintaining, align the content with the actual site structure, and update synchronously when changing columns. There is never a log entry, but the site is mainly long text and technical content: just keep the automatically generated one, do not need to manually refine each entry, and put the working hours on whether the paragraph can stand alone as an answer. The site structure and content standards change frequently: solve consistency first, then talk about maintenance, otherwise, every time the site is changed, the file needs to be changed, and the profit is lower than the cost.

AnQiCMS's llms.txt is automatically generated with the site, changes are concentrated on whether it should always be consistent with the current category, and it does not need to be maintained separately as a duplicate. Duplicate maintenance is most likely to result in inconsistencies between the two versions.

The review is of the log, not the file

Writing the review action in four steps is very cost-effective.

Step 1, obtain access logs for a period of time, filter requests by file path, and confirm if there are read records and from which visitors. Step 2, check the release status of robots rules within the same time window, confirm that the pages that can be obtained are indeed accessible, and avoid the situation where files are written but the main text cannot be obtained. Step 3, compare the site column structure with the described paths in the file item by item, and correct the generation logic first if there are inconsistencies. Step 4, repeat the first three steps every one to two months, looking at the trend rather than the single result.

The value of doing this does not lie in 'keeping a file', but in putting the investment into a verifiable chain: logs can prove whether it has been read, and the content can prove whether it is usable after being read.

Common questions

Does zero requests mean that this file should be deleted?It cannot be pushed directly. This indicates that the file is not the main entry point found, deleting it will not make the situation worse, and keeping it will not incur additional crawling costs. The decision should be based on how much manual effort is required to maintain it.

Should a long file content be written specifically?There is no corresponding relationship between length and whether it is cited. The file description structure provides extractable paragraphs, and the quality of the paragraphs determines the opportunity to be cited.

The log does not have any read records, should we still care about robots?Yes. Robots decide whether they can crawl, which is unrelated to whether the file is read, the access rules for sensitive directories and background paths need to be checked separately.

Related articles

Only a little over 30% of the pages referenced in the AI overview come from the top ten, so should we still pay attention to the ranking?

A large-scale study shows that only about 30% of the pages referenced in AI overviews are also in the top ten natural search results, while this proportion was about 75% a year ago. The decline in overlap does not mean that the ranking has become ineffective: the ranking is still the entry point, but is no longer a condition for reference. The focus should be on which issues the reference hits, whether paragraphs can be extracted separately, and whether the keyword library and anchor text can stably identify the content.

2026-11-01

The garbage station update in September took nearly fourteen days to end, when is it suitable for review?

This round of garbage content update in September started in the late month and was marked completed in the early October, lasting for more than thirteen days. Before the update was completed, the ranking reading was the intermediate state within the rolling period, which cannot be taken as a conclusion; the time suitable for review should meet two conditions: the official panel is marked completed, and the data after the completion day returns to a stable interval. The waiting period should first check basic items such as robots and site maps.

2026-11-01

The window for the secure version and the major version release are scheduled in the same week, which one should the site follow first?

When the fixed secure version window and the annual major version are scheduled in the same week, follow the secure version first. The secure version closes known exploitable entries, while the major version changes data structures and template behaviors, with a higher verification cost. This article explains the differences in the nature of the two types of updates, how to evaluate the cost of delaying the major version by one week, and the two-step verification and rollback preparation to be reserved in the scheduling.

2026-11-01

Components that cannot be found in the vulnerability library, can they be considered risk-free?

A component that cannot be found in the public vulnerability library cannot be considered risk-free, as it is often just that the scope of collection and naming criteria do not match. The vulnerability library categorizes entries by core, extension, and theme, and verifies them one by one. Self-built and commercial components are often not included. This article provides three reasons for not finding them, the handling criteria for the worst-case scenario, and how to turn the asset list into an executable mitigation action sequence.

2026-11-01

The adoption rate statistics of the two llms.txt files are very different, which decision口径 should be adopted?

Two studies that were checked on the same day are very different: The adoption rate is 8.3% when statistics are taken from the top ten thousand domain names, and it reaches 49.2% when statistics are taken from more than two thousand sites. The difference does not come from who made a mistake in the calculation, but from the denominator range and the method of judgment. When making decisions, do not take the average of the two numbers, but first see which category your site falls into among the samples, and then confirm whether the口径 can be used by checking the three-step verification method.

2026-11-01

The training crawler was blocked in robots, why can't the content still be retrieved

There are tests from more than 2,300 sites showing that only 3.6% of the sites block training crawlers in robots, but when accessing the homepage directly by real identity, 18.8% cannot access usable content. There is a gap between declarations and actual usability: writing rules does not mean they take effect, and there are also cases where the side that grants access cannot access the text. Self-inspection should be in order of four types of signals, not just whether that line of rules is in the text.

2026-11-01

Which type of spam submissions do the honeytrap field and captcha each block, can only one be left

The honey pot field blocks script submissions that do not parse styles, there is no visible challenge for real visitors; the scoring captcha is scored according to the possibility of robots, at the cost of mistakenly harming some real users. The two methods do not face each other, and whether only one can stand depends on the composition of garbage: when the amount of scripts is mainly used, the honey pot is enough, but when there are submissions with changed addresses and natural content, only the honey pot is not enough.

2026-11-01

After the site is hacked, should the cleaning costs and loss of business be calculated together?

The cost of cleaning after being hacked is quoted per occurrence, the loss of business is accumulated over time, and the two are different in terms of scope but must be calculated together: reporting only the cleaning fee will underestimate the cost of the event, as it does not include the recovery of search visibility, order loss, and subsequent reinforcement hours. The order of handling is to isolate first, clean next, and then check the entry, and after recovery, use backups and recycle bins to verify whether it has returned to a clean state.

2026-11-01