Only about 10% of sites have installed llms.txt, can this number indicate that it is useless?
Only about 10% of sites have installed llms.txt, can this adoption rate number prove that it is useless? No. The adoption rate describes 'how many sites have done this', and the effect describes 'whether the probability of being cited after doing it has changed', which are two independent evidence chains. The numbers from the two studies of the same period are 10% up and down and 9.3%, both of which can only show that it has not yet become a common practice.
What does the adoption rate number indicate?
A study was conducted on nearly 300,000 domain names, with only 10.13% hosting llms.txt; another study, by top-level domain, showed that 9.3% of the first thousand domains provided the file, dropping to 8.3% among the first ten thousand, with significant differences between industries, with travel sites reaching a high of 16.0%.
This set of numbers can be concluded: First, the absolute proportion is low, and second, the distribution is extremely uneven. The low adoption rate may come from cognitive lag or deployment cost, and it has no direct relationship with whether the tool is useful. Considering 'few people use it' as 'useless' is equivalent to using popularity as a substitute for effectiveness evaluation.
How should one read when the correlation is not obvious?
Another conclusion from the same batch of data is: there is no correlation between the existence of llms.txt and the number of times it is cited by AI. Here are three points to pay attention to.
The first, the sample is observational, not a controlled experiment, the content structure, weight, and theme of the site that installed the files are all different, and the confounding factors have not been eliminated. The second, being read does not mean being cited, citation also depends on the search hit and the form of the answer, and the contribution of the reading link is easy to be invisible in statistics. The third, 'no relevance' does not mean 'no effect', nor does it mean harmful, it only means that the difference cannot be measured with the existing口径.
A more practical way to judge: treat it as a reading path
Move llms.txt to the 'Model Understanding Site Structure' layer, not the 'Improve Ranking' layer. Its function is similar to a directory for the reader: what sections does the site have, what are the key pages, and how to enter. AnQiCMS will automatically generate the site LLMs.txt, and editors do not need to manually maintain this summary; this step solves the reading path issue.
Access control belongs to another matter: which crawlers can enter and which sections are not readable, is determined by the robots rules and the site's own permissions. Discussing both in one file can create the illusion of 'installing without effect' - because the reader never gets to that step.
Cross-verify with logs and sitemap
To find out if it has been read, look for three clues:
- Whether there is a corresponding crawling request in the server log, what is the crawling frequency and the accessed file path;
- In the automatically generated collection range of the sitemap, whether this file is exposed together with other key pages, or blocked outside by the rules;
- After crawling, did it extend to the main content page? Hitting a single file and reading multiple articles following the links are two completely different effects.
The site map is responsible for being 'found', llms.txt is responsible for being 'understood', and both need to be cross-verified to locate where the problem is stuck. If there are no requests in the logs, the problem is not with the file content, but with the accessibility and the path to be discovered.
Common questions
Should we install it now?If the site structure is complex and the sections are numerous, the cost is very low; if the basic inclusion has not been solved, solve the inclusion and site map first, then discuss the reading path.
Will it be considered cheating if installed?It is a document for the reader, not part of the manipulation result, the judgment standard is whether the content is consistent with the actual page.
How often is it reviewed?The site section is re-generated when there are additions or deletions, trends are observed monthly with logs, and fluctuations on a weekly basis should not be concluded accordingly.
Can it be deleted if it has no effect?It can be kept for observation first, and before deletion, confirm that there are no corresponding requests in the logs to avoid misjudgment as 'deleted is better'.