So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The adoption rate statistics of the two llms.txt files are very different, which decision口径 should be adopted?

📅 2026-11-01 👁️ 0

Two llms.txt adoption rate statistics are very different, which standard should be used for decision-making? None of them can be used directly. Both numbers are correct, but the denominators are different: one is sampled by the head domain, and the other is counted by the sites included in the dataset. The task is not to take the average value, but to judge which type of sample your site is closer to, and then check the standard in three steps to decide whether it can be used as a basis.

How do the two numbers come from

The statistics of the top-level domain names, taking the top one thousand and ten thousand domain names in terms of traffic ranking, with adoption rates of 9.3% and 8.3% respectively. The tourism industry has the highest adoption rate, reaching 16.0%. The statistics of the publicly available dataset cover more than two thousand three hundred sites, with an adoption rate of 49.2%.

More than 60% of the difference between the two comes from the sample range. The top-level domain names are dominated by large companies, with a long process for configuring new files, more legal and brand audits; the site structure in the dataset is different, with a higher proportion of small and medium-sized content sites, and the configuration actions are lighter.

The same dataset also provides two aspects of crawling: 3.6% of sites blocked training crawlers in robots, while 18.8% of sites could not obtain usable content when accessed directly as a real visitor. This indicates that the dataset samples themselves are more biased towards sites with observable crawling behavior, different from the sampling口径 according to rankings.

What does a different denominator bring?

The denominator determines who the ratio answers the question of.

The proportion of sampling from the top ranked, which answers the question 'How many large companies have done this thing', is suitable for judging whether the trend has been adopted by the mainstream. The proportion of sites included in the statistics answers the question 'How many of the sites willing to be included in this kind of dataset have done it', and the sample itself has self-selection bias: sites being included in the dataset often means that their crawling behavior is easier to observe.

A specific consequence is that averaging two numbers directly results in a value that does not represent either the head or the tail, while decision-making precisely requires knowing where one is on either end. When making technical choices, it is more useful to know 'how many people in the industry are doing it' than 'how many sites of the same scale and content type as mine are doing it.'

The second-level difference caused by the judgment method

In addition to the denominator, the judgment method will also widen the proportion.

The first level determines whether the path can be accessed. A file is considered published if it returns a normal status code, which includes temporary generated empty files and cached old versions by CDN. The second level determines whether the content is valid, requiring the file to have site structure and descriptions, with a significantly lower ratio than the first level.

The time of publication also affects the reading. The research in September was sampled early when the file appeared, and the dataset published in October was sampled after it was generated by default on multiple hosting platforms, making the two time points inherently unequal.

The table lists the differences between the two diameters, making it convenient to take numbers according to your own position.

Comparison items Header domain research Public dataset
Sample Range Top 10,000 domains in traffic ranking More than 2,300 sites
Adoption rate 9.3%, 8.3% 49.2%
Industry distribution Travel category leading with 16.0% Not split by industry
Sampling time point September Published on October 20th
Appropriate question Whether the leading company has started to configure Configuration of sites of the same scale

Three-step check of third-party statistics

Step one, look at the denominator. Check if the sample is drawn according to ranking, industry, or crawlability, and if the sampling method is different from the type of your site, the ratio cannot be directly cited.

Step two, look at the judgment. Confirm whether the condition of 'considered as adopted' is path accessibility or content validity, as the two studies use different judgments, the numbers have no comparability.

Step three, look at the time point. This type of configuration will jump after being generated by default on the hosting platform. The readings before the jump point will underestimate the current situation, and when taking data, the one with the update date closest to the present should be prioritized.

If you still cannot determine it after completing the three steps, use indicators that you can verify yourself instead of industry ratios: take the number of log reads on the site side and the consistency of files and column structure as the basis for judgment, as they are not affected by others' sample口径.

Common questions

The difference between the two numbers is so great, is there a mistake in one of the calculations?Cannot judge this way. The denominators, criteria, and sampling points of the two are different, and each is internally consistent, the problem lies in comparing them as the same standard.

Which plan should I invest in?According to the category of your site: content type, sites with a scale in the middle segment and below, the reference dataset standard is more reflective of peers; only look at the domain name standard when facing competition from leading brands.

Can I use both?Yes, but it should be stated separately for use: one to explain whether the trend is established, and the other to explain the actual configuration rate of sites of the same scale, do not mix them in the same conclusion.

Related articles

Most llms.txt files have never been requested, and is it still worth maintaining them?

The analysis of logs from more than 100,000 sites shows that 97% of the llms.txt files have never been requested, with reading concentrated in a few sites. This result does not mean that the file is useless, but rather that it does not solve the two issues of being discovered and intercepted. Whether to continue maintaining it should be judged by looking at whether there are corresponding requests in the access logs of your own site, rather than by looking at whether the file has been uploaded.

2026-11-01

Only a little over 30% of the pages referenced in the AI overview come from the top ten, so should we still pay attention to the ranking?

A large-scale study shows that only about 30% of the pages referenced in AI overviews are also in the top ten natural search results, while this proportion was about 75% a year ago. The decline in overlap does not mean that the ranking has become ineffective: the ranking is still the entry point, but is no longer a condition for reference. The focus should be on which issues the reference hits, whether paragraphs can be extracted separately, and whether the keyword library and anchor text can stably identify the content.

2026-11-01

The garbage station update in September took nearly fourteen days to end, when is it suitable for review?

This round of garbage content update in September started in the late month and was marked completed in the early October, lasting for more than thirteen days. Before the update was completed, the ranking reading was the intermediate state within the rolling period, which cannot be taken as a conclusion; the time suitable for review should meet two conditions: the official panel is marked completed, and the data after the completion day returns to a stable interval. The waiting period should first check basic items such as robots and site maps.

2026-11-01

The window for the secure version and the major version release are scheduled in the same week, which one should the site follow first?

When the fixed secure version window and the annual major version are scheduled in the same week, follow the secure version first. The secure version closes known exploitable entries, while the major version changes data structures and template behaviors, with a higher verification cost. This article explains the differences in the nature of the two types of updates, how to evaluate the cost of delaying the major version by one week, and the two-step verification and rollback preparation to be reserved in the scheduling.

2026-11-01

The training crawler was blocked in robots, why can't the content still be retrieved

There are tests from more than 2,300 sites showing that only 3.6% of the sites block training crawlers in robots, but when accessing the homepage directly by real identity, 18.8% cannot access usable content. There is a gap between declarations and actual usability: writing rules does not mean they take effect, and there are also cases where the side that grants access cannot access the text. Self-inspection should be in order of four types of signals, not just whether that line of rules is in the text.

2026-11-01

Which type of spam submissions do the honeytrap field and captcha each block, can only one be left

The honey pot field blocks script submissions that do not parse styles, there is no visible challenge for real visitors; the scoring captcha is scored according to the possibility of robots, at the cost of mistakenly harming some real users. The two methods do not face each other, and whether only one can stand depends on the composition of garbage: when the amount of scripts is mainly used, the honey pot is enough, but when there are submissions with changed addresses and natural content, only the honey pot is not enough.

2026-11-01

After the site is hacked, should the cleaning costs and loss of business be calculated together?

The cost of cleaning after being hacked is quoted per occurrence, the loss of business is accumulated over time, and the two are different in terms of scope but must be calculated together: reporting only the cleaning fee will underestimate the cost of the event, as it does not include the recovery of search visibility, order loss, and subsequent reinforcement hours. The order of handling is to isolate first, clean next, and then check the entry, and after recovery, use backups and recycle bins to verify whether it has returned to a clean state.

2026-11-01

The domestic website building system announced the repair of remote screenshot risks, what should the site without an upgrade do first.

PbootCMS has strengthened the server-side request forgery protection for the remote capture feature of the editor in V3.2.25, added connection timeout, size limit, re-binding protection, and internal network address blocking, and supplemented the configuration of trusted proxies. For sites that cannot be upgraded temporarily, they should first collect background permissions and server out-of-network, then make a backup, clean up the cache after the upgrade, and check the pseudo-static rules, rather than just wait for automatic updates.

2026-11-01