So Outside Technology Security CMS
AnQiCMS
Online Demonstration

After the release of LLMs.txt, ninety percent of the files were not requested, is this number looking at the access log?

📅 2026-11-01 👁️ 0

After the LLMs.txt file was released, more than 90% did not receive any requests. Is this statistic looking at the access log? Yes, this is the server access log criteria, not the crawler list or reference statistics. The two studies reviewed that day had different tasks: one counted about three million domain names, looking at 'whether this file was placed', about 10% were placed; the other looked at 'whether there were any requests after being placed' in the self-analysis data of more than ten thousand sites, and the conclusion was that 97% of the files did not receive any visits. The two numbers measure different things, and they were misunderstood when put in one sentence.

What do the two groups of numbers measure

The adoption rate is the result of external scanning: the program goes to request the fixed path of the target domain, and the content is returned even if there is no. It answers the question of 'how many people are doing this', and belongs to ecological observation.

The read rate is the result of the site's own logs: in the domains where files have already been placed, filter out the model grab by user-agent and see if there are any requests for the path. It answers the question of 'is it useful after doing it', and belongs to effect observation.

The latter is harder because it does not rely on guessing: the crawling behavior leaves real records in the logs. It is also more conservative - even if requests appear in the logs, it does not mean that the content is therefore cited. There was also a judgment in the second study reviewed that day: this file currently does not affect how the system sees or references content.

What columns should be considered in the log scope?

Follow three steps when self-checking. The first step is to filter paths in the log, isolate requests for the file, and note that some crawlers may request both with and without a slash. The second step is to group by user-agent, separate mainstream model crawlers from self-built spiders, and only then can you get a readable conclusion. The third step is to check the return status, and files that exist but return an abnormal code will be counted as 'requests made but not read'.

Do not skip the second step in sequence: just looking at the total number of requests will also include ordinary user access, resulting in an obviously higher conclusion.

After placing the file, what needs to be verified?

The purpose of placing the file itself is not the goal, it solves the problem of 'how the model understands the site structure'. The real factor in whether it can be read is crawling and collection: the sitemap must allow new pages to be discovered, and link push and active push should shorten the interval between publishing time and crawling. llms.txt is parallel to these two things, not a substitute.

Therefore, a reasonable self-check list is: whether the file is accessible and returns normally, whether the crawler appears in the logs, whether the sitemap covers the answer pages, and how long after a new page is published does it appear in the index. Only one of the four items is related to this file.

Common questions

Almost no one requests it, should we still release it?The cost of releasing it is very low, it is automatically generated within the site and does not require manual maintenance; but don't treat it as the main means of visibility.

Is it blocked if there are no requests in the log?It's not necessarily so, it could be that the crawler has never come, or it could be that the path formatting or cache return is inconsistent, and we need to check separately in three steps.

Can it be used to replace the sitemap?No, the two serve different reading methods, and the inclusion side still takes the sitemap and push as the standard.

Related articles

The total number of vulnerability libraries is increasing day by day, should the webmaster pay attention to the total amount or the increase?

The total number of responses is about whether the ecosystem is expanding or contracting, and the new responses are about whether there are any entries related to you today, both cannot be mutually exclusive. The executable approach is to filter the newly added entries using the component list, leaving only those related to this site and versions within the affected range, and then arrange them in order of whether they can be triggered anonymously. The list check can be exported in bulk via an interface, and there is no need to manually flip through pages.

2026-11-01

After AI capabilities entered CMS, why does the background also need to be managed as an attack surface?

The AI ability makes the background go from 'viewing content' to 'being able to act': one conversation can write libraries, change configurations, and adjust interfaces, making the login state and interface credentials high-value targets. The management method is to tighten three types of login-free interfaces separately, set exposure ranges according to ability domains, retain operation approvals, and pay attention to security changes at the version level. The two types of high-risk issues fixed in the current version exactly illustrate that the input on the backend side and the login tickets should be treated as attack surfaces.

2026-11-01

When connecting a large model, should it be done using a third-party module or the built-in capability, and where do the risks differ?

The risks of the two routes differ in who holds the key and permissions, and whether the entry point of the capability is controlled. The third-party module packages the model key, write permissions, and call links into the extension, and needs to be upgraded in bulk once the upstream issues an announcement; the built-in capability converges the entry point within the program, and can set the exposure range by domain and perform operation approval by round. The selection should consider the density of announcements and the boundary of permissions, not just whether this feature exists.

2026-11-01

Where does the difference lie in a self-built program that can run with 1GB of memory and 256MB?

The difference in memory threshold of self-built programs mainly comes from the running mode rather than the number of functions: the stack of resident application servers and databases consumes more memory than the combination of single-process running and lightweight storage. The criteria reviewed that day clearly divides the two types of thresholds, and the selection should be judged by three factors: concurrency, whether multiple sites are shared, and whether expansion is necessary, rather than ticking off the feature list. Memory usage belongs to the official approximate criterion and cannot be taken as an exact multiple.

2026-11-01

To what extent has the number of times the site blocking the training crawler has been cited by AI dropped?

The sample reviewed on the day is over a thousand domain names, the proportion of blocked training crawlers varies from 17% to 14%; the sites blocked by search crawlers have a difference of two orders of magnitude in the number of times cited compared to those not blocked. This does not mean that the entire site should be allowed to pass, but rather that it is more cost-effective to grade by page type: product and document pages are allowed to pass, while draft and low-value archived pages are blocked.

2026-11-01

AI referral sessions only account for 1% but have higher conversion rates, which two indicators should you believe first

Believe in the conversion rate first, but use it to test the reception ability, not directly as an expansion basis. The review on the day was: across more than ten thousand domain names, AI recommendation on average only accounts for about 1% of the total sessions, but the conversion rate of recommended visitors is much higher than that of natural search sources. The share index fluctuates greatly and is affected by the entry; the quality index is more suitable for long-term baseline, ensuring that a small number of highly interested visitors do not lose interest first.

2026-11-01

About one in ten sites that are directly rejected by the crawler, should we check robots or service configuration first?

A nearly 10% of the sites return a denial status code to the crawler, the troubleshooting sequence is to first check the server configuration rather than change the robots file: robots is just a polite guideline, a denial code means the intercept occurs at the service layer. Locate by grouping status codes for rules, firewall, or pseudo-static, and verify layer by layer, changing only one layer at a time, so that the robots file is changed but the denial code remains the same.

2026-11-01

The quality indicators of the same batch of sites differ by 30%, how much are hosting and website construction methods accounted for

The real user experience passing rate of different website building platforms can vary by 30%, and the platform data verified that day showed that the hosted website building platform is generally higher than the free website building program, and the full station baseline is only equal to the worst mainstream platform. On attribution, the weight of the website building method and the theme category is often greater than that of the hosting category, and it is better to measure the real user data first and then decide which side to move, so as not to waste the budget on the layer with small returns.

2026-11-01