So Outside Technology Security CMS
AnQiCMS
Online Demonstration

How much of the AI crawling proportion in the log needs to be limited separately

📅 2026-11-01 👁️ 0

How much of the crawling in the log is performed by AI before it needs to be limited separately? There is no universal threshold that can be directly applied. The proportion itself only indicates who initiated the request, while the limitation aims to solve the server load and crawling intent, which need to be judged separately: if the load is stable, adjust the robots instruction layer configuration first, and only adjust the bandwidth and frequency if the load is really overwhelmed.In the external statistics, the proportion of AI-based crawlers in the number of requests is no longer marginal, Googlebot accounts for 23.80%, Meta-ExternalAgent is 13.28%, ClaudeBot is 13.23%, and another 7.9% of requests were blocked by 403 Forbidden.

How to separate AI crawlers in the log

The first step is to classify by user-agent, not by IP. The identifiers and purposes of search engine crawlers and training crawlers are different: retrieval crawlers crawl to build indexes and bring visitors to the site; training crawlers only take content without generating return visits. Combining the two types in a statistics results in a proportion that has no decision-making value.

After classification, pair two columns of data: the number of requests and the actual number of visitors generated. The former is pressure, the latter is return, the speed limit decision looks at the ratio of the two. It is worth taking action only for categories with high request volume and low visitor return.

To what extent is it worth taking action based on the proportion?

The conditions for taking action do not look at the individual proportion, but whether three things are simultaneously established.

Observation item What does it mean Corresponding action
The proportion of crawling requests continues to rise. The crawling structure is changing. First check the robots instruction configuration.
The server concurrency pressure is rising, and the page loading speed is slowing down. Real resource competition. Do frequency and bandwidth limitations.
High crawling volume but almost no corresponding visitors. Only take and no return Consider content layer measures

AnQiCMS is developed using GoLang, a single machine can carry about 5 million PV, and it is not easy for normal-scale enterprise stations to be overwhelmed by crawlers at the load level. If it is indeed observed that the page loading speed is affected, first confirm whether it is a collection task or image resources consuming resources, and then decide whether to limit external crawling.

The sequence of action: first instructions, then bandwidth, then content layer

The order is wrong and it will hurt the collection. The first step is the robots instruction layer: AnQiCMS supports Robots.txt configuration, clearly identifying the crawlers to be treated differently, which is the smallest impact step, only changing whether the request is welcomed without changing the server behavior.

The second step is the frequency and bandwidth layer, which acts on gateway or service configuration, the risk is that the rules are written too broadly and the search engine is limited as well, which requires using logs to first verify the objects that are limited.

The interference code solves another matter.

The anti-crawling interference code is aimed at the mass collection of content, which belongs to the content layer. It does not reduce the number of crawling requests and does not participate in throttling. Some people hope to deal with the pressure of crawling by means of interference codes, but these two things are not on the same layer: interference codes reduce the quality of the content that is moved, throttling reduces the number of requests received by the server, one manages the result, and the other manages the pressure.

Similarly, a high proportion of being blocked back by 403 indicates that the site is already using rule layer or access layer rules. At this time, it is more appropriate to review the scope of the rules rather than add an additional layer of restriction.

Common questions

Should we block training type crawlers from the beginning?

Not recommended. First, count the actual visitors and crawling pressure it brings, and then decide; command layer configuration can express attitude, and there is no need to change the server behavior immediately.

Will speed limit affect normal inclusion?

Yes, so the order is very important. The configuration of robots command layer has controllable impact on search engines, and frequency limitation needs to be confirmed by the log before doing.

Can interference codes and robots be used together?

Yes, there is no conflict between the two. Robots control whether to come, and interference codes control the quality of the things taken away after coming.

Related articles

If the search exposure is increasing but the clicks are not, first distinguish between zero clicks or rankings dropping.

The exposure increases while the clicks do not, first use question keywords to split the two sets of data: if the decline is concentrated in question queries, it is mostly caused by zero clicks due to generative abstracts; if the exposure of brand words and model words declines synchronously, it is a ranking issue. Confirm and then check the site map, link push, static rules and 301 redirect one after another, these four steps are about being discovered and seen as soon as possible, not solving the abstract interception.

2026-11-01

After AI abstracts consume most of the clicks, what indicators should the enterprise site pay attention to?

When AI abstracts appear, the click-through rate of traditional results decreases significantly. At this time, the enterprise station should not focus on total traffic, but on three indicators: brand word search volume and direct visits, visitor quality brought by AI citations, and the conversion rate of inquiries and leads. AnQiCMS's keyword library is maintained by question, and is supported by LLMs.txt and AI writing capabilities, so that each group of data has a verifiable source.

2026-11-01

The back-end user field is missing a whitelist to change to password hash, where should the permission be collected in that layer.

If the quick change value interface of the back-end does not limit the writable fields, visitors can direct the submitted fields to sensitive columns such as password hash, which is equivalent to bypassing login to obtain identity directly. Permissions should be collected in three places: the list of changeable fields, group authorization and write verification, and hiding the entrance on the interface does not count as collecting. Password storage and session mechanisms should be matched, and content layer sensitive word filtering and review belong to another layer.

2026-11-01

If the front-end query conditions are not parameter-bound, it will be dragged into the database. Which item on the upgrade list should be pre-checked.

When using string concatenation for front-end query conditions without parameter binding, the values entered by the visitor can change the SQL semantics, which is the root of injection problems. When checking the upgrade log, first determine whether the fix is for injection surface: query conditions and sorting parameters take priority, followed by filtering enhancement. The content system itself has fixed list sorting parameter injection in v3.6.6, built-in SQL injection and XSS protection belong to the system level, and sensitive word filtering belongs to the content level.

2026-11-01

How should the section be cut when AI answers extract content by paragraph?

The section should be divided according to 'a paragraph that can independently answer a question', not by word count or visual layout. The smallest unit taken away is a paragraph, so each section should only discuss a sub-question, provide a conclusion in the first sentence, and use references that do not depend on the previous text; the section title should be written as the question itself. The AI writing output of AnQiCMS still needs to be manually reviewed for facts, and then content review and TDK configuration are used to unify the口径.

2026-11-01

Visitors brought by AI are more likely to convert than search visitors, so which end of the content site should be expanded?

When the visitor quality is higher, it should first expand the acceptance area, and then expand the content capacity. Visitors brought by AI have already entered the station with a clear intention. If the product description, quotation, and order path are incomplete, more content will only waste more expensive traffic. Content capacity can be expanded with AI writing and bulk import, and the order of expansion of the transaction chain should first be connected with order, distribution, and financial data, and the order of expansion determines whether this group of visitors can be retained.

2026-11-01

How should the monthly rent of the host of the self-hosted site be selected within what range

What is the monthly hosting cost for self-hosted sites? Determine the scale first and then the tier: Showcase websites fall within the shared hosting range of $2 to $15 per month, content volume and concurrency determine VPS at $10 to $50 or dedicated hosting above $30 to $100, and dedicated servers start from $100 per month. Domain names cost an additional $15 to $20 per year, and memory capacity determines how many sites can be opened with the same functionality.

2026-11-01

The quotation for outsourcing corporate website development differs several times, what are the differences in several work items?

Outsourcing the quote for the corporate website is several times different, what are the differences in the workload? The price difference mainly comes from three aspects: the amount of content modeling and custom field definition, the interface coordination brought by front-end and back-end separation, and the number of customized pages according to the scenario. In the publicly available quote range that can be verified on the day, the cost of building a mature system website is about 15,000 to 60,000 US dollars, the headless solution is about 40,000 to 150,000 US dollars, and fully customized starts at 80,000 US dollars.

2026-11-01