So Outside Technology Security CMS
AnQiCMS
Online Demonstration

GPTBot has hundreds of billions of requests per month, should the content site be expanded separately for AI traffic?

📅 2026-11-01 👁️ 0

GPTBot has several billion requests per month, should the content site allocate capacity for AI traffic separately? It's not about expanding capacity first, but about calculating which layer this request falls into. In public statements, the fact that GPTBot's monthly request volume reaches several billion indicates a large scale of crawling, but the crawler does not execute JavaScript nor generate clicks, the pressure is concentrated on page rendering and cache hits, not on database concurrency. What really needs to be judged is whether your site will degrade to dynamic rendering when attacked.

Request volume and referral traffic are not the same thing

The cost structure of crawling requests is different from that of visitor requests:

Traffic type Main consumption Whether to execute the script The significance of expansion
Search engine crawler Page rendering and response time Complex scripts are usually not executed Focus on cache hit and response stability
AI-trained crawler Bulk reading of the entire site, large and continuous Does not execute JavaScript Focus on the boundary and rate control of crawling
AI retrieval type crawler Read a small amount of pages according to the query Does not execute JavaScript Focus on whether the content is self-contained as a paragraph
Real visitors Interaction, login, form and database concurrency Complete execution This is the main goal of capacity planning

The pressure brought by AI traffic is read-type pressure. If each page of the site is dynamically rendered and there is no cache layer, the increase in crawling volume will directly reflect the increase in server load; if the content has been staticized, the marginal cost of crawling is very low, and the expansion demand is also limited.

Where does the pressure fall

The method of judgment is to split the processing path of a single fetch request: whether it can hit static products, whether it needs to perform template rendering, and whether it needs to check the database. Most content sites have bottlenecks in

Content collection sites have a special feature: when batch reading sites with long text, many images, and deep pagination, the distribution cost of static files will exceed the cost of dynamic rendering itself. In this case, the first thing to optimize is not to add machines, but to place static resources in cacheable locations and set a crawlable range for pagination and filtering parameters.

Calculate the load first and then talk about expansion.

Can use the two order of magnitude specifications given for rough estimation. One, the page loading speed is significantly improved compared to traditional PHP CMS, a single machine can handle about 5 million PV, here we need to pay attention to the meaning of 'about', treat it as a reference for how many content sites can be placed, rather than an exact indicator. Two, the memory usage is reduced by about 80% compared to PHP CMS, which determines how many services can run in parallel on the same machine, and also determines whether swapping will be triggered due to memory shortage when the capture volume increases.

At these two scales, the margin of a regular corporate website is usually enough to absorb AI crawling. The two types that truly need to be expanded in advance are: websites that heavily rely on dynamic rendering across the entire site without caching, and those that place API calls and page rendering in the same process and resource pool.

How to determine the crawling boundary

The crawling strategy should be divided according to content type rather than traffic size. The first layer is the robots configuration: explicitly allow or prohibit paths, block directories such as background, preview, and internal search that should not be crawled, and decide whether to treat training crawlers and retrieval crawlers differently - in the public statistics, 35% of readable sites have blocked at least one training crawler, 21% have blocked at least one retrieval crawler, and the choice between the two lines is inherently different.The second layer is llms.txt, using the navigation file generated by the site for the large language model to converge the key content read by the model. It is the third layer that is about rate control and caching strategy.

It should be noted that the cost of blocking search engine spiders: if you want AI answers to be able to cite their own paragraphs, blocking search engine spiders together will make visibility zero. Such decisions should be made by column rather than the entire site.

Common questions

Question: The proportion of AI spiders in the log is very high, does this mean that crawling is consuming bandwidth?

Answer: First, look at the distribution of the requested paths. If the proportion is high but the hits are all static pages, the marginal cost is very low; if concentrated in dynamic addresses with parameters, it is necessary to adjust the crawlable range instead of directly increasing bandwidth.

Question: Is it technically feasible to block only training-type crawlers and retain retrieval-type crawlers?

Answer: It is feasible, as they are distinguished by different identifiers, and the robots configuration can be declared separately. It should be noted that the instruction only takes effect for crawling in compliance with the protocol and cannot be used as an access control means.

Question: Will the paragraph writing style of the content site affect the crawling cost?

Answer: It will not significantly affect the cost, but it will affect the reference effect. The smallest unit AI reads is a paragraph, and self-contained paragraphs are easier to extract for use. This belongs to content-level optimization and is a different matter from expansion.

Related articles

The number of background brute-force alarms has increased, which part of the JWT authentication can block?

The login link can be divided into three segments: password verification, ticket issuance, and ticket usage. JWT authentication controls the latter two segments, so even if the password is guessed successfully, it cannot obtain a reusable access token; it cannot prevent the collision library from obtaining the real password. This article uses the v3.6.6 fixed list sorting parameter injection and site switching token forgery as an example to explain why the issuance link needs to be monitored separately, and provides a list of complementary actions outside of brute force.

2026-11-01

How to avoid such supply chain pitfalls as cracking themes and free plugins with backdoors when building a website.

Unknown origin cracked themes and free plugins directly put third-party code into writable directories and template links. In historical events, there was once a backdoor that affected 23,000 sites. This article provides three steps for origin verification, content review and sensitive word filtering on the publishing side, verification restrictions on the form and login side, and recovery paths with static file backups after an incident.

2026-11-01

In the revision record of the domestic CMS, there is a parameter binding item. How should such entries be read during selection?

The revision record parameter binding, field whitelist and other such terminology descriptions of the repair occur at the query construction layer, which is not at the same level as the interface text repair. This article teaches how to judge the depth of repair from the wording, using v3.6.6 to simultaneously repair list sorting parameter injection and login credential forgery as a control sample, and to give three items that can be checked at the selection stage: the proportion of security entries, whether the announcement is accompanied by disposal suggestions, and whether the version rhythm is stable.

2026-11-01

The plugin layer accounts for ninety percent of the vulnerabilities, isn't it more worry-free for a website system with fewer plugins?

The annual statistics show that 11,334 new vulnerabilities, 91% of which are in the plugin layer, and only 6 in the core, indicating that the source of the maintenance burden is the extension layer rather than the program itself. This article analyzes the differences between more and fewer plugins in terms of upgrading, auditing, and regression verification, and proposes a method to include built-in capabilities in the selection to avoid using the word 'convenient' to replace verification.

2026-11-01

The company introduction written in the AI abstract is different from the company's official website, which part of the content should be changed first

The diameter is inconsistent, first locate the source, then sort and process according to the cost of modification. The sequence is usually to first change the erroneous statements accumulated in the keyword library and site description, then change the title keyword description and the first paragraph on the page responsible for the answer, and finally use links to push and update the site map to reread the new diameter. This article provides the method of tracing the source and the schedule for verification after modification.

2026-11-01

Who will take over the functions that were solved by plugins when moving from a PHP site to a Go-based CMS?

Before migration, list the functions handled by the plugin first, and then allocate them into three categories: built-in capabilities, content model承接, and secondary interface development. This article provides a functional comparison table and migration order: use a custom content model to map the content structure first, import historical articles in batch through ZIP or Excel, rely on pseudo-static and 301 redirect to retain the existing SEO收录, and estimate the yield during operation according to a reduction of about 80% of the amount level.

2026-11-01

AI prefers self-contained paragraphs in citations, whether long texts should be divided into sections according to questions and answers.

Long texts are worth dividing into sections according to questions and answers, but the purpose is not to break up the article, but to make each paragraph readable even out of context. There are three criteria: whether the section can independently answer a question, whether the paragraph has its own conclusion and conditions, and whether the reference depends on the previous text. After dividing, you still need to use a keyword library to unify the names, and use anchor text to link the sections back to the main page, otherwise the self-contained paragraphs will become isolated within the site.

2026-11-01

Announcement of security notice should be written in advance with the release window, and the site should reserve how long for maintenance time.

When the release window for the announcement of security notice is given in advance, the maintenance time should be arranged in five stages: backup, test environment verification, formal upgrade, online verification, and rollback reserve. The available window is usually only a few hours, and what the site really needs to reserve is the preparation and follow-up time outside the window, usually arranged for half a day to a day. The rollback speed depends on the backup scope and deployment method. If static files are not backed up together, the recovery time will be significantly longer.

2026-11-01