So Outside Technology Security CMS
AnQiCMS
Online Demonstration

About one in ten sites that are directly rejected by the crawler, should we check robots or service configuration first?

📅 2026-11-01 👁️ 0

Almost one in ten sites block spider requests into a status code of rejection. When checking, you should first look at the server configuration and then the robots file, as their mechanisms of action are different: the robots file is a polite guidance, no matter how strict it is, it will not cause the request to return a rejection code; once the crawling party receives a rejected status, blocking must occur at the service layer or rule layer, and changing the robots will not change the result.

The flow and rejection ratio in the day's aperture

On October 30, 2026, the distribution of a group of traffic verified from the original text is: Googlebot accounts for 23.80% of the crawling requests, another meta tag class proxy accounts for 13.28%, and the two agents doing model training and retrieval are 13.23% and 8.05% respectively; In the same statistical data, the proportion of traffic rejected with 403 is 7.9%, close to 10%. These numbers indicate two things: The blocked sites are not isolated cases, but a normal state; The blocking occurs at the service layer, so the form of expression is status code rather than guidance生效.

Read clearly first: The error code is returned by the visited party, and the guidance is voluntarily observed by the other party. Reversing the troubleshooting order will result in the illusion that traffic is still the same after modifying the robots file.

Each layer has its own characteristics of three-level denial.

Interception layer Typical return Observable characteristics What should be confirmed first?
robots guidance layer Normal status code, the request does not occur There is no corresponding path access in the crawling record Is the path written in the prohibited section?
Rule and Protection Layer Deny status code or verification page Browser accessible from the same address, proxy caught Whether the proxy identifier is on the blacklist
URL rewrite layer Deny or not found Parameterized address is normal, static address caught Which type of request does the pseudo-static rule take effect on

Self-check order layered by status code

The first step is to group the crawling log by status code: first look at how many requests fall into the deny code, and then see which paths and prefixes these requests are concentrated. Concentrated in a few prefixes is usually directional interception by the rule layer; scattered throughout the site and related to the proxy identifier are often batch interceptions by the protection layer according to the proxy list.

In AnQiCMS and similar self-built programs, the robots file and URL rewriting rules are two independent entries: the former controls which paths are not recommended to be crawled, and the latter determines the address form in which the same content is accessed. Changing the pseudo-static rules can easily affect the crawling path, so don't ignore this layer when investigating.

How to verify after the modification

The most convenient way is to leave verification records grouped by proxy. Record the request volume and return code distribution of each search proxy and training proxy separately, test one layer after changing one layer. Change only one layer at a time: change the robots section, whitelist, and rewrite rules at the same time, and the results cannot be attributed. Next time, when the same problem occurs, it is still guessing.

If you really want to intercept a certain type of proxy, align the interception action with the purpose: if you just want to reduce the content collected, using guidance and干扰 codes such as content layer means is more controllable; only in the service layer should you return a refusal code and reserve a fallback position to completely refuse access.

Common questions

Ask: Can modifying the robots file prevent training agents? Answer: Cannot guarantee. Guidance relies on the voluntary compliance of the other party, and the return code is the mandatory result of the service layer.

Ask: The browser access is normal, but the proxy gets a rejection code, where is the problem? Answer: First, check whether the rules and protection layer have distinguished according to the proxy identifier, which is the most common cause.

Ask: After changing the rewrite rule, the crawling volume has decreased, how to locate it? Answer: Compare the distribution of return codes between static addresses and parameterized addresses, if only static addresses are captured, the rewrite rule is a suspect point.

Related articles

AI referral sessions only account for 1% but have higher conversion rates, which two indicators should you believe first

Believe in the conversion rate first, but use it to test the reception ability, not directly as an expansion basis. The review on the day was: across more than ten thousand domain names, AI recommendation on average only accounts for about 1% of the total sessions, but the conversion rate of recommended visitors is much higher than that of natural search sources. The share index fluctuates greatly and is affected by the entry; the quality index is more suitable for long-term baseline, ensuring that a small number of highly interested visitors do not lose interest first.

2026-11-01

To what extent has the number of times the site blocking the training crawler has been cited by AI dropped?

The sample reviewed on the day is over a thousand domain names, the proportion of blocked training crawlers varies from 17% to 14%; the sites blocked by search crawlers have a difference of two orders of magnitude in the number of times cited compared to those not blocked. This does not mean that the entire site should be allowed to pass, but rather that it is more cost-effective to grade by page type: product and document pages are allowed to pass, while draft and low-value archived pages are blocked.

2026-11-01

After the release of LLMs.txt, ninety percent of the files were not requested, is this number looking at the access log?

Yes, these extremely low transaction rate numbers come from the site's own access logs: first, filter out the domains that have placed files, and then check the server logs for corresponding crawling requests. The two sets of criteria reviewed that day need to be read separately, one is the adoption rate, which measures how many domains have placed files; the other is the read rate, which measures whether there are requests after placement. The two cannot be mixed into one sentence.

2026-11-01

The total number of vulnerability libraries is increasing day by day, should the webmaster pay attention to the total amount or the increase?

The total number of responses is about whether the ecosystem is expanding or contracting, and the new responses are about whether there are any entries related to you today, both cannot be mutually exclusive. The executable approach is to filter the newly added entries using the component list, leaving only those related to this site and versions within the affected range, and then arrange them in order of whether they can be triggered anonymously. The list check can be exported in bulk via an interface, and there is no need to manually flip through pages.

2026-11-01

The quality indicators of the same batch of sites differ by 30%, how much are hosting and website construction methods accounted for

The real user experience passing rate of different website building platforms can vary by 30%, and the platform data verified that day showed that the hosted website building platform is generally higher than the free website building program, and the full station baseline is only equal to the worst mainstream platform. On attribution, the weight of the website building method and the theme category is often greater than that of the hosting category, and it is better to measure the real user data first and then decide which side to move, so as not to waste the budget on the layer with small returns.

2026-11-01

Shared hosting costs as little as a few dollars per month, when is it necessary to switch to a virtual private server

Shared hosting's lowest-tier monthly fee is just a few dollars, do you need to switch to a dedicated virtual server without looking at the number of website pages, but rather at three resource curves: whether the response time is extended during peak hours, whether the resident memory is at the top, and whether the concurrent peak is queuing. The domain annual fee and hosting monthly fee should be calculated separately; when the program-side memory usage is low, the same type of load can maintain a lower tier.

2026-11-01

The website package with a low price for the first year's renewal doubles, and the price difference appears in which month.

The first-year low price, the renewal plan is almost doubled, the price difference does not appear in the first month, but is raised to the monthly bill in one go when the renewal cycle changes, and the sample checked on the day is about nine dollars more in the first year, and the renewal price rises to seventeen to twenty dollars. By separating the first-year price and the renewal price into two columns and recalculating the annual expenditure from the second year, the plans are truly comparable.

2026-11-01

How to calculate the hidden labor cost into the annual cost, besides the monthly computing power cost of building a program for several tens of dollars

The monthly computing power cost of building a content management system is just the accounting cost, and the hidden labor cost should be calculated into the annual cost per head per hour: patches and upgrades, backup and rollback drills, daily inspections and fault recovery are all converted into hours, and then multiplied by the mixed working hour unit price. The sample checked that day shows that the labor cost can reach several times the computing power cost, and the built-in backup and batch replacement capability compresses these steps of working hours.

2026-11-01