So Outside Technology Security CMS
AnQiCMS
Online Demonstration

The robots file has been changed, but the crawling log has not changed, first check whether it takes effect or check the writing method.

📅 2026-11-01 👁️ 0

If the robots file has changed but the crawling log is not changed, first check whether it is effective or check the format? First check whether it is effective, then check the format. If the format is wrong, it will always be wrong, but if the effectiveness is not in place, even if the format is changed three times, you will not see any changes. The most common pitfall is to take 'I changed it' as 'the crawler read it'.

First confirm whether the changes have been read.

robots configuration is changed in the background, after landing, it needs to pass two stages: whether the file itself outputs new content, and whether the crawler uses the new cache to read it again.

The first one can be verified by itself: directly request the robots.txt under the root path of the site, and see if the returned version is the one just written. The most common unexpected event here is that pseudo-static or cache redirects the request to old content, or changes may only exist in the configuration and have not yet been generated into actual files.

The second one needs to wait. It is common for the crawler to read new instructions with a delay, and there is no change in the daily logs after the change; it is more effective to wait a few days to see the trend. Judging whether it takes effect should look at the trend rather than a single log entry, and the differences in the single crawl record are more from the task rhythm.

The three most common mistakes in writing style

  • The scope of action is written too small or too large. The rules for grouping by user-agent only take effect on the crawlers that match, and when both the general group and the named group exist, which group to read is determined by the crawler's own implementation. Changing one does not mean it takes effect globally.
  • The path prefix is not aligned. The instruction matches the prefix, whether the directory ends with a slash, and whether the case is consistent with the real address, any deviation will cause the rule to fail or expand.
  • Two rules conflict with each other. When the same prefix has both allowance and prohibition, the result may not be as expected. Deleting the conflicting entries and rewriting them is more reliable than adding one to cover.

Also check if the site map address is still hanging in the file. Many people, when changing the shielding rules, also remove the entry.

Crawl control and anti-crawling are not the same thing.

There is a third possibility if the log does not change: what you want to solve is not a crawling problem.

robots control whether to take it and which paths can be taken, it cannot prevent the distribution of content by sites that have already obtained it. The protection at the content level takes another approach: interference codes are inserted into the page in invisible fragments to ordinary readers and harmful to copyists, preventing collection is aimed at taking actions rather than crawling actions. The verification methods of the two levels are also different - the former looks at the crawling log and the collection area, and the latter looks at the source code of the page and the actual presentation on external sites.

When deploying across multiple sites, the robots configuration of each site takes effect independently, changing the main site will not affect the sub-sites, and verification needs to be done for each site.

How long should the verification take effect?

Leave a few days to two weeks before looking at the crawling trend to get closer to the actual situation, especially in large file quantity or multi-site scenarios. Maintain a stable configuration in the middle, and changing while watching will not match the cause and effect.

Common questions

Can you use the crawling records to verify immediately after the modification?Only the file output can be verified, whether the crawler has been reread cannot be verified, and the trend needs to wait.

The logs are full of crawler names, but it is not clear who they are?First group and count by user-agent, then look at the corresponding rule block for each group, not just the total amount.

If robots are banned, will it still be crawled?Banning is a request rather than a mandatory measure, historical addresses and entries referenced by other sources may still appear.

Related articles

The site map entries are almost at the limit, how to arrange the submission order after splitting into index files?

The single sitemap file is constrained by both the number of entries and volume limits, and it should be split once either limit is reached. After splitting, use the index file to summarize multiple slices, and only the index file needs to be submitted when submitting, without submitting each slice individually. The order is to first generate and make the slices accessible independently, then attach them to the index file, and finally clarify the division of labor between the proactive push of new content and the sitemap, to avoid duplicate submissions and disruption of crawling.

2026-11-01

Language tags do not improve ranking but are responsible for pairing versions, where do multi-language sites often go wrong in several steps?

The language tag is used to pair different language versions of the same content to the correct visitor, it does not participate in ranking calculations. Common mistakes on multilingual sites include four steps: improper code writing, pointing to unindexed pages, lack of bidirectional links, and not providing a default language fallback. First, fix the pairing relationship, then discuss translation quality and indexing.

2026-11-01

The page cache of domestic website building systems can also be executed by the code, how to view the list of historical version upgrades?

The page cache was executed as code, the problem lies in the cache write not stripping the script tag, the step of reading the cache is equivalent to executing the concatenated content. When reviewing the history of version upgrade lists, sort by the extent of harm: first confirm whether the cache and template injection class fixes are within the used version, then check parameter binding and security filtering, and finally look at feature updates and compare them with your own deployed version number by version number.

2026-11-01

Open source release platform splits the welcome email into paid tiers, what are the missing links in member operation?

Splitting the welcome email into paid tiers solves the match rate of the first email, but what are the missing links in member operation? Such as: funnel statistics of entry process, permission and content visibility range by group, reminder points for expiration and renewal, as well as comments and interaction back to the operation list. Only supplement the welcome email, the middle section of the funnel is still invisible.

2026-11-01

After the AI entry file goes online, the number of references has not increased, first distinguish between the two steps of crawlable and citable references

The entry file (llms.txt) for large model sites solves the issues of discoverability and navigation, and has a low adoption rate: Only 10.13% of the nearly 300,000 domain samples have this file, and it does not affect how the AI system sees or references content. The normal increase in references after launch should focus on verifying the writing style of the sections that can be referenced and the accessibility of crawling.

2026-11-01

The memory baseline of self-hosted programs is written in the deployment document, how to pick the content system for low-end servers

The memory baseline in the deployment document is a hard requirement for selecting low-end servers: The public threshold of mainstream self-hosted programs is usually hundreds of MB to 1GB or more. When the baseline is higher than the machine's surplus, upgrading the host is more likely to happen earlier than changing the architecture. In the low-end environment, pick in the order of resident memory first, then dependent components, and finally deployment methods. The content system of the Go technology stack is about 80% lower in memory than PHP programs

2026-11-01

When moving a hosting website building platform, articles can be exported, but why do we need to rebuild the pages?

The export of hosting website platforms usually only covers structured article content, titles, text, and tags can be taken away, the layout and components belong to the platform's private rendering configuration, which is not in the data, so the home page and landing page pages like these need to be remade in the new system. When moving, separate the schedule for batch importing articles and page reconstruction, and the old address redirection should be in place before the switch.

2026-11-01

When the week of the major version release and the discontinuation of old branches fall on the same day, which step should the site take first?

When the major version release of an open-source content system and the discontinuation of old branches fall on the same day, the site should first do is to apply the current security patches and perform backup rollback drills, rather than rushing to go online. The major version change affects the compatibility of templates and extensions, while the discontinuation affects the patch supply, with different risk rhythms; schedule the upgrade window of the content system itself into the same table to avoid gaps where no one is responsible.

2026-11-01