After AI training crawling and AI search crawling are divided into two categories, how to write the robots file without affecting indexing?

After dividing AI training crawling and AI search crawling into two categories, the writing of the robots file should no longer be 'completely blocked' or 'completely open.' The industry practice has been clarified: blocking the training crawling while retaining the search crawling can still allow the page to enter the AI answer's citation scope. To avoid damaging the inclusion, you should answer three questions before writing the rules, and there must be an inspection step after the modification, and it is clear what the robots, site map, llms.txt, and link push each manage.

Why do the two types of crawling need to be separated

The goal of the training type crawling is to incorporate content into the model corpus, and the boundaries that webmasters are concerned about are usually whether the content assets are absorbed in the long run; the goal of the search type crawling is to retrieve information in real time, and the page being cited will bring traceable access sources. The two types of intentions are different, and their impact on the site is also different, so controlling them separately according to their purposes is a reasonable granularity. Mixing the two types together and sealing them into a switch is a common consequence, which is not much to stop the training, but rather to close the accessible entry first.

The three questions before writing the rules

Question 1: Should this directory not have been crawled in the first place? Background paths, preview links, in-site search results pages, and list pages with filtering parameters are part of the 'ban before discussing AI' section. These should also be handled even though they are unrelated to AI.

Question 2: Is the ban for copyright boundaries or to save crawling budget? The wording is different. Saving budget should be controlled by parameter rules and site map priority, not by blocking the entire content directory.

Question 3: Do you still want to be cited after the ban? If so, a passage should be left for search engine crawling, and the premise of 'content will be read in real time' should be accepted.

How to write specifically

Adhere to three principles in writing. First, write rules by grouping according to crawling identifiers, rather than using a global rule to cover all crawlers, because different crawlers have inconsistent parsing capabilities for the same instruction. Second, the instructions should be written in a verifiable form: when prohibiting a directory, write the path in full, and avoid using overly broad matching to take away the content directory. Third, after making changes, check the logs, the actual effect of the robots.txt file depends on whether the crawler complies with it, and the paper rules are often inconsistent with online behavior.

AnQiCMS's robots configuration is completed in the background, no need to modify the file directly; the same site can also configure 301 redirection and static URL rewriting, which will affect the crawling path. When modifying robots, both should be checked together to avoid errors such as 'robots have been allowed, but the redirection points to a prohibited directory.'

How to self-inspect after modification

Suggest following this order:

1. Directly access the robots file address in the browser, confirm that the content is consistent with the background configuration and is not cached with an old version;
2. Check the crawlable status of specific pages using a crawling tool, focusing on whether the blocking rules mistakenly hit the content pages;
3. Check the crawling statistics in the search console or similar tools, confirming that the decrease in crawling volume after the ban is the directory you plan to converge;
4. Observe the changes in indexing and citation, as search crawling being blocked will be reflected in the inclusion and citation, usually not appearing on the day, so leave an observation window to judge;
5. Keep a record of changes that can be rolled back, AnQiCMS supports backup of data and static files, and leave a copy before and after configuration changes that can be restored quickly.}

Work division with sitemap, llms.txt, and link push

These four things are often discussed together, but actually they each manage a section:

• robots manage 'whether to crawl'
• The sitemap (sitemap) manages 'where to find pages', AnQiCMS supports automatic generation;
• llms.txt manages 'entry explanation for large models', automatically generated by the system, allowing the model side to read the site structure and key pages;
• Link push and active push manage 'when new content is known', responsible for timely notification of updates to search engines.

Only write robots without mentioning the sitemap is like turning on the door switch but not telling the other party where the entrance is; only placing llms.txt but blocking the content directory in robots, even if the file is written perfectly, there will be no crawling. Aligning the four in order is the complete crawling control plan.

Common questions

Will blocking the training class crawling affect the natural search inclusion?

It will not have a direct impact, as long as you only write rules for training class identification and retain the search engine itself and search class crawling. The risk point lies in the broad writing, be sure to check the content page status according to the self-inspection list.

Can it be prohibited only in robots without additional measures?

Robots are request-based constraints that depend on the compliance of the crawling party. If the goal is stronger technical isolation, it is also necessary to combine server-side authentication and access frequency control, leaving sensitive interfaces within the scope that requires logged-in status.

How long after changing the rules can you see the change in inclusion?

It depends on the crawling frequency and index processing, usually lagging behind by days. When judging, do not just look at the daily data, but at least cover a complete crawling cycle.