The robots file contains instructions not listed in the official documentation, will the crawler follow them?
robots.txt wrote instructions that are not in the official documentation, cannot assume that the crawler will support or follow. The search engine side is very clear: instructions not included in the official documentation should not be assumed to be parsed and followed. Whether non-standard writing can take effect depends on the implementation of each crawling program, not on how reasonable the line of text in the configuration file looks.
What robots commands does the official support?
The scope that can be relied upon is only the fields listed item by item in the document: user group matching, allowed and prohibited paths, sitemap declaration, and a few extensions declared separately by a few companies. The criteria is not 'others have supported it', but 'this one has promised it in its own document.'
This means that the same robots text may have different effects on different crawling programs. When creating a multi-engine site, it is more stable to write the control target as a subset supported by each company than to pile private instructions from each company.
Three outcomes of non-standard instructions
| Conclusion | Performance | Risk |
|---|---|---|
| Ignored | The crawler continues to crawl according to the other rules | You think you've blocked it, but actually you haven't |
| Partially supported | Only effective on one crawler | Multiple engine calibers split, difficult to reproduce |
| Parsing interrupted | Unknown syntax causes the following lines to be skipped | Invalidated together with the regular rules in the same file |
The most troublesome is the third kind. Some parsers will skip the entire segment when encountering unrecognized fields, resulting in the prohibition path written correctly also becoming invalid. These problems will not report an error, and can only be discovered by checking the crawling records
What kind of syntax should be used to achieve the desired effect
First determine which layer the target falls into, then select the tool:
- Don't want a certain path to be crawled: Use the forbidden path rules within the document, write separately for each user group.
- Don't want a certain page to be indexed: Crawl blocking and index blocking are two different things; forbidding crawling might instead cause the page to continue existing in the index with external links, at which point you should go back to 301 redirection or permission control.
- Want to change the URL form: This belongs to the responsibility of the pseudo-static rule, written in the site configuration, so that the external address corresponds to the internal route one by one, and robots only declare which paths should not be crawled.
- Make new pages easily discoverable: hand them over to sitemap and link submission, and keep the sitemap declaration in robots file.
In the AnQiCMS backend, robots configuration, pseudo-static rules, and sitemap generation are distributed in three locations. The advantage is that the output of each layer can be checked separately; the disadvantage is that it is easy to write the demands of one layer into another layer, such as writing a private instruction in the robots file for 'pages that should not appear'.
What else needs to be done beyond the declaration?
To fix the verification action into two steps: First, after modifying robots, observe whether the actual request distribution of various crawlers in the capture log has changed; second, check whether the generated sitemap still only contains addresses that can be accessed normally. Both are more indicative of the results than reading the configuration file.
In addition, do not rely solely on robots for controlling visibility. For content that should not be accessible to the outside, make the page itself inaccessible using access permissions or status bits, which is more reliable than depending on a single line of instructions.
Common questions
Question: Will there be any penalties for writing private instructions?Answer: There is no penalty, the problem is that it may not be executed, and it may even affect the parsing of the rest of the rules in the same file, causing you to think that the interception is effective when it is actually ineffective.
Question: How to control multiple spiders separately?Answer: Write paragraphs separately according to user groups, each group only uses the fields supported by the company's document; do not expect a general private instruction to cover all companies.
Question: How to confirm whether the instruction is read or not?Answer: Take the crawling log as the standard, compare the distribution of requests before and after the change; the text of the configuration file itself cannot prove the result.