How to split the index file when the site map entries reach the upper limit.
How to split with index files when the site map entry reaches its limit? In two steps: first, cut the addresses into several fragment files according to the column or publishing time, and then write an index file listing the addresses of these fragments. According to the standard of the day, the limit of a single file is 50,000 addresses and an uncompressed volume of 50MB. Either one reaching the top needs to be split. The index file itself does not list the page addresses directly, but only undertakes the directory responsibility.
What is the limit?
Two limitations are effective at the same time: a maximum of 50,000 addresses, and a volume limit of up to 50MB uncompressed. In practice, it is usually the number limit that is reached first; if the shard contains many fields or the addresses are very long, the volume may reach the limit first.
It is easy to misjudge the performance when reaching the limit: files are truncated, the end address is lost, or the system only recognizes the part within the limit after submission. Therefore, splitting should not be done when an error occurs, but should be planned and reserved for capacity at the stage of planning.
Split by what dimension?
There are four dimensions to choose from: by category, by publish date, by content model, and by language or site. The selection is based on 'which dimension has the least impact when regenerated.'
Categorization is most suitable for the backend structure, as only the corresponding slices are regenerated when the category content changes; date-based segmentation is suitable for information streams, as historical slices are no longer modified and only new slices are generated; content model segmentation is suitable for sites with large field differences; in multi-site and multi-language scenarios, segmentation by site or language allows each slice's address domain name to be unique, making it easier to troubleshoot.
Do not recommend splitting according to pinyin or random hash, the slices have no semantics, and it is impossible to locate the specific column when a problem occurs.
How to write the index file
The index file contains the addresses of the slice files, optionally with the last modification time of each slice. It does not directly list the page URL and should not be mixed with a common sitemap.
Only submit the address of the index file when submitting, which points to all the subfiles. This approach has three advantages: when a new slice is added, only the index file needs to be changed; when a slice has a problem, it can be replaced separately; the submission entry is stable and will not lose the registration record due to renaming.
Be careful not to write the jump address in the index file. The system's 301 redirect management is used to set URL redirection. If the path of the shard address has been changed, the index file should be updated to the target address after the redirect, otherwise it will consume an extra request each time the crawling is done.
How to push after splitting
Splitting solves the problem of full registration, and the acceleration of discovery for new additions and modifications still relies on pushing. AnQiCMS's link push management supports pushing new content to search engines to accelerate inclusion, including active push to Baidu and Bing; this is a parallel channel with the automatic generation of site maps.
The division of labor can be defined in this way:
| Fragmentation method | Applicable site form | Maintenance actions |
|---|---|---|
| By column | Column structure stable, large number of pages | Re-generate this fragment after the column content changes, update index file time |
| By release date | Information flow, frequent updates of blogs | Historical fragments remain unchanged, only generate the current period fragments and append them to the index |
| By content model | Products and document sites with large differences in fields | Re-generate the corresponding fragments after adjusting the model, to avoid mixed layout of overly long addresses |
| Language or site | Multilingual, multisite | Each shard address domain name is unique, convenient for separate troubleshooting of crawling failures |
Newly released documents are sent through active push, the full amount relies on periodic resubmission of shard files, and the purpose of the index is managed separately from the registered purpose, and they do not substitute each other.
How to verify after splitting
Look at three indicators: whether the number of addresses in each shard is within the capacity, whether the list of failed captures is concentrated in a particular shard, and whether the difference between the index volume and the number of shard addresses converges.
If only a certain shard has a high failure rate, the problem is generally in the generation rules of that shard, such as mixed in error pages or jump addresses; if all shard indices are low, the direction should return to content quality and on-site entry structure, rather than continuing to split.
Common questions
Question: How many layers can the index file be nested?It can be nested one more layer, but the deeper the level, the longer the crawling path, generally one layer is enough, and it is not recommended to use nesting.
Question: Does the shard address need to be renamed?Do not rename frequently. Renaming will make the original registration invalid and will also introduce jump consumption; if it is necessary to rename, synchronize update the index file and configure 301 redirect.
Question: Should the small station also be split?It is not necessary. When the number of addresses is far below the capacity limit, a single file is more convenient, and splitting is only beneficial when reaching or about to reach the top.
Question: How to maintain the last modified time in the shard?Fill in the actual generation time of the shard or the time of real content changes, do not reset all shard times to new values when rebuilding the entire site.