Feature preview
The Sitemap protocol specifies the maximum limit for a single file: not more than 50,000 URLs, and the uncompressed volume should not exceed 50MB. After splitting, the index list should be submitted instead of individual subfiles, and the inclusion status of each subfile should also be checked. This article explains the principles of splitting, submission location, monitoring scope, and the division of labor with Baidu and Bing's active push.
What to do after splitting the sitemap into multiple sub-files and submitting for indexing monitoring? Three things: submit an index list instead of individual sub-files, check the indexing status of sub-files separately, and entrust the timeliness of new content to active push instead of waiting for the sitemap to be re-crawled. The split threshold given by the protocol is very clear: a single sitemap file does not exceed 50,000 URLs, and the uncompressed volume does not exceed 50MB.
First, split according to content type. Articles, products, documents, images, and attachments are separated into files. The advantage is that when troubleshooting, it can be located immediately: an anomaly in the collection of a certain type of content only needs to check the corresponding sub-file.
Second, split according to update frequency. High-frequency updated columns mixed with permanently unchanged archives in a file, the cost of re-parsing the entire file by the crawling party is very high, and the actual effect is that the high-frequency part is also slowed down.
An index list must be provided after splitting, listing the addresses of each subfile. There is no limit to the number of subfiles, but the entries in each index list are also subject to the same protocol constraints. If the scale is large, further stratification is required.
Submit each subfile separately to the webmaster platform. This is not an error, but it will waste quotas and monitoring perspectives: the platform needs an index list. After submitting the list, each subfile will be automatically discovered; submitting individually will cause each subfile to occupy independent quotas and status bits, making it difficult to maintain as the scale of the site grows.
The correct sequence of actions is: file generation at a fixed address, list all in the list, submit only the list address, and then check the status of the list itself to see if it is read normally.
AnQiCMS supports automatic generation of site maps, and files are updated with the site as columns and content change, without the need to maintain scripts manually. This is more critical in split scenarios: manually maintaining multiple sub-files and one list, the probability of missing updates increases linearly with the number of files.
| Monitoring items | What to look at | Common reasons for exceptions |
|---|---|---|
| Status of list reading | Whether the list has been crawled, and whether the number of entries is consistent with the number generated | List address changed and not resubmitted |
| The inclusion rate of each sub-file | The ratio of included to discovered but not included | Duplicate content or insufficient quality of pages |
| Time delay for discovery of new content | Time difference between publication and first crawling | Low crawling frequency, needs to be handed over to active push |
| Proportion of invalid addresses | Entries with error status codes returned in the list | Not synchronized after deletion or revision |
The most worth watching is the last item. Error addresses in the sitemap will not directly affect ranking, but they will continue to consume crawling quota and distort monitoring data - you think the decrease in inclusion rate is a content issue, but actually it is a batch of expired addresses in the list.
Mention another entry for crawling quota: Do not mistakenly block the paths to be crawled in Robots.txt. AnQiCMS supports backend Robots.txt configuration, be sure to retrieve the file with a public address to confirm the effectiveness of the content after changes, as the cache layer may continue to return the old version.
Sitemap solves the “complete list”, active push solves the “new links found quickly”. Both are complementary relationships: Only relying on the sitemap, new content has to wait until the next file is crawled to be discovered; only relying on push, historical content and unpushed columns are ignored.
AnQiCMS link push management supports Baidu and Bing active push, send out the address after publishing or updating to accelerate inclusion; automatic generation of site map is responsible for providing a full list periodically. In daily operation, a more practical combination is to push new content, and the parts that fail to push or have not reached the amount are covered by the site map.
Static links also play a role here: after the URL structure is unified into a stable form, the addresses in the site map will not fail in batches due to revision, and 301 redirect is responsible for converging historical addresses to new addresses.
Question: Is the site small and needs to be split?Answer: No need. The benefits of splitting come from scale, it's more convenient for a small station to add a file and submit once.
Question: Is it better to have more sub-files?Answer: No. Splitting to 'can locate problems by type' is enough, over-splitting will make list management and monitoring more complex without increasing benefits.
Question: How can you confirm that the push has been processed?Answer: Check the success and failure counts returned by the platform, and sample-check whether the push address can be found in the platform's collection status; just looking at 'submission successful' is not enough, successful submission does not mean it has been processed.
Preview
Today's statistics show that more than 60% of form submissions without protection are spam or machine behavior, and the captcha itself can be bypassed. This article gives four levels of checks beyond the captcha: hidden fields and frequency control, whether the submission target can be listed, content layer sensitive word filtering and review, as well as source protection for messages and comments.
Preview
The most difficult thing when being collected in bulk is to prove the source. The idea of interference codes is to let the copied copy carry invisible noise and recognizable fingerprints, which can prevent direct copying collection, but cannot prevent manual rewriting and interface-level grabbing. This article explains the scope of the function of interference codes, the judgment of the collected signal, and the safety closures related to anchor text and content collection features
Preview
Manage multiple brand stations with a single background, isolation should be configured according to the site boundary, role boundary, and operation domain boundary in three layers. AnQiCMS supports multi-site management and is suitable for multi-brand and multi-theme scenarios; user management and VIP group are responsible for setting permissions by group, and operations such as backup, upgrade, and multi-site are by default closed at the station level, and opened explicitly as needed.
