How to control the crawling depth of list pages when pagination and infinite scrolling coexist
How to control the crawling depth of list pages when pagination and infinite scrolling coexist? The key point is to entrust the crawling depth to the internal links that can be crawled: infinite scrolling can only be used as progressive enhancement, and each screen of content must be returned at a real pagination address with complete HTML and with links pointing to the detail page. The pagination relationship tag itself no longer bears the role of inclusion signal, and hoping it can pull deep pages into the index is the most common misjudgment of small websites.
How is the crawling depth formed
The crawler starts from the homepage and moves layer by layer along the links. The number of jumps from a page to the entry is its crawling depth. The list page is the hub of this chain: which page of the column homepage can still be caught along the way depends on whether each page leaves links to the next page and the detail page. When only scrolling loading is used, the links after the second screen do not exist in the original HTML, and the depth is truncated here.
Footer pagination for infinite scrolling
The criteria are very simple: turn off the script, and see if this screen's content is still there. The safe way is to let scrolling and pagination share the same set of data, each page has an independent address, returns a complete list of full text and entry links, and scrolling only adds the content of the next page to the current address. The three writing methods below have明显 differences in controllability.
| Pagination style | Address form | Crawl accessibility | Applicable scenarios |
|---|---|---|---|
| Path-based pagination | Column hierarchy progression | Leave a link to the next page on the previous page to go deeper on the way | Column article quantity is moderate |
| Query parameter pagination | Add page number parameters to the same address | Parameters need to be fixed to avoid combinatorial explosion | Conform to the static rules for address convergence |
| Load only by scrolling | No independent address | Deep content is not accessible after the script is closed | Not recommended for use alone |
Complement deep content with internal links and sitemaps
In addition to pagination, two fallbacks are needed: first, anchor text should be used to hang deep entries to the column's first screen or aggregation page, so that important content does not depend on the number of page layers; second, only standard pagination addresses and formal document addresses should be listed in the site map, and temporary addresses generated by scrolling and duplicate addresses with filtering parameters should not be included. The pagination address on the list page should be consistent with the standard address; otherwise, even if the control of the crawling depth is good, the included results will be scattered across multiple similar addresses.
The parameters should be fixed to avoid combination expansion
Once pagination parameters are allowed to be stacked freely, multiplying page numbers with sorting and filtering will generate a large number of approximate pages, and crawling quotas will be consumed on duplicate content. The safe order is: first determine which parameters participate in the address, and then use pseudo-static rules to converge the parameters involved in the address into a fixed form. The rest of the parameters only take effect in the session and do not appear in the links that can be crawled.
Common questions
Will infinite scrolling definitely harm the inclusion?Not necessarily. As a progressive enhancement and when each screen can be accessed with complete HTML at an independent pagination address, crawling is not affected.
Do you still need to write pagination relationship tags?It is no longer a signal for collection, can be used as a supplementary readability, but cannot replace the real link and pagination address.
How deep should the page numbers go?Based on the actual content volume, it is better to display deep entries on the first screen than to expand empty pages and thin pages all the way.
Should the sitemap list every page?Generally, only the home pages of columns and standard pagination entries are listed, and fully铺page numbers will dilute the focus of crawling.