The content has been moved away from the entire site, but the traffic report does not show any abnormalities, so first check which level.
The site content was moved away as a whole, but the traffic report did not show any abnormalities - this is the normal form of such accidents, not that the report failed. Which layer should the investigation start with?The order is below. The reason is that the collection behavior naturally bypasses front-end statistics: the collector uses scripts to pull the text page by page from the list page, the requests do not pass through埋点、do not leave source records, and even if they fall into the number of visits, they are also drowned in the normal collection and crawler traffic.And the search exposure obtained by the plagiarism station is recorded on the other party's report, and will never appear in your numbers. Therefore, the investigation cannot start from the report, the correct level is from bottom to top: first look at the crawling and access logs, then look at the signs of the search phase, and finally it is the protection and disposal.
First layer: crawling and access logs
Batch transfer in the form of logs is very stable, look for three traces. The first is the arithmetic traversal of list pages and content pages in a short period of time - the same IP scans articles by number or pagination parameters within a few minutes, with page numbers continuous, evenly spaced, and a single UA.The second is a request for 'read-only text' that is concentrated in exceptions: only pulling the detail page without pulling styles and images, indicating that the requesting party only wants text. The third is a comparison of robots records: crawling class UA will declare identity according to the rules, while搬送脚本 often ignores or forges UA;The paths explicitly prohibited in the site's Robots.txt configuration are still frequently accessed, and can basically be excluded from normal crawling.This level's output is 'Who is pulling, and which pages are being pulled away,' which is the basis for all subsequent actions.
Second layer: signs in the search phase
Estimate the victim area with two low-cost signals. The first is the in-site search term: After the content is moved, the original reader will search for the complete sentence to find you. If
Which layer does the interference code manage?
Confirm after being moved, the protective measures directly counter the anti-capture interference code: AnQiCMS's built-in interference code adds content that is only taken away by the capture script and is not visible on the browser side during page rendering. When the script pulls the entire site, it also copies these interferences. Meaningless segments mix into the main text of the copied site, and the quality of the finished product is destroyed.To understand its scope of action: The interference code protects the
Don't treat the collection ability as a protective layer
A common concept reversal: treating 'content collection' as protection. AnQiCMS's content collection is the ability to automatically import content from external websites, which is the opposite direction of anti-crawling (anti-being taken), and the two are often confused with the same name.The allocation after sorting is: The discovery layer relies on access logs and search clues; The countermeasure layer relies on anti-crawling mechanisms such as interference codes, plus complaints and bans against confirmed abusers;The rule layer relies on robots and site statements to define the crawling boundaries. When compiling the team's incident manual, write actions according to these three layers, do not write 'Turn on anti-crawling and it will be safe' - single-layer plans will fail in such incidents.
Summary of troubleshooting sequence
Execute in this order: First, pull access logs, aggregate by IP to find arithmetic progression and read-only text request patterns; second, check the deviation between robots.txt declaration and actual crawling behavior; third, search for whole sentence hits in in-site search terms; fourth, search externally to extract core articles for duplicate page comparison; after confirming the victim in two steps, enable interference codes to mitigate losses, leave evidence, and file a complaint. The front-end report does not need to be touched throughout the process - it is not a basis for judgment at any stage.
Common questions
Question: Isn't it more convenient to block for a while with CDN?Answer: It can reduce the load, but it cannot stop the script that mimics normal access rhythm, nor does it solve the fact that historical content has been moved; it can be part of a disposal combination, but cannot replace log positioning.
Question: Should a complaint be filed for the moved page?Answer: File a complaint after securing evidence: a time-stamped original page snapshot, the crawling evidence of the other page, and the corresponding moving window records in the log; the three sets together will result in the highest complaint efficiency.
Question: What should be done if the small station does not have the ability to view logs?Answer: Reduce the frequency by one level: Aggregate the header traffic of access logs by IP once a week, add a monthly check of duplicate pages with full titles, both tasks take only a few minutes, which is enough to find the whole station transfer.