How to handle duplicate articles during batch import, where should the title duplicate check be placed

How to handle the situation when the same topic article already exists in the library during batch import? The answer is to search first and then write: the search end will see multiple similar pages, the internal anchor text and site map will dilute the weight at the same time, and readers will find it difficult to judge which article is the current version. The reasonable order is to check for duplication first, then submit, and then verify, so where to call the title duplication check interface has also been concluded - it should be placed before assembling the data, rather than cleaning up after an error.

The three types of consequences brought by duplicate content

The first type is scattered collection. Two pages with the same theme compete with each other, and the external links and internal links are separated, making it difficult to get on both sides. The second type is maintenance fragmentation. Only one article was changed during the revision, and the other continued to be published with the old criteria, resulting in inconsistent answers for readers. The third type is statistical distortion. The number of visits and citations were split into two addresses, and the subsequent judgment of content quality had a low reading.

Different sources have different repetitive forms. Migrating historical content is easy to bring out the old version; multi-source collection is easy to bring out the same question with different structures; when multiple people collaborate, the mixing of renaming and creating new ones will also result in cases where the title is similar but the content is different.

The order of checking for duplicates before submission

The first step is to organize the titles of the items to be imported into a list, retaining the original title and the normalized comparison key, to avoid them being considered as two different titles just because of a single space or punctuation. The second step is to call the query interface item by item to confirm whether the title already exists in the database, and divide the results into new, updatable, and manually judged categories. The third step is to assemble the text, classification, and tags and submit them.

The interface for importing document content within the site, located at /api/import/archive, supports querying whether the title already exists. These two tasks need to be combined to place duplicate checking before writing. If only writing is done without querying, duplicates will appear in the official database; if only querying is done without storing in the database, a conflict list still needs to be manually processed.

Differences in the three batch routes

There are two routes for compression packages and tables, each with its own applicability: the former brings attachments and images together, and the latter is superior in clear structure and easy for manual verification. The cost of the interface route is that you have to write retry and batch logic yourself, and the benefit is that duplicate checking can be done on each item.

Leave an interval for verification when submitting in batches. Once all the content is pushed in, it will take a second look for verification, and repeated and incorrectly categorized items will be mixed together; proceed by batches, check the quantity and status after each batch is completed, and the problem can be located within the batch.

Status check after import

Check the status distribution after the import is completed. The document status is divided into four categories: formal documents, drafts, pending publication, and recycling bin. The results of the batch write should first fall into the draft, and then be released after confirmation to avoid half-finished products from directly entering the visible range to the public. The draft can be previewed to see the display effect without needing to be published first.

Please check the deleted records. The deleted documents go to the recycle bin instead of being physically deleted. This step determines whether deleted items can be recovered, and it also means that after duplicate content is cleaned up, it still needs to be confirmed in the recycle bin once, don't leave placeholders. Take a sample of the presentation of categories, tags, and site maps to see if the newly imported entries appear as expected in the list and navigation.

After checking, release the status in batches, and finally trigger a link push to make new and updated content be timely perceived by search engines.

Common questions

Is it considered a duplicate if the titles are similar but not completely the same?Judging by the normalized comparison key, those that only differ in punctuation and spaces should be considered the same article, while those with different main themes are a different matter and require manual decision-making.

If the import fails, should we redo the whole batch?It is not recommended. Locate failed entries in batches, remove or correct them, and then try them separately to avoid rewriting parts that have already been successfully written.

Will checking for duplicates first slow down the import?It requires one more query call, but it saves time for post-cleaning, which is especially noticeable when there are thousands of entries.