Before batch import, check the existence of the title first, how to resume interrupted batches

Before batch import, check the existence of the title first, how to resume interrupted batches, the correct order is 'first get the list, then compare the titles, only fill the gaps'. The AnQiCMS document content import interface /api/import/archive supports querying whether the document already exists by title, the resumption should be based on this query action to rebuild the pending list, rather than running the original file from the beginning again.

What does the query action of the import interface do

The import and archive actions have different duties: archive is responsible for writing the entire package into the document library, while the import side query is responsible for answering 'Does this title exist?'. In batch scenarios, the latter is more critical, because with thousands of entries being imported at once, the break point is almost impossible to fall on the boundary of an entire article, usually it breaks off at some point when the network or process fails.

The first step of the resumption is to take the completed list.发起一次存在性查询to each title in the source file, and skip the entries that already exist, only entering the second batch for those that do not exist. This way, even if the previous batch is interrupted in the middle, there will be no duplicate entries with the same title.

Two types of entry: compressed packages and tables.

The batch import supports two methods: ZIP compressed packages and Excel spreadsheets, and they cannot be mixed during resumption. The compressed package entry includes files such as the main text and illustrations of this type of document structure, and the directory hierarchy within a package will be read as the document's attribution; the spreadsheet entry includes field mappings for one article per row, and columns such as titles, categories, and tags need to be aligned in advance.

The trouble brought by mixing is not an error, but a silent misalignment: when resuming transmission with the table entry for the same batch of content, the attachment path in the text may not be included. Therefore, when resuming, the same entry as the previous batch must be used, and the field descriptions of the two entries in the interface document should be kept at hand for verification.

The order of resumption after batch interruption

Follow this order:

Stop the original task to avoid writing to the same category at the same time. Then export the title list from the source file, query each title one by one, and divide the results into two groups: '入库' and '未入库'. The third group is可疑 items, that is, entries with titles that exist but have obvious discrepancies in the length of the text. These need to be reviewed manually and should not be overwritten directly.

Only resubmit the '未入库' group. When submitting, throttle according to the API call frequency, sending in small batches is easier to locate breakpoints than pushing all at once. In case of interruption, the last successful entry can be directly seen in the logs.

How to clean up incorrect repeated imports

When the same title is duplicated during the import, the way to handle it is to delete the extra copy instead of changing the title. Deleting official documents in AnQiCMS is to move them to the recycling bin instead of a physical deletion, so the cleanup action can be rolled back: first confirm that the content of the retained copy is complete, then move the duplicate items to the recycling bin, and observe for a period of time before doing a complete cleanup.

The 'title existence check' before the batch import was to eliminate this cleanup step. The cost of the query action is much less than the cost of manually comparing titles one by one after the event, especially when running the same source file repeatedly on the same site.

Common questions

Will the difference in the title be unable to be found?

Will. Search by title matches literally, any extra space or full/half width punctuation will be considered non-existent. Standardize the title before comparison.

Should the entire package be regenerated after interruption?

Not necessary. The ZIP package itself has no state, reassembling the package only increases the chance of errors. Use the original package to transmit the missing list filtered by title search.

How long will it take for several thousand records?

It depends on the round-trip times of writing and querying per record, there is no fixed value. A safe approach is to run a small batch to measure the time first, then arrange the subsequent batches according to linear relationships, and set the time factor during off-peak hours.