Before importing tens of thousands of documents in bulk, you need to confirm which few things

Before importing tens of thousands of documents in bulk, you need to confirm several things? The answer is four: whether the field mapping is fixed, how to handle documents with the same topic, how to divide the batch rhythm, and how to roll back after importing incorrectly. These things should be clear before starting, otherwise the import may turn into rework. AnQiCMS supports batch importing documents in ZIP compressed packages and Excel spreadsheets, and also provides an interface for importing document content for programmatic batch storage. Capacity is not an issue, almost all problems occur before the import preparation.

Determine the field mapping before importing.

Field mapping refers to 'which field in the system does each column of the source data correspond to', this step must be documented and cannot rely on on-site judgment. Common mapping errors include: automatically truncating the summary as the first paragraph of the main text, compressing multi-level categories into a string causing the category tree to not match, and synonyms in the label column not being unified causing the number of labels to explode.

Please confirm each item in the table below before taking action:

The last line is particularly important. Setting tens of thousands of articles to formal documents at once is equivalent to betting the collection and display of the entire site on a batch operation that has not been verified; a more secure approach is to first enter as a draft or pending publication status, and then batch convert to formal after sampling passes.

How to handle duplicate and topic conflict?

The most common issue in batch import is 'duplicate entry with the same title', usually from previous imports that left no records. AnQiCMS provides a document content import interface /api/import/archive, which supports checking if a document already exists by title, and is used for duplicate checking before batch import.

The recommended implementation sequence is as follows: First, run a query on the list of titles to be imported, dividing them into two groups: 'Existing' and 'Non-existing'; specify the handling strategy for the 'Existing' group (skip, update, or save as new), and only the 'Non-existing' group enters the import queue. This step may seem time-consuming, but it actually saves the cost of manually cleaning up duplicate pages after import.

In the same topic strategy, 'Update' means covering the existing text, which is the highest risk and it is imperative to make a backup first; 'Skip' is the safest but may cause gaps in content; 'Save As' will produce pages with similar structures, and subsequent redirection and merging will be required. In most cases, the combination of 'Skip + manual review list' is the most controllable.

The rhythm of importing in batches

Submitting all documents at once can be too large a scope when problems arise. Cut into batches by category or time, make a sampling check after each batch is completed, and then decide whether to continue.

• Batch size: limit it to the amount that can be sampled within a short time, rather than based on the system's capacity.
• Batch order: Run long-tail content first, leave the core pages for last, so that the mapping issues have been exposed in the first few batches;
• Batch record: Record the files, number, results, and sampling conclusions for each batch to form a traceable ledger;
• Interval observation: After each batch is completed, take a look at the crawling and access situation, stop first if there is an exception.

Leave a rollback point before import: AnQiCMS supports data and static file backup and recovery, a complete backup can take the content library and the static resources generated back at the same time, which is more reliable than revoking one by one later.

How to recover if the import is wrong?

Recovery is divided into two levels. Single or small-scale errors go to the recycle bin: document deletion is moved to the recycle bin rather than physical deletion, deleted documents can be restored, and draft and pending release status can temporarily remove error content without affecting the entire site.

Large-scale errors go through backup recovery. The criteria are whether the error spans multiple batches, and whether it affects the classification and tag structure. If so, the time cost of correcting each item is usually higher than directly returning to the backup point before import, and then re-importing according to the corrected mapping.

After recovery, there are two things to do: first, check the sitemap (sitemap) and link push status to ensure that the list of content submitted externally is consistent with the actual; second, write back the mapping issues exposed in the previous confirmation table to prevent the same mistakes from happening in the next batch.

Common questions

Which one should I choose, Excel or ZIP?

Use Excel for structured fields more often. Use ZIP when the relationship between the main text and the attachment is complex and the directory structure needs to be preserved. Both can be used together: first use Excel to build the structure and metadata, and then use ZIP to supplement the main text and images.

Can I publish at the same time during the import process?

It is not recommended. When batch writing is performed simultaneously with external services, pages that have not been verified may be crawled and indexed before the sampling is completed. It is more controllable to store them as drafts first and then convert them to official ones.

Will tens of thousands of articles slow down the site?

The main pressure is on the import window rather than long-term operation. It is more practical to execute in batches during off-peak hours, check the cache and static file generation status after import, rather than worrying about capacity.