AnQiCMS process crashed under high load, what clues can the PID check log provide?

As a website operator who has been deeply involved with AnQiCMS for many years, I know that logs are our most loyal partners when facing the stability challenges of the system under high load. When the AnQiCMS process crashes unfortunately under high load,PIDCheck the logs, especiallycheck.logwhich can provide us with key initial clues, guiding us in deeper problem diagnosis.

Understand AnQiCMS's process management and log mechanism

AnQiCMS is an enterprise-level content management system developed based on the Go programming language, known for its high concurrency and performance characteristics. To ensure the system runs stably under various conditions, AnQiCMS usually works with a guardian script, such asstart.shto monitor the running status of the main process.

start.shOne of the core responsibilities of the script is to periodically check if the AnQiCMS main process exists.If the process is detected to have stopped, it will try to restart AnQiCMS.During this check, the script will write the check results to a file namedcheck.logfile. At the same time, when the AnQiCMS process is started, its standard output and standard error will be redirected torunning.logThis file contains detailed information about the application's runtime and any error reports.

PIDCheck the logscheck.logProvided preliminary clues

check.logThe file acts as a "sentinel" role, recording the survival status of the AnQiCMS process. It is instart.shIn the script, each check records the number of current AnQiCMS processes, usually in the form ofPID anqicms check: Xof whichXwhich is the current one foundanqicmsnumber of processes.

direct indication of process status

Under normal operation of AnQiCMS,check.logrecorded inXThe value should always be greater than 0 (usually 1, indicating that the main process is running). If we find in the logs thatPID anqicms check: 0The record, this is a clear signal that the AnQiCMS process has stopped running during the check. Under high load scenarios, if the process was normal before, then suddenly it appearedX=0The record indicates that the AnQiCMS process has almost certainly crashed.

The time and frequency of crashes occurred.

check.logEach record carries a timestamp. By analyzing these timestamps, we can precisely grasp the specific time point of the AnQiCMS process crash. Ifcheck.logRepeatedly stop and restart the process (i.e., frequently occur)PID anqicms check: 0It then returned to normal, suggesting that the system may have instability issues and may be related to peak website traffic, scheduled task execution, or other predictable external events.These time points are crucial for associating issues with specific operations, system load patterns, or fluctuations in external dependencies.

Identify abnormal restarts

Combine with system startup scripts (such asstart.shThe execution frequency (for example, if it is configured to be executed once a minute by a cron job),check.logCan help us determine if the process has unexpectedly restarted. OnceX=0records will usually followanqicms NOT runningandnohup ... &attempted restart information. By observing these patterns, we can confirm whether the crash was detected and attempted to recover by the guardian script

Combinerunning.logcarry out in-depth analysis

Althoughcheck.loginform usIs AnQiCMS process crashingBut it cannot explainWhy is it crashing. To understand the root cause of the crash, we need to turn torunning.log.

running.logContains standard output and standard error information generated by the AnQiCMS application during runtime. When a Go application encounters runtime errors (such aspanic）Or when encountering other fatal issues, detailed stack trace information and error messages are usually recorded in this file. Common causes of crashes in high-load scenarios may include：

• Go Runtime PanicThis is the most direct crash manifestation of the Go program, usually accompanied by detailed Goroutine stack information. It may be due to null pointer references, array out of bounds, concurrency safety issues (such asmapConcurrency read-write without lock) etc.
• Memory overflow (OOM - Out Of Memory): Although Go language has a garbage collection mechanism, under high concurrency, if the number of Goroutines increases sharply, there are memory leaks, or handling a large amount of data leads to rapid growth in memory, it may still lead to the system running out of memory and eventually being killed by the system.running.logIt may not directly display OOM, but a reduction in log volume before the application stops or errors indicating resource allocation failure may be clues.
• Database connection pool exhausted or deadlock: AnQiCMS relies on the database, under high load, if the database connection configuration is not properly set, it may lead to the exhaustion of the connection pool, making it impossible to obtain a database connection, which in turn may cause application errors.A database deadlock can also cause requests to be blocked for a long time, and accumulating too many requests can eventually cause the application to crash.
• Timeout or error from external service: AnQiCMS may depend on external APIs, caching services, or file storage.If these external services respond slowly or return errors under high load, and AnQiCMS does not handle them properly, it may also cause itself to crash.
• Go Goroutine leak: A Goroutine is a lightweight thread in the Go language.If a Goroutine is created and fails to exit normally, and continues to consume resources, prolonged accumulation can lead to exhaustion of system resources.running.logMay not directly display the leak, but will display other resource-related errors orpanic.

By adding inrunning.logsearch inpanic/fatal error/runtime error/out of memory/timeout/too many connectionsBy using keywords, we can locate the specific error information and code location when a crash occurs, thereby providing the most valuable basis for problem diagnosis.

Comprehensive troubleshooting steps and suggestions

Whencheck.logDirect AnQiCMS process to crash under high load, I recommend you follow the following steps for comprehensive troubleshooting:

1. Confirm the crash time and frequency: Carefully checkcheck.log, Record allPID anqicms check: 0The time point, and compare it with the website traffic curve, system resource monitoring data (such as CPU, memory, network I/O, disk I/O) and any known scheduled task execution time.
2. In-depth analysisrunning.log: Locate to the time point before and after the crashrunning.logContent. Look for any exceptions, errors, warnings orpanicStack information. This is a critical step in diagnosing the root cause of the problem
3. System resource monitoringCheck the CPU utilization, memory usage, network bandwidth, and disk I/O before and after the server crashes.Under high load, the memory growth curve of Go applications is worth paying special attention to, as it may indicate Goroutine leaks or OOM.
4. Database performance check: View database logs and performance metrics, including connection counts, slow query logs, lock waits, and more.Ensure the database can stably support AnQiCMS requests under high load.
5. Health status of external dependenciesIf AnQiCMS depends on other services, check the logs and health status of these services.For example, caching services (such as Redis), message queues, image processing services, or third-party APIs.
6. Review AnQiCMS configuration: Checkconfig.jsonThe related configurations, especially those related to concurrency, caching, timeouts, and database connection pools.不合理配置under high load may become a bottleneck.

By this systematic method, we willcheck.logact as an initial abnormal signal,running.logAs a core diagnostic tool, combined with comprehensive system resources and external dependency monitoring, we can more efficiently locate and resolve the crash problems of AnQiCMS under high load.

Frequently Asked Questions (FAQ)

AnQiCMS process crashed under high load, what clues can the PID check log provide?

Q1:check.logfrequent appearance insideexists=0Howeverrunning.logno obvious error message, what should I do?

A1:This may be because the application was forcibly terminated by the system under high load, such as due to memory exhaustion (OOM Killer) or reaching process resource limits. In this case, the application may not have had a chance to write detailed error information.running.log. You need to check the system's kernel log (for example, on Linux, the/var/log/messagesordmesgCommand output), check for records of OOM Killer or notifications of terminated processes. Also, checkulimitAre system-level resource limits too low. In addition, to increaserunning.logThe detail level (if AnQiCMS supports) or adding a more robust error handling and logging mechanism may be helpful.

Q2: Why might AnQiCMS crash after running for a while after reboot?

A2:This phenomenon usually points to a resource accumulation problem or a defect in a specific code path under high load. Common causes include:

1. Memory leak: The application may have a memory leak, causing memory usage to gradually increase after each restart, eventually exhausting system resources.
2. The database connection has not been released.: The database connection may not have been closed or returned to the connection pool correctly, which can lead to a rapid increase in the number of connections under high concurrency, and ultimately cause a crash.
3. Goroutine leak: Go language's Goroutine, if it does not exit normally, will continue to occupy memory and other resources.Under high load, a large accumulation of Goroutines can lead to a sharp decline in system performance and eventually crash.
4. The external service is unstable: AnQiCMS relies on external services (such as Redis, remote APIs) that may be intermittent and unstable, causing cumulative effects during system operation.

Q3: Besidescheck.logandrunning.logWhat other logs can help troubleshoot the problem?

A3:In addition to the logs of AnQiCMS itself, the following important system and application logs are also available:

1. System Logs (System Logs): such as on Linux/var/log/syslog//var/log/messagesOr throughjournalctlView. These logs record the events of the operating system, including OOM Killer, hardware failure, network problems, etc.
2. Web Server Logs (Nginx/Apache Logs): If you use Nginx or Apache as a reverse proxy, their access logs (access.log) and error logs (error.log) Can help you understand the traffic pattern of requests, response time, and errors at the proxy level.
3. Database Logs (MySQL Logs): Database error logs, slow query logs, and binary logs (if enabled) are crucial for diagnosing performance bottlenecks or errors at the database level.
4. Cloud service monitoring logsIf AnQiCMS is deployed on a cloud platform, the cloud service provider's monitoring and logging services (such as AWS CloudWatch, Alibaba Cloud Log Service) can provide a more comprehensive set of resource metrics and application log aggregation analysis.