If the server resources are tight, how will the PID log show when the AnQiCMS process is killed by the system OOM (Out Of Memory)?

As an experienced website operations expert, I am well aware that the health of server resources is crucial for the stable operation of a CMS system, especially for systems like AnQiCMS that emphasize high performance and concurrent processing.When server resources, especially memory, are tight, the system kernel may terminate processes that use too much memory without mercy to maintain overall stability, which is what we often call the 'OOM (Out Of Memory) kill' process.

So, when the AnQiCMS process unfortunately gets killed by the system OOM, how should we capture these 'exceptional signals' in the logs?This usually wouldn't leave a detailed 'will' like a program exiting actively, but by observing the AnQiCMS itself daemon process logs and system kernel logs, we can still piece together the truth of the event.

OOM-Killed AnQiCMS: A quiet 'death' and a hurried 'rebirth'

When the system faces the crisis of memory exhaustion, the OOM Killer mechanism of the Linux kernel will be activated, which will select and kill one or more processes with high memory usage according to a complex scoring algorithm (OOM Score) to release resources and avoid the complete collapse of the system.For AnQiCMS such a Go language application, being killed by OOM means that the process terminates suddenly without any warning, and it does not even have time to perform any cleanup operations, let alone record the specific reason why it was killed by OOM in its application logs.

However, AnQiCMS usually uses a watchdog script (such as the one mentioned in the document)start.sh) to ensure its continuous operation. This script periodically checks for the existence of the AnQiCMS process, and if it finds that the process has unexpectedly terminated, it will immediately attempt to restart it.This is the mechanism of 'revival from the dead' that left a key clue in the logs for us to trace the OOM events.

Tracing the 'traces' in the log: Tracking abnormal interruptions of AnQiCMS

To determine if AnQiCMS has been killed by OOM, we need to pay attention to two types of logs: AnQiCMS's own daemon process logs (especially those responsible for checking and restarting) and the system kernel logs.

1.check.logAbnormal pulse in heartbeat record:

According to the deployment method of AnQiCMS (especially throughstart.shLinux deployment managed by script), there will usually be acheck.logOr similar log files, recording the information of the guardian script periodically checking the process status of AnQiCMS.

Normalcheck.logThe record will show that the AnQiCMS process ID (Process ID) persists, for example:

20240723 10:00:01 anqicms PID check: 1
20240723 10:01:01 anqicms PID check: 1
20240723 10:02:01 anqicms PID check: 1
...

Once the AnQiCMS process is killed by OOM, the guardian script will find that the original PID does not exist (orps -efUnable to find, and will try to restart it. At this time,check.logAn abnormal pattern will appear in it:

20240723 10:03:01 anqicms PID check: 1  # 进程正常运行
20240723 10:04:01 anqicms PID check: 0  # 进程被杀死，检查发现不存在
20240723 10:04:01 anqicms NOT running # 守护脚本记录进程未运行
20240723 10:04:01 (启动命令...) # 守护脚本尝试启动新进程
20240723 10:05:01 anqicms PID check: 1  # 新进程启动，新的PID出现
...

You will see a PID change from '1' to '0', followed by the 'NOT running' prompt and the startup command, and then the new PID appears again.This "PID disappearance-restart-new PID appearance" pattern is a strong signal that the AnQiCMS process has been interrupted abnormally and has been automatically restored by the guardian script.Although it does not directly indicate OOM, but combined with the sudden stop of the application log, it points to the possibility of system forced termination.

2.running.log: A sudden stop of tranquility

running.log(Or the actual output log path of the AnQiCMS application) usually records the operation status, request processing, and error information of the AnQiCMS application.When the AnQiCMS process is killed by OOM, due to the sudden termination, the application has no opportunity to write any shutdown or error information.

Therefore,running.logThe characteristics manifested in it are:

• The log output suddenly interruptedBefore the OOM occurred, the logs may still be outputting normally, but after that, they would suddenly stop, without any logs indicating an 'graceful shutdown' or 'error exit'.
• Log timestamp jumpAfter the guardian script restarts AnQiCMS, a new process will start writing logs.You will find that the log timestamp suddenly jumps to the time after the restart, and the context of the new log has no relation to the old log, as if it is a completely independent session.

This log's 'cliff-style' interruption is a strong indication that the application process was forcibly terminated by external forces.

3. System kernel log: A conclusive 'death certificate'.

To obtain the final and most convincing evidence of the AnQiCMS process being killed by OOM, we need to check the system kernel log.These logs record all important events at the system level, including the activities of OOM Killer.

In the Linux system, you can find the relevant logs at the following location:

• /var/log/syslogor/var/log/messages(Different according to the Linux distribution)These are the main system log files.
• dmesgCommand output:dmesgDisplay information about the kernel ring buffer, which includes detailed records of OOM events.

You can usegrepUse commands with keywords to search for OOM events. For example:

grep -i 'oom|out of memory|killed process' /var/log/syslog

Or

dmesg | grep -i 'oom|out of memory|killed process'

When AnQiCMS is killed by OOM, the kernel log usually contains records similar to the following:

kernel: Out of memory: Kill process 12345 (anqicms) score 999 or sacrifice child
kernel: Killed process 12345 (anqicms) total-vm:4123456kB, anon-rss:3987654kB, file-rss:123456kB, shmem-rss:0kB

Among them:

• process 12345 (anqicms)It clearly points out the name and PID of the killed process.
• Out of memoryorKilled processDirectly indicates the nature of the event.
• total-vm/anon-rssWill display the memory usage of the process before it is killed, helping you analyze which type of memory is using too much.

These kernel logs are the golden standard for diagnosing OOM problems, they provide direct evidence and contextual information of the process being forcibly terminated.

Why can AnQiCMS written in Go language also be killed by OOM?

Although Go language is known for its efficient memory management and lightweight Goroutines, this does not mean that Go applications are immune to OOM. When AnQiCMS runs in the following scenarios, even Go applications may face OOM risks:

• High concurrency and instantaneous traffic peak:Although Goroutines are lightweight, if a large number of requests flood in all at once, each Goroutine allocates a small amount of memory, and the cumulative total may quickly exceed the available physical memory.
• Handle large files or large datasets:When AnQiCMS needs to handle large files uploaded, carry out a large amount of content collection or batch import, or operate ultra-large data structures in memory, it may occupy a large amount of memory in a short period of time.
• Memory leak (rare but still possible):Even though Go has a garbage collection mechanism, if the program logic is not designed properly, such as holding references to objects that are no longer needed for a long time, or handling interactions with Cgo incorrectly, it may still lead to memory not being released in time, causing memory leaks.
• Insufficient server resource allocation:The most direct reason is that the memory allocated to AnQiCMS by the server is not enough to support its normal operation and business peak.

Summary

When the AnQiCMS process is killed by OOM, the most direct clue ischeck.logthe sudden change in PID, as well asrunning.logThe output of the log stopped abruptly. But to diagnose an OOM event, the system kernel log (such assyslogordmesgThe clear records of "Out of memory" or "Killed process" are the most critical "death certificates".As a website operations expert, deeply understanding these log signals can help us quickly locate problems, optimize system configurations or application code, and ensure the stable and efficient operation of AnQiCMS.

Frequently Asked Questions (FAQ)

1. Ask: AnQiCMS'check.logthe PID changes frequently, does it necessarily mean OOM? Answer:It is not necessarily. Frequent changes in PID are indeed signals of abnormal interruption, in addition to OOM, it may also be caused by other reasons leading to process crash or termination, such as logical errors in the application's own code (Go language's panic), manual termination of the process by other system administrators, or server hardware failure, etc.However, combinerunning.logThere is no sudden interruption with no error stack or exit information, and whether there are OOM records in the system kernel log