How to modify the `start.sh` script so that it automatically sends an alert email when AnQiCMS fails to start?

As a senior CMS website operator, I am fully aware of the importance of system stability for business operations.Faced with the possible boot failure of AnQiCMS, establishing an automatic alarm mechanism is the key to ensuring service continuity.This article will detail how to modify AnQiCMSstart.shScript to send an alert email in a timely manner when the startup encounters an exception, so that the operations team can obtain information and take countermeasures at the first time.

Understand AnQiCMS'start.shscript

During the deployment of AnQiCMS, we usually take advantage ofstart.shscripts to check and start the AnQiCMS application. According to the provided documentation, the originalstart.shThe main function of the script is to determine whether the AnQiCMS process is running, and if it is not running, try to start it. This script is usually executed throughcronThe task is executed every minute to ensure the service's automatic recovery capability. The core logic of the script is to useps -ef | grep '\<anqicms\>' |grep -v grep |wc -lcommands to countanqicmsthe number of processes, if the number is zero, then executenohup $BINPATH/$BINNAME >> $BINPATH/running.log 2>&1 &Command to start the AnQiCMS application.

Although this script can automatically attempt to restart the service, it lacks the ability to notify operations personnel when the startup fails.This means that if AnQiCMS tries to start multiple times consecutively and still fails, we may not know in time, which may lead to a long service interruption.In order to solve this problem, we need to introduce an email alerting mechanism.

Prerequisites for introducing the email alert mechanism.

Before modifyingstart.shBefore the script, we need to ensure that the server environment has the ability to send emails. The most common way is that the Mail Transfer Agent (MTA) is already configured on the server, for examplesendmail/Postfixormsmtpand it was installedmailcommand (usuallymailxThe package provides). If the server has not configured the mail sending service, you need to install and configure it according to the Linux distribution you are using.In simple terms, to execute in the command lineecho "Test" | mail -s "Test Subject" [email protected]If the email can be received successfully, it means the email environment is ready

If the server cannot configure the email client, you can also consider usingcurlCommand with the email service provider's API, or write a simple Python script to send an email via SMTP. But in order to maintainstart.shThe simplicity and versatility of the script, we recommend using itmailcommand.

modifystart.shScript to implement alarm function

Now, we will address the originalstart.shThe script has been modified, adding fault detection and email alert logic. Below is the modified script content and a detailed explanation of the new sections.

#!/bin/bash
### check and start AnqiCMS with email alert
# author fesion
# the bin name is anqicms
BINNAME=anqicms
BINPATH=/www/wwwroot/anqicms

# --- Email Alert Configuration ---
RECIPIENT_EMAIL="[email protected]" # 接收告警邮件的邮箱地址
SENDER_EMAIL="[email protected]" # 发送告警邮件的邮箱地址
ALERT_SUBJECT="AnQiCMS 服务启动失败告警" # 告警邮件主题
# ---------------------------------

LOG_FILE="$BINPATH/running.log"
CHECK_LOG="$BINPATH/check.log"
TIMESTAMP=$(date +'%Y-%m-%d %H:%M:%S')

# Check if AnQiCMS is already running
exists=`ps -ef | grep '\<anqicms\>' |grep -v grep |wc -l`
echo "$TIMESTAMP $BINNAME PID check: $exists" >> "$CHECK_LOG"
echo "PID $BINNAME check: $exists"

if [ $exists -eq 0 ]; then
    echo "$TIMESTAMP $BINNAME NOT running, attempting to start..." >> "$CHECK_LOG"
    echo "$BINNAME NOT running, attempting to start..."

    # Attempt to start AnQiCMS
    cd "$BINPATH" && nohup "$BINPATH/$BINNAME" >> "$LOG_FILE" 2>&1 &

    # Give AnQiCMS a moment to start up
    sleep 10

    # Re-check status after attempted start
    exists_after_start=`ps -ef | grep '\<anqicms\>' |grep -v grep |wc -l`
    echo "$TIMESTAMP Re-check $BINNAME PID: $exists_after_start" >> "$CHECK_LOG"
    echo "Re-check $BINNAME PID: $exists_after_start"

    if [ $exists_after_start -eq 0 ]; then
        ALERT_MESSAGE="AnQiCMS 服务在 $TIMESTAMP 尝试启动后仍然失败！\n"
        ALERT_MESSAGE+="请检查 $BINPATH 目录下的 '$LOG_FILE' 文件获取详细错误信息。\n"
        ALERT_MESSAGE+="服务器信息：$(hostname)\n"
        ALERT_MESSAGE+="启动脚本路径：$0\n\n"
        
        # Add last few lines of the running log to the email for quick diagnosis
        ALERT_MESSAGE+="最新的 $LOG_FILE 日志内容 (最近20行):\n"
        ALERT_MESSAGE+=$(tail -n 20 "$LOG_FILE" 2>&1 || echo "无法读取日志文件或日志文件为空。")
        
        echo -e "$ALERT_MESSAGE" | mail -s "$ALERT_SUBJECT" -r "$SENDER_EMAIL" "$RECIPIENT_EMAIL"
        echo "$TIMESTAMP AnQiCMS 启动失败，已发送告警邮件到 $RECIPIENT_EMAIL。" >> "$CHECK_LOG"
        echo "AnQiCMS 启动失败，已发送告警邮件。"
    else
        echo "$TIMESTAMP $BINNAME 成功启动。" >> "$CHECK_LOG"
        echo "$BINNAME 成功启动。"
    fi
else
    echo "$TIMESTAMP $BINNAME is already running." >> "$CHECK_LOG"
    echo "$BINNAME is already running."
fi

Script modification instructions:

• Email configuration variables: Added at the beginning of the scriptRECIPIENT_EMAIL,SENDER_EMAIL,ALERT_SUBJECTThese variables, which can be modified according to your actual situation to set the recipient's email, sender's email, and email subject. Please make sure to replace[email protected]and[email protected]with the actual email address.
• Log path and timestamp: DefinedLOG_FILEandCHECK_LOGThe variable points to the running log of AnQiCMS and the check log of the script itself, and introducesTIMESTAMPThe variable is used for logging.
• Double status check: After the first judgment that AnQiCMS was not running and after trying to start it, we addedsleep 10(wait for 10 seconds, you can adjust the waiting time according to your actual situation) andexists_after_startPerform a second check on the variable. This second check is the key to determining whether the startup was successful
• Alarm email content: If a second check finds that AnQiCMS is still not running, the script will construct a warning email.The email contains the failure time, suggested log file path, server hostname, and startup script path, which helps to quickly locate the problem.
• Log segment attachment:tail -n 20 "$LOG_FILE"The command will retrieve the last 20 lines of the AnQiCMS runtime log and attach them to the alert email.This is very helpful for initial diagnosis of problems, as the reasons for startup failure are often reflected at the end of the log.
• Send email:echo -e "$ALERT_MESSAGE" | mail -s "$ALERT_SUBJECT" -r "$SENDER_EMAIL" "$RECIPIENT_EMAIL"Command to send email.-sSpecify the subject,-rSpecify the sender.echo -eUsed to parse line breaks in email content.

Configure Cron job

Modifiedstart.shThe script still needs to be executed.cronSchedule the task to run automatically for checking and alerting. Typically, we would configure it to run every minute.

Open the cron configuration file:

crontab -e

Add or modify a line similar to the following in the open editor:

*/1 * * * * /bin/bash /www/wwwroot/anqicms/start.sh >> /www/wwwroot/anqicms/cron.log 2>&1

Make sure the path/www/wwwroot/anqicms/start.shmatches the actual installation path of your AnQiCMS.>> /www/wwwroot/anqicms/cron.log 2>&1isstart.shRedirecting the standard output and error output of the script to a log file is useful for debuggingstart.shIt is very useful for monitoring the running status of the script.

Test the alert system

To verify that the alert system is working properly, you can perform the following test steps:

1. Manually stop the AnQiCMS process: Useps -ef | grep anqicmsFind the PID of the AnQiCMS process and then usekill -9 <PID>the command to manually kill the process.
2. Wait for the Cron task to execute: Wait for one minute, letcronTask triggered after modificationstart.shscript.
3. Check email: Check inRECIPIENT_EMAILthe email address you set, to see if you have received any alert emails.
4. Check the logs: Check$BINPATH/check.logand$BINPATH/running.logFile, confirm whether the script execution records and the AnQiCMS startup log are correctly generated.

If everything is configured correctly, you should receive an alert email, the content of which includes information about the failure of AnQiCMS startup and relevant log fragments. At the same time, due tostart.shThe script will attempt to restart AnQiCMS, after the alarm, the service should try to start again.

Summary

By following these steps, we have established a robust self-monitoring and alerting mechanism for AnQiCMS.This mechanism can automatically detect the service status, attempt to recover automatically when an anomaly is found, and timely notify the operations team if the recovery fails.As website operators, this means we can focus more comfortably on content creation and user experience, while entrusting part of the system's stable responsibility to automated processes.

Frequently Asked Questions (FAQ)

Question: My server does notmailcommand, ormailHow can I set up email alerts since the command cannot send emails normally?

Answer: IfmailThe command is unavailable or the configuration is complex, you can consider using other methods to send email. A common and flexible way is to use a Python script. You canstart.shIn the script, a simple Python script is called to send an email, which can use the built-in PythonsmtplibThe module connects to your company's SMTP server or a public email service (such as Gmail, Outlook) to send emails. Another option is if your server can access the internet, you can use some email service providers' HTTP APIs to send emails, usually throughcurlCommand can be implemented. For example, instart.shAdd apython /path/to/send_mail.py "subject" "body"call, or acurl ...command.

Ask: I am worried that frequent alert emails may cause email bombing, especially when the service is continuously unable to start. How can I optimize the alert frequency?

Answer: This is a very practical question. You can introduce a simple flood control mechanism to optimize the alarm frequency. For example,start.shIn the script, you can check a temporary file (for example,last_alert_time.txt). If the time since the last alert was sent is less than X minutes (or hours), then do not send the alert email this time.Only after exceeding X minutes (or hours), it will send again.This can avoid receiving a large number of duplicate emails during a continuous fault, allowing you to focus more on solving the problem.At the same time, ensure that each alert email includes sufficient information, such as the count of multiple failure attempts or the latest error log.

Ask: What key information should be included in the alert email to help me quickly locate the cause of the AnQiCMS startup failure?

Answer: An effective alert email should contain key information that can help you quickly diagnose the problem. In addition to the timestamp, server hostname, startup script path, suggested log file path, and log snippet mentioned in this article, you can also consider adding the following information:

• AnQiCMS version information: Helps determine if it is a problem with a specific version.
• Server load informationFor exampleuptimeThe output of the command, which can understand the overall system operation.
• Disk space usage:df -houtput, insufficient disk space may also cause startup failure.
• Memory usage:free -houtput, insufficient memory may also affect the startup of Go applications.
• Database connection status: If possible, try to execute a simple database connection test command and include the results in the email, as a failure to connect to the database is a common reason for CMS startup failure.