BASH: Scheduled script to check linux service status and sms/email while preventing repeated alerts

Following is linux base bash script which can be scheduled to run every X minutes and it can do following …

Personnel Note:

This script was made for an operator who was facing strange problem that in random timings , the mysql or some times radiusd service stopped automatically, unfortunately the error was not sorted out as it occurs in awkward timings when no one is at NOC, so I made an workaround in the form of this script so that we can diagnose the issue later in details and in the meanwhile manually startup of services should not be required. The script was scheduled to run after every 10 minutes.

Note: This is not a solution, Just a workaround for small period of time.

This script will check one or more services status, if found stopped, it will try to re-start them, and then check there status again and output there status in email.

In this example I took MYSQL and RADIUS services.

Its much easier to use some centralized network monitoring system like Mikrotik DUDE or NAGIOS  or likewise , but as we know that every system have its known limitations, and sometime it is un necessary to setup a network monitoring system just to monitor single PC or service.
In such situation where resources are limited OR you want some thing very customized solution of your own choices with your required bells and whistles ,  its recommended to do it with simple bash without needing any 3rd party tool.

 

PSEUDO CODE:

• Check for Service status example mysqld
• If it found it running, then do nothing. Exit
• If it found it STOPPED, it will send you SMS alert for down status (one time only until next status change) via KANNEL sms gateway. It will also gonna try to start the service one time.
• If it found the service running on next run, it will send service UP info via sms. (one time only until next status change)
• Email Alerts for UP n DOWN added on 4th Dec, 2015

---

SCRIPT:

mkdir /temp
touch /temp/checksrv.sh
chmod +x /temp/checksrv.sh
nano /temp/checksrv.sh

Now paste the following

---

#!/bin/bash
# Scheduled Script to check linux service status after every 5 minutes.
# If found stopped, send sms or email Alerts, but donot repeat it untill next status change.
# Script Designed by Syed Jahanzaib
# aacable at hotmail dot com
# https://aacable.wordpress.com
# Created : 25-NOV-2015
# Last Modified = 4th-NOV-2015 1600 hours
# Pakistan !!!
#set -x

#SRV="$1"
SRV1="$1"
DATE=`date`

STATUS_HOLDER="/tmp/$SRV1.txt"
touch $STATUS_HOLDER

# COMPANY NAME
COMPANY="ZAIB"

# Hostname
HOSTNAME=`hostname`

# KANNEL SMS Gateway Info
KANNELURL="KANNELIP:13013"
KANNELID="kannel"
KANNELPASS="kannelpass"
CELL1="03333021909"

# GMAIL ACCOUNT INFORMATION AND RELATED ....
MAILER="/temp/sendEmail-v1.56/sendEmail"
TO1="aacable at hotmail dot com"
GMAILID="YOURGMAILID@gmail.com"
GMAILPASS="YOURGMAILPASSWORD"
MAILBODYDOWN="$SRV1 DOWN"
MAILBODYUP="$SRV1 UP"
MAILSUBJECTDOWN="$COMPANY ALERT: $HOSTNAME - $SRV1 is now DOWN @ $DATE"
MAILSUBJECTUP="$COMPANY INFO: $HOSTNAME - $SRV1 is now UP @ $DATE"


# Colors Config  . . . [[ JZ . . . ]]
ESC_SEQ="\x1b["
COL_RESET=$ESC_SEQ"39;49;00m"
COL_RED=$ESC_SEQ"31;01m"
COL_GREEN=$ESC_SEQ"32;01m"
OS="1"
osver=`cat /etc/issue |awk '{print $1}'`

# OS checkup for UBUNTU
if [[ $osver == Ubuntu ]]; then
echo -e "$COL_GREEN OS = Ubuntu $COL_RESET"
set OS="Ubuntu"
OSPKG="apt-get install -y chkconfig"

else

echo -e "$COL_GREEN OS = CENTOS $COL_RESET"
set OS="CENTOS"
OSPKG="yum -y install chkconfig"
fi

########################################

# Check if no service name is given
if [ "$SRV1" == "" ]; then
echo -e "$COL_RED No service name have been provided. $COL_RESET"
echo "Usage exmaple:"
echo -e "/temp/checksrv.sh mysqld"
echo "OR"
echo -e "/temp/checksrv.sh (Depend on your OS deployment)"
echo
exit 0
else

# Check if CHKCONFIG command is installed or not.
CHK=`which chkconfig`
if [ "$CHK" == "" ]; then
echo -e "$COL_RED CHKCONFIG command is not installed.
Please install it with following command

$OSPKG $COL_RESET"
exit 0
else

# Check if service is installed / valid or not
CHKSRV=`chkconfig | grep -w $SRV1`
if [ "$CHKSRV" == "" ]; then
echo -e "$COL_RED NO SERVICE is INSTALLED WITH $SRV1 NAME. Exiting ...$COL_RESET"
exit 0
else

#########################################
############## IF ALL OK, PROCEED FURTHER
#########################################

# Query Service UPTIME
QUERYSRVUPTIME=`ps -eo "%U %c %t" |grep "$SRV1" |grep -v grep| grep root`
uptime=`echo $QUERYSRVUPTIME |awk '{ print $NF }'`

SERVICE1="$SRV1"
SUBJECT="ALERT: $SRV1 is Down..."

# SMS Msgs test for up n down
MSG_UP="$COMPANY Info:

$HOSTNAME - $SERVICE1 is now UP @ $DATE.
$SRV1 UPTIME is $uptime.
Powered by Syed.Jahanzaib"
fi

MSG_DOWN="$COMPANY Alert:

$HOSTNAME - $SERVICE1 is now DOWN @ $DATE. Trying to restarting it. wait 1 mnt for next result.

Powered by Jz"

##### CHECK FOR SERVICE STATUS

for SRVCHK in $SERVICE1
do
PID=$(pgrep $SERVICE1)
if [ "$PID" == "" ]; then
echo -e "$COL_RED $SRVCHK is down $COL_RESET "
if  [ $(grep -c "$SRVCHK" "$STATUS_HOLDER") -eq 0 ]; then
echo -e "$COL_RED ALERT: $SERVICE1 is down at $(date) / trying to restart and SENDING SMS and EMAIL....$COL_RESET"
echo "$MSG_DOWN" > /tmp/$SERVICE1_down.sms

# Sending DOWN SMS via KANNEL
cat /tmp/$SERVICE1_down.sms | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@-

# Sending DOWN alert EMAIL via GMAIL
$MAILER -t $TO1 -u "$MAILSUBJECTDOWN" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -o message-file=/tmp/$SERVICE1_up.sms

# Start $SERVICE1 service if found down
service $SERVICE1 start
echo "$SRVCHK" >> $STATUS_HOLDER
fi

else
echo -e "$COL_GREEN $SRVCHK is alive with Uptime of $uptime
and its PID are as follows... $COL_RESET  \n$PID"

# IF Service found UP send SMS ONE TIME
if  [ $(grep -c "$SRVCHK" "$STATUS_HOLDER") -eq 1 ]; then
echo -e "$COL_GREEN INFO ALERT : $SERVICE1 is UP at $(date)
$SRV1 Uptime is $uptime
SENDING SMS and EMAIL.... $COL_RESET"
echo "$MSG_UP" > /tmp/$SERVICE1_up.sms

# Sending UP SMS via KANNEL
cat /tmp/$SERVICE1_up.sms | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@-
sed -i "/$SRVCHK/d" "$STATUS_HOLDER"

# Sending UP alert EMAIL via GMAIL
$MAILER -t $TO1 -u "$MAILSUBJECTUP" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -o message-file=/tmp/$SERVICE1_up.sms


fi
fi
done
fi
fi

---

USAGE:

/temp/checksrv.sh mysql
#OR
/temp/checksrv.sh squid

As showed in the image below …

SMS RESULT

---

EMAIL RESULTS:

FOR EMAIL ALERTS,

2- Download sendEmail tool to send email alerts

Install sendEmail Tool

wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz tar zxvf sendEmail-v1.56.tar.gz cd sendEmail-v1.56/

ADD SUPPORTING LIBRARY

for ubuntu

apt-get -y install libio-socket-ssl-perl libnet-ssleay-perl perl

for centos

yum -y install perl perl-Crypt-SSLeay perl-IO-Socket-SSL

 

EMAIL CONFIGURATION AND SAMPLE RESULTS OF RECEIVING

Try to send email using command line , make sure you fill the variables with your original values like mail address password etc: Example

/temp/sendEmail-v1.56/sendEmail -t $TO1 -u "Test Email" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -m "hi"

 

SCHEDULE the SCRIPT in CRON:

Open crontab in editor

crontab -e

Now add following

# Run CHECK SERVICE script after very 5 minutes
*/5 * * * * /temp/checksrv.sh mysqld
OR
*/5 * * * * /temp/checksrv.sh mysql

SAVE and Exit!

---

 

Regard’s

Syed Jahanzaib

Advertisements