Following is linux base bash script which can be scheduled to run every X minutes and it can do following …
Personnel Note:
This script was made for an operator who was facing strange problem that in random timings , the mysql or some times radiusd service stopped automatically, unfortunately the error was not sorted out as it occurs in awkward timings when no one is at NOC, so I made an workaround in the form of this script so that we can diagnose the issue later in details and in the meanwhile manually startup of services should not be required. The script was scheduled to run after every 10 minutes.
Note: This is not a solution, Just a workaround for small period of time.
This script will check one or more services status, if found stopped, it will try to re-start them, and then check there status again and output there status in email.
In this example I took MYSQL and RADIUS services.
Its much easier to use some centralized network monitoring system like Mikrotik DUDE or NAGIOS or likewise , but as we know that every system have its known limitations, and sometime it is un necessary to setup a network monitoring system just to monitor single PC or service.
In such situation where resources are limited OR you want some thing very customized solution of your own choices with your required bells and whistles , its recommended to do it with simple bash without needing any 3rd party tool.
PSEUDO CODE:
- Check for Service status example mysqld
- If it found it running, then do nothing. Exit
- If it found it STOPPED, it will send you SMS alert for down status (one time only until next status change) via KANNEL sms gateway. It will also gonna try to start the service one time.
- If it found the service running on next run, it will send service UP info via sms. (one time only until next status change)
- Email Alerts for UP n DOWN added on 4th Dec, 2015
SCRIPT:
mkdir /temp touch /temp/checksrv.sh chmod +x /temp/checksrv.sh nano /temp/checksrv.sh
Now paste the following
#!/bin/bash # Scheduled Script to check linux service status after every 5 minutes. # If found stopped, send sms or email Alerts, but donot repeat it untill next status change. # Script Designed by Syed Jahanzaib # aacable at hotmail dot com # https://aacable.wordpress.com # Created : 25-NOV-2015 # Last Modified = 4th-NOV-2015 1600 hours # Pakistan !!! #set -x #SRV="$1" SRV1="$1" DATE=`date` STATUS_HOLDER="/tmp/$SRV1.txt" touch $STATUS_HOLDER # COMPANY NAME COMPANY="ZAIB" # Hostname HOSTNAME=`hostname` # KANNEL SMS Gateway Info KANNELURL="KANNELIP:13013" KANNELID="kannel" KANNELPASS="kannelpass" CELL1="03333021909" # GMAIL ACCOUNT INFORMATION AND RELATED .... MAILER="/temp/sendEmail-v1.56/sendEmail" TO1="aacable at hotmail dot com" GMAILID="YOURGMAILID@gmail.com" GMAILPASS="YOURGMAILPASSWORD" MAILBODYDOWN="$SRV1 DOWN" MAILBODYUP="$SRV1 UP" MAILSUBJECTDOWN="$COMPANY ALERT: $HOSTNAME - $SRV1 is now DOWN @ $DATE" MAILSUBJECTUP="$COMPANY INFO: $HOSTNAME - $SRV1 is now UP @ $DATE" # Colors Config . . . [[ JZ . . . ]] ESC_SEQ="\x1b[" COL_RESET=$ESC_SEQ"39;49;00m" COL_RED=$ESC_SEQ"31;01m" COL_GREEN=$ESC_SEQ"32;01m" OS="1" osver=`cat /etc/issue |awk '{print $1}'` # OS checkup for UBUNTU if [[ $osver == Ubuntu ]]; then echo -e "$COL_GREEN OS = Ubuntu $COL_RESET" set OS="Ubuntu" OSPKG="apt-get install -y chkconfig" else echo -e "$COL_GREEN OS = CENTOS $COL_RESET" set OS="CENTOS" OSPKG="yum -y install chkconfig" fi ######################################## # Check if no service name is given if [ "$SRV1" == "" ]; then echo -e "$COL_RED No service name have been provided. $COL_RESET" echo "Usage exmaple:" echo -e "/temp/checksrv.sh mysqld" echo "OR" echo -e "/temp/checksrv.sh (Depend on your OS deployment)" echo exit 0 else # Check if CHKCONFIG command is installed or not. CHK=`which chkconfig` if [ "$CHK" == "" ]; then echo -e "$COL_RED CHKCONFIG command is not installed. Please install it with following command $OSPKG $COL_RESET" exit 0 else # Check if service is installed / valid or not CHKSRV=`chkconfig | grep -w $SRV1` if [ "$CHKSRV" == "" ]; then echo -e "$COL_RED NO SERVICE is INSTALLED WITH $SRV1 NAME. Exiting ...$COL_RESET" exit 0 else ######################################### ############## IF ALL OK, PROCEED FURTHER ######################################### # Query Service UPTIME QUERYSRVUPTIME=`ps -eo "%U %c %t" |grep "$SRV1" |grep -v grep| grep root` uptime=`echo $QUERYSRVUPTIME |awk '{ print $NF }'` SERVICE1="$SRV1" SUBJECT="ALERT: $SRV1 is Down..." # SMS Msgs test for up n down MSG_UP="$COMPANY Info: $HOSTNAME - $SERVICE1 is now UP @ $DATE. $SRV1 UPTIME is $uptime. Powered by Syed.Jahanzaib" fi MSG_DOWN="$COMPANY Alert: $HOSTNAME - $SERVICE1 is now DOWN @ $DATE. Trying to restarting it. wait 1 mnt for next result. Powered by Jz" ##### CHECK FOR SERVICE STATUS for SRVCHK in $SERVICE1 do PID=$(pgrep $SERVICE1) if [ "$PID" == "" ]; then echo -e "$COL_RED $SRVCHK is down $COL_RESET " if [ $(grep -c "$SRVCHK" "$STATUS_HOLDER") -eq 0 ]; then echo -e "$COL_RED ALERT: $SERVICE1 is down at $(date) / trying to restart and SENDING SMS and EMAIL....$COL_RESET" echo "$MSG_DOWN" > /tmp/$SERVICE1_down.sms # Sending DOWN SMS via KANNEL cat /tmp/$SERVICE1_down.sms | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@- # Sending DOWN alert EMAIL via GMAIL $MAILER -t $TO1 -u "$MAILSUBJECTDOWN" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -o message-file=/tmp/$SERVICE1_up.sms # Start $SERVICE1 service if found down service $SERVICE1 start echo "$SRVCHK" >> $STATUS_HOLDER fi else echo -e "$COL_GREEN $SRVCHK is alive with Uptime of $uptime and its PID are as follows... $COL_RESET \n$PID" # IF Service found UP send SMS ONE TIME if [ $(grep -c "$SRVCHK" "$STATUS_HOLDER") -eq 1 ]; then echo -e "$COL_GREEN INFO ALERT : $SERVICE1 is UP at $(date) $SRV1 Uptime is $uptime SENDING SMS and EMAIL.... $COL_RESET" echo "$MSG_UP" > /tmp/$SERVICE1_up.sms # Sending UP SMS via KANNEL cat /tmp/$SERVICE1_up.sms | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@- sed -i "/$SRVCHK/d" "$STATUS_HOLDER" # Sending UP alert EMAIL via GMAIL $MAILER -t $TO1 -u "$MAILSUBJECTUP" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -o message-file=/tmp/$SERVICE1_up.sms fi fi done fi fi
USAGE:
/temp/checksrv.sh mysql #OR /temp/checksrv.sh squid
As showed in the image below …
SMS RESULT
EMAIL RESULTS:
FOR EMAIL ALERTS,
2- Download sendEmail tool to send email alerts
Install sendEmail Tool
tar zxvf sendEmail-v1.56.tar.gz cd sendEmail-v1.56/ |
ADD SUPPORTING LIBRARY
for ubuntu
apt-get -y install libio-socket-ssl-perl libnet-ssleay-perl perl |
for centos
yum -y install perl perl-Crypt-SSLeay perl-IO-Socket-SSL |
EMAIL CONFIGURATION AND SAMPLE RESULTS OF RECEIVING
Try to send email using command line , make sure you fill the variables with your original values like mail address password etc: Example
/temp/sendEmail-v1.56/sendEmail -t $TO1 -u "Test Email" -s smtp.gmail.com:587 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o tls=yes -m "hi"
SCHEDULE the SCRIPT in CRON:
Open crontab in editor
crontab -e
Now add following
# Run CHECK SERVICE script after very 5 minutes */5 * * * * /temp/checksrv.sh mysqld OR */5 * * * * /temp/checksrv.sh mysql
SAVE and Exit!
Regard’s
Syed Jahanzaib
hi syed for ubuntu not working?
LikeLike
Comment by antonio — November 26, 2015 @ 1:19 PM
Ok i have modified it to check OS level as well. and use chkconfig utility which can work for both ubuntu and centos as well. I have tested it now with Centos 6.x and Ubuntu 12.x , its working fine. alerts working good. added colors coding as well to make it a bit fancy, as we all love colors
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — November 26, 2015 @ 3:29 PM
[…] for sendEmail Utility https://aacable.wordpress.com/2015/11/26/bash-scheduled-script-to-check-linux-service-status-and-sms… […]
LikeLike
Pingback by Sending SMS/Email Alert For Reseller Account Renewal/Deposit | Syed Jahanzaib Personal Blog to Share Knowledge ! — February 9, 2016 @ 1:04 PM
Dear Jhanzaib,
I like this script and i want to apply it in my RHEL 6.5 for some different service , is this working ? also on this script i am unable to find mysqld service bcoz i want to change as per my requirement. pls help
LikeLike
Comment by osamamansoor — February 10, 2016 @ 12:05 PM
Dear Jhanzaib ,
ok now i understand my this point
<<>
just need to know it is also works on RHEL 6.5
LikeLike
Comment by osamamansoor — February 10, 2016 @ 1:33 PM
it can work on about any flavor of linux, just few syntax can be different but the logic will remain same.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2016 @ 1:51 PM
I have problem
[root@server temp]# ./checksrv.sh mysqld
OS = CENTOS
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
If you want to list systemd services use ‘systemctl list-unit-files’.
To see services enabled on particular target use
‘systemctl list-dependencies [target]’.
[root@vmi140561 temp]# ./checksrv.sh mysqld.service
OS = CENTOS
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
If you want to list systemd services use ‘systemctl list-unit-files’.
To see services enabled on particular target use
‘systemctl list-dependencies [target]’.
NO SERVICE is INSTALLED WITH mysqld.service NAME. Exiting …
LikeLike
Comment by emir — December 28, 2017 @ 11:13 PM