Syed Jahanzaib Personal Blog to Share Knowledge !

June 13, 2016

KANNEL not responding problem occasionally or after modem reset

Filed under: Linux Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 10:22 AM

kannellogo


Scenario:

We have Ubuntu 12.x installed and serial modem is attached (Teltonika G10). Kannel is installed and serving as SMS gateway.

Problem:

Sometimes modem stops responding and when we restart the modem, kannel starts giving error of Routing Failed. After restart of KANNEL service, the kannel starts responding to sms request and works fine.

Requirement:

This little issue can be really annoyed if you are doing remote administration OR if sms alerts are important for you to keep informed about various aspects of the network. We want some automatic mechanism that can detect this specific error and then act accordingly.

Example …

  • If KANNEL service is down , try to start it
  • If failed to start the service, then print error, and sends email about incident, (one email per incident), and exit script
  • If service starts successfully, then print info, and sends email about successful attempt. (one email per incident), and continue further
  • Test Kannel log last entries, and look for word Routing failed, if found, then restart kannel service and email. (one email per incident).
  • If it does not found any entry in LOG for specific word, then Print OK Result for all.

Solution!

 

Disclaimer:

This may or may not work for you. Because I made it for some very particular situation, and this script helped. So this script checks for two things only. 1st kannel service, 2nd kannel log error for routing failed. I am sure this may or may not help you in same situation because its possible that error can be due to something else. I carefully examined my situation and made script for it. You can get some idea by it and modify the script as required.

Schedule Script to run after every 15 minutes or whatever.

Cron Example

# Run KANNEL MONITOR Script after every 15 minutes
*/15 * * * * /temp/kanneltest.sh

 


the SCript !

mkdir /temp
cd /temp
touch /temp/kanneltest.sh
chmod +x /temp/kanneltest.sh
nano /temp/kanneltest.sh

Copy paste following script. Make sure you read it carefully and modify things accordingly.


#!/bin/bash
# No part of this script is copied from anywhere, you are free to use it, modify or distribute.
# Linux BASH script to check KANNEL service Status and internal component system (example kannel service is ok,but not responding to request giving error in logs
# If KANNEL not responding, then send EMail alert to admin (trigger one time for each status changed)
# Useful for remote admins, who want to be informed when KANNEL have any responding problem.
# Created: 9th-JUNE-2016
# Syed Jahanzaib
# aacable at hotmail dot com / https://aacable.wordpress.com
#set -x
# Colors Config . . . [[ JZ . . . ]]
ESC_SEQ="\x1b["
COL_RESET=$ESC_SEQ"39;49;00m"
COL_RED=$ESC_SEQ"31;01m"
COL_GREEN=$ESC_SEQ"32;01m"
# Service name to monitor ...
KANNEL="kannel"
KANNEL_SERVICE_ERROR="kannel_main_service_down"
KANNEL_BEARERBOX="bearerbox"
# Text that will be checked in kannel logs output file,
# So we will simply use the grep to catch word **Routing failed**. You may use other technique that may work for you. Chill / zaib
TEXT_TO_CATCH="Routing failed"
# Hostname and other Variables
HOSTNAME=`hostname`
COMPANY="zaib (Pvt) Ltd."
FOOTER="Powered By Syed.Jahanzaib"
DATE=`date`
# Temporary file holder for KANNEL status
KANNEL_STATUS_TMP_HOLDER="/tmp/KANNELstatus.txt"
KANNEL_SERVICE_TMP_HOLDER="/tmp/kannelservice.txt"
KANNEL_SERVICE_TMP_HOLDER_ERR="/tmp/kannel_servic_error.txt"
KANNEL_QUERY_RESULT="/tmp/KANNEL_query_result.txt"
KANNEL_LOG_FILE="/var/log/kannel/bearerbox.log"
# Create temp file if not already present, usually for 1st time execution
touch $KANNEL_STATUS_TMP_HOLDER
touch $KANNEL_SERVICE_TMP_HOLDER
touch $KANNEL_SERVICE_TMP_HOLDER_ERR
touch $KANNEL_QUERY_RESULT
# EMAIL RELATED and KANNEL INFO
# for down status, we have to use GMAIL to send email
KANNELURL="127.0.0.1:13013"
KANNELID="kannel"
KANNELPASS="KANNELPASS"
CELL1="03333021909"
# GMAIL Section
GMAILID="YOURGMAILID@gmail.com"
GMAILPASS="YOURGMAILPASS"
ADMINMAIL1="aacableAThotmail.com"

###########################################################################################
# Testing KANNEL Service bearerbox status by its PID
echo -e "$COL_RED
Step 1:$COL_RESET INFO: Testing $KANNEL Service status , testing its PID ..."
PID=`pgrep $KANNEL_BEARERBOX`
if [ -n "$PID" ]; then
echo -e "$KANNEL Service Status = $COL_GREEN OK $COL_RESET with pid $PID"
sed -i "/$KANNEL_SERVICE_ERROR/d" "$KANNEL_SERVICE_TMP_HOLDER"
else
echo -e "$COL_RED$KANNEL Service = NOT RUNNING, trying to restarting it ...$COL_RESET"
service $KANNEL stop > /dev/null 2>&1
sleep 5
killall -9 $KANNEL_BEARERBOX > /dev/null 2>&1
sleep 2
service $KANNEL start > /dev/null 2>&1
sleep 5
# IF KANNEL MAIN SERVICE found DOWN, and email/SMS not already sent, then send it one time to $CELL1/x
PID=`pgrep $KANNEL_BEARERBOX`
if [ -z "$PID" ]; then
KSRVAFTRES="DOWN"
echo -e "Script tried to restart $KANNEL MAIN SERVICE and final status is = $COL_RED $KSRVAFTRES $COL_RESET..."
else
KSRVAFTRES="UP/OK"
echo -e "Script tried to restart $KANNEL MAIN SERVICE and final status is = $COL_GREEN $KSRVAFTRES .$COL_RESET..."
if [ $(grep -c "$KANNEL_SERVICE_ERROR" "$KANNEL_SERVICE_TMP_HOLDER") -eq 0 ]; then
echo -e "$COL_RED Sending SMS/EMAIL for KANNEL MAIN SERVICE...$COL_RESET"
PID=`pgrep $KANNEL_BEARERBOX`
if [ -n "$PID" ]; then
KSRVSTATUS="UP/OK"
echo -e "$KANNEL sevice was not running, After script attempt to restart it, its status is $COL_GREEN $KSRVSTATUS $COL_RESET with PID $PID"
KSRVUPMSG="/tmp/ksrvup.msg"
echo "$KANNEL sevice was not running, After script attempt to restart it, its status is $KSRVSTATUS with PID $PID" > $KSRVUPMSG
/temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL service was not working and finally its $KSRVSTATUS @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$KSRVUPMSG -o message-content-type=text
else
KSRVSTATUS="DOWN"
echo -e "After kannel restart attempt, its status is still $COL_RED $KSRVSTATUS, $COL_RESET... SMS/EMAIL Already sent ..."
#Updating its down log, so script should not repeat sms/email sending
echo "$KANNEL_SERVICE_ERROR" > $KANNEL_SERVICE_TMP_HOLDER
fi
fi
fi
fi

###########################################################################################
# Testing again if above step have started the service or not , if not EXIT the script with the error
# Because if kannel is not running, then checking modem is useless, kannel must be running first in order to proceed further
PID=`pgrep $KANNEL_BEARERBOX`
if [ -n "$PID" ]
then
echo ""
else
echo -e "$COL_REDALERT ALERT: $KANNEL service failed to respond on Service restart request .... please check$COL_RESET
$COL_RED Script cannot continue, exiting ...$COL_RESET"
#exit 0
fi

############## zaib final down
PID=`pgrep $KANNEL_BEARERBOX`
if [ -z "$PID" ]; then
if [ $(grep -c "$KANNEL_SERVICE_ERROR" "$KANNEL_SERVICE_TMP_HOLDER_ERR") -eq 0 ]; then
KSRVSTATUS="DOWN"
KSRVDOWNMSG="/tmp/ksrvdown.msg"
echo "After kannel restart attempt by the script, its status is still $KSRVSTATUS. You need to check it manualy. Now only Human can see what is going on ... "
echo "After kannel restart attempt by the script, its status is still $KSRVSTATUS. You need to check it manualy. Now only Human can see what is going on ... " > $KSRVDOWNMSG
/temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL service failed to restart. Status = $KSRVSTATUS @ $DATE / Check it manualy!" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$KSRVDOWNMSG -o message-content-type=text
#Updating its down log, so script should not repeat sms/email sending
echo "$KANNEL_SERVICE_ERROR" > $KANNEL_SERVICE_TMP_HOLDER_ERR
exit 0
fi
else
sed -i "/$KANNEL_SERVICE_ERROR/d" "$KANNEL_SERVICE_TMP_HOLDER_ERR"
fi

###########################################################################################
PID=`pgrep $KANNEL_BEARERBOX`
if [ -n "$PID" ]
then
echo ""
else
echo -e "$COL_REDALERT ALERT: $KANNEL service failed to respond on Service restart request .... please check$COL_RESET
$COL_RED Script cannot continue, exiting ...$COL_RESET"
exit 0
fi

###########################################################################################
# Run the script to test kannel internal processing ...
echo -e "$COL_RED
Step 2:$COL_RESET
Testing Kannel internal Service Status in LOGS for internal responding @DATE ..."
for KANNEL in $KANNEL
do
# Match $TEXT_TO_CATCH in the log file grep query result
STATUS=`tail -n 10 $KANNEL_LOG_FILE | grep "$TEXT_TO_CATCH"`
if [ -n "$STATUS" ]; then
# Print Result for information purposes
echo "ALERT: text ** $TEXT_TO_CATCH ** found in $KANNEL_QUERY_RESULT"
echo -e "$COL_RED
$KANNEL service is UP but internal system is not responding to SMS ... Trying to restarting $KANNEL service ...$COL_RESET"
# IF KANNEL found DOWN, and email/SMS not already sent, then send it one time to $CELL1/x
if [ $(grep -c "$KANNEL" "$KANNEL_STATUS_TMP_HOLDER") -eq 0 ]; then
# Restart the KANNEL service
service kannel stop > /dev/null 2>&1
sleep 5
killall -9 bearerbox > /dev/null 2>&1
sleep 2
service kannel start > /dev/null 2>&1
sleep 5
echo -e "$COL_RED
$KANNEL service is UP but internal system is not responding $DATE .. SENDING DOWN SMS/EMAIL for current incident / for one time only ...$COL_RESET"
# Update KANNEL down status in a file so that sms/email should not be sent again and again
echo "$KANNEL" > $KANNEL_STATUS_TMP_HOLDER
# Sending DOWN SMS via KANNEL
echo -e "$COL_REDSending SMS , if KANNEL SERVICE is down or internal system not responding, SMS will NOT be sent ...$COL_RESET
"
# Update down message to be sent via sms/email
echo "$MSG_DOWN" > $MSGDOWNHOLDER
# sen email using sendemail tool
cat $MSGDOWNHOLDER | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@-
######## EMAIL SECTION ##############
# Make sure you install sendEMAIL tool and test it properly before using email section.
#SEND EMAIL Alert As well using sendEMAIL tool using GMAIL ADDRESS.
# If you want to send email , use below ...
echo -e "$COL_REDSending EMAIL ALERT to $ADMINMAIL1 ...$COL_RESET
"
/temp/sendEmail-v1.56/sendEmail -u "ALERT: $KANNEL not responding @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$MSGDOWNHOLDER -o message-content-type=text
fi

###########################################################################################
# Else if $TEXT_TO_CATCH does not found in $KANNEL_QUERY_RESULT, then consider KANNEL is UP
else
echo -e "$COL_GREEN
INFO:$COL_RESET Word ** $TEXT_TO_CATCH ** was NOT found in $KANNEL_LOG_FILE
$COL_RED
FINAL RESULT:$COL_RESET
$KANNEL service = $COL_GREEN OK $COL_RESET
$KANNEL iternal system = $COL_GREEN OK $COL_RESET / All seems to be responding OK!
"
# If KANNEL found UP and last status was DOWN, then send UP msg one time
if [ $(grep -c "$KANNEL" "$KANNEL_STATUS_TMP_HOLDER") -eq 1 ]; then
echo "$KANNEL internal system is now responding at $(date)... SENDING UP SMS for current incident one time only ...
"
echo "$MSG_UP" > $MSGUPHOLDER
# Sending UP SMS via KANNEL
echo -e "Sending $COL_GREENUP$COL_RESET SMS , if KANNEL process is down, sms will not be sent ...
"
cat $MSGUPHOLDER | curl "http://$KANNELURL/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$CELL1" -G --data-urlencode text@-
############## EMAIL SECTION ##############
# Make sure you install sendEMAIL tool and test it properly before using email section.
#SEND EMAIL Alert As well using sendEMAIL tool using GMAIL ADDRESS.
# If you want to send email , use below ...
echo "Sending SEMAIL UP/OK to $ADMINMAIL1 & $ADMINMAIL2 ...
"
/temp/sendEmail-v1.56/sendEmail -u "INFO: $KANNEL internal system is now responding OK @ $DATE" -o tls=yes -s smtp.gmail.com:587 -t $ADMINMAIL1 -xu $GMAILID -xp $GMAILPASS -f $GMAILID -o message-file=$MSGDOWNHOLDER -o message-content-type=text
# Update KANNEL UP status in a file so that sms/email should not be sent again and again
sed -i "/$KANNEL/d" "$KANNEL_STATUS_TMP_HOLDER"
fi
fi
done
# Script Ends Here ...
# z@iB


Script Results:

1- script execute result

 

1- srv down and restarted ok up result

 

1- srv down failed to restart

1-upndown

3 -intok


Regard’s
Syed Jahanzaib

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: