Syed Jahanzaib Personal Blog to Share Knowledge !

September 17, 2015

Monitoring multiple WAN links in PCC using BLACK-HOLE route approach !

Filed under: Mikrotik Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 12:18 PM

bh

Scenario:

Dual pppoe-client wan links are configured in mikrotik with PCC load balancing.

Task:

To monitor both (ow more) wan links via some fixed routes and email in case any goes down or take other action as required.

Solution:

You must be aware that to achieve any task, there are multiple ways to do so, Select whatever is best and whatever works for you (offcourse without affecting any other functionality). I tried various solutions to monitor pppoe-wan clients, but most of them didn’t worked as I wanted. So I used blackhole route approach and it worked 100%.

Example:

[This example is just for demonstration purpose only. In the real production environment you MUST use multiple host monitoring , because it is very possible that if you monitor single host, and for some reason ISP blocks it, or the owner of the host close it for maintenance then what happens? YES you will get false alarm even if the internet is working fine. To avoid such false alarms ,You must use multiple host to monitor each wan link I wrote multiple wan monitor script in some previous post, search it.)

For WAN-1 link we will monitor 4.2.2.1  [DNS Server]
For WAN-2 link we will monitor 208.67.222.123 [Open DNS server IP]

 

/ip route
add comment="WAN-1  /  FORCED ROUTE FOR LINK MONITORING / ZAIB" disabled=no distance=1 dst-address=4.2.2.1/32 gateway=pppoe-out1 scope=30 target-scope=10

add comment="WAN-2  /  FORCED ROUTE FOR LINK MONITORING / ZAIB" disabled=no distance=1 dst-address=208.67.222.123/32 gateway=pppoe-out2 scope=30 target-scope=10

PROBLEM:

The problem is that as soon as one WAN (pppoe-out1 disconnects for any reason like line dead etc, the PING will then look in main table and whatever route it found (example pppoe-out2) it will pass the traffic from that available wan link, and this is BAD for our monitoring script because we wanted to FORCE specific route to always pass via specific link only.

For this reason we will duplicate above routes, BUT this time we will use type ‘blackhole‘ and create higher distance value so that when default route FOR SPECIFIC MONITORED HOST goes down, then next route with higher distance value will be enabled automatically and will send packets to black-hole resulting in timeout which we will be using in net watch monitoring scripts.

 

add comment="WAN-1  blackhole /  FORCED ROUTE FOR LINK MONITORING / ZAIB" disabled=no distance=2 dst-address=4.2.2.1/32 type=blackhole

add comment="WAN-2  blackhole /  FORCED ROUTE FOR LINK MONITORING / ZAIB" disabled=no distance=2 dst-address=208.67.222.123/32 type=blackhole

So as soon as WAN1 goes down, the ping to 4.2.2.1 will go to BLACKHOLE / timeout dueto above rules. same for wan2.

 


Example script to monitor wan link and email or take other action

Complete script to monitor wan1 is as follows. Just for reference purposes.


# Syed Jahanzaib / aacable @ hotmail.com
# https://aacable.wordpress.com
# WAN Monitor script (for single host) and email if down/up
# Kindly configure tools/Email first to send email alert
# Modified on : 26-NOV-2015 added KANNEL sms support, on request of mr anotnino ITALY

:local i 0;
:local F 0;
:local date;
:local time;
:set date [/system clock get date];
:set time [/system clock get time];

:global DSL1netstatus;
:global DSL1netLastChange;

:local adminmail1 "YOUR MAIL ADDRESS"
:local gmailid "YOURGMAILID@gmail.com"
:local gmailpass "YOURGMAILPASS"

# If you have KANNEL , you can modify below

:local KANNEHOST "192.168.100.1"
:local KANNELID "kannel"
:local KANNELPASS "kannel"
:local cell1 "03333021909"

# Donot use spaces in following, instead use plus sign to accomodate space
:local COMPANY "MY_ISP_ZAIB"
:local UPSMS "$COMPANY+INFO:+WAN-1+is+UP+now+at+$date+$time"
:local DOWNSMS "$COMPANY+ALERT:+WAN-1+is+DOWN+now+at+$date+$time"


# Check WAN1 GATEWAY to be monitored (Currently we are monitoring internet host)
:global wan1host1 4.2.2.1

# Gmail SMTP Address
:global gmailsmtp
:set gmailsmtp [:resolve "smtp.gmail.com"];

# Ping Internet Host 10 times,
:for i from=1 to=10 do={
if ([/ping $wan1host1 count=1]=0) do={:set F ($F + 1)}
:delay 1;
};

# If no response (10 out of 10 Ping fails for each item, Times out, then LOG down status and take action
:if (($F=10)) do={
:if (($DSL1netstatus="UP")) do={
:set DSL1netstatus "DOWN";

# Also add status in global variables to be used as tracking
:set date [/system clock get date];
:set time [/system clock get time];
:set DSL1netLastChange ($time . " " . $date);


##################################################
####### FOR DOWN STATUS, CHANGE THE RULES ########
##################################################
# If the link is down, then LOG info and warning in Mikrotik LOG window [Zaib]
:log error "WAN1 Gateway Not Responding. Please Check DSL1 Connectivity..."


# "Emailing the DOWN status. . . "
/tool e-mail send to="$adminmail1" password=$gmailpass subject="$[/system clock get date] $[/system clock get time] -- ALERT: PTCL DSL-1 is DOWN" from=$gmailid server=$gmailsmtp tls=yes body="$[/system clock get date] $[/system clock get time] : ALERT: PTCL DSL-1 is DOWN"

# IF USB MODEM IS CONNECTED WITH LOCAL RB THEN USE FOLLOWING
#/tool sms send port=usb3 phone-number=$cell4  message="INFO: xxxxx Network DSL-1 is DOWN ... / by Jz."  channel=0

# IF YOU HAVE SMS GATEWAY LIKE KANNEL USE FOLLOWING
# Send DOWN SMS
#/tool fetch url="http://$KANNELHOST:13013/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$cell1&text=$DOWNSMS"


##################################################
####### FOR UP STATUS, CHANGE THE RULES ########
##################################################
# If ping is ok 5/5 reply received, then LOG UP and take action as required

} else={:set DSL1netstatus "DOWN";}
} else={
:if (($DSL1netstatus="DOWN")) do={
:set DSL1netstatus "UP";
# If link is UP, then LOG info and warning in Mikrotik LOG window [Zaib]
log warning "WAN1 Gateway RESTORED ..."

# "Emailing the UP  status. . . "
/tool e-mail send to="$adminmail1" password=$gmailpass subject="$[/system clock get date] $[/system clock get time] -- INFO: PTCL DSL-1 is UP Now." from=$gmailid server=$gmailsmtp tls=yes body="$[/system clock get date] $[/system clock get time] : ALERT: PTCL DSL-1 is UP Now."

# IF USB MODEM IS CONNECTED WITH LOCAL RB THEN USE FOLLOWING
#/tool sms send port=usb3 phone-number=$cell4  message="INFO: xxxxx Network DSL-1 is UP ... / by Jz."  channel=0

# IF YOU HAVE SMS GAETWAY LIKE KANNEL USE FOLLOWING
# Send UP SMS
#/tool fetch url="http://$KANNELHOST:13013/cgi-bin/sendsms?username=$KANNELID&password=$KANNELPASS&to=$cell1&text=$UPSMS"


:set date [/system clock get date];
:set time [/system clock get time];
:set DSL1netLastChange ($time . " " . $date);

} else={:set DSL1netstatus "UP";}
}

 

EMAIL ALERT

 

alerts

SMS ALERT

sms_alert

LOG

WAN_DOWN_ALERTS_LOGS

 

You can perform other customized actions on DOWN or UP  too 🙂

 

Regard’s
SYED JAHANZAIB

June 12, 2014

Mikrotik WAN monitoring script with multiple host check

Filed under: Mikrotik Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 2:31 PM

eagle_map


Following is an Mikrotik RouterOS script designed to test internet connectivity by checking connectivity (by Ping method) for two different internet hosts, (like google dns and root dns server in this example). Main benefits of using script is that you can perform customized actions, like sending SMS or Email Alerts / acquire almost accurate results, avoiding false alarms etc etc.

In this script we are doing ping check on two internet hosts if it fails to receive single reply from both hosts, then it will consider the link DOWN. If one host is working and second is down, it will consider the link UP. It is done for cross verification. All sections in the script is customizable to meet your requirements , so make yourself home  😉

Multiple HOST check is recommended, Because if you are using single host check script or net-watch,then it can sometimes give you false alarm for any reason like maintenance or firewall blockage .


ROS SCRIPT CODE: (Script name= wan1mon )

 

# Mikrotik WAN/Intenret Link monitoring SCRIPT with optional Email and SMS Alert,
# We are using local KANNEL as SMS gateway and GMAIL as email sender
# Make sure to change email settings, because without backup link, email will not be send using gmail
# by Syed Jahanzaib
# https://aacable.wordpress.com
# Email : aacable at hotmail dot com
# Script Last Modified : 16th-AUG-2017

# Setting Variables
:local i 0;
:local F 0;
:local date;
:local time;
:local sub1 ([/system identity get name])
:local sub2 ([/system clock get time])
:local sub3 ([/system clock get date])
:local company "GALAXY Pvt Ltd"
:local ISP1 "ZAIB Primary Link"
:local ISP1NICK "JZ"

# Number of Ping Count, how many times mikrotik should ping the target device
:local PINGCOUNT "5"
# Ping threshold
:local PINGTS "10"

# GMAIL SMTP DYNAMIC Config Section, Make sure to change these values to match your's / Jz
# Gmail SMTP Address / you can use resolve command too, but i prefere to use manual smtp for some reasons

# Additional Check for GMAIL SMTP Resolving.
# If gmail SMTP failed to resolve, then it should set manual IP for gmail smtp
# So that email should go even if there is a dns failure.
# By ZAIB - 18-Aug-2017

:local GMAILSMTPDNSNAME "smtp.gmail.com"
:global GMAILSMTP ""
:local RESOLVELIST {"$GMAILSMTPDNSNAME"}
:foreach addr in $RESOLVELIST do={
:do {:resolve server=8.8.8.8 $addr} on-error={:set GMAILSMTP "0";}}
:if ($GMAILSMTP = 0) do={
:set GMAILSMTP "64.233.167.108";
} else={
:set GMAILSMTP [:resolve "$GMAILSMTPDNSNAME"];
}
:local adminmail1 ADMIN_MAIL_1@hotmail.com
#below Gmail ID will be used to send alerts / email
:local gmailid "YOUR_GMAIL_ID@gmail.com"
:local gmailuser "YOUR_GMAIL_ID"
:local gmailpass "YOUR_GMAIL_PASS"
:local gmailport 587

:local SUBJECTDOWN "$[/system clock get date] $[/system clock get time] -- GALAXY_ALERT: $ISP1NICK is DOWN."
:local BODYDOWN "$ISP1 Internet Link at GALAXY DATA CENTER not responding, Please check Linkconectivity. \n\n\n\n Auto Failover Script for Mikrotik Designed by \n *** Syed_Jahanzaib ****"

:local SUBJECTUP "$[/system clock get date] $[/system clock get time] -- GALAXY_INFO: $ISP1NICK have restored."
:local BODYUP "$ISP1 Internet Link have been Restored ... \n\n\n\n Auto Failover Script for Mikrotik Designed by \n *** Syed_Jahanzaib ****"

:global ISP1STATUS;
:global ISP1LASTCHANGESTATUS;

# Setting KANNEL INFO and SMS Number for admin
:local KID "KANNEL_ID"
:local KPASS "KANNEL_PASSWORD"
:local KHOST "192.168.0.1:13013"
:local CELL1 "03002767701"

# Setting UP SMS message
:local MSGDOWN "ISP+ALERT:+$ISP1NICK+Fiber+is+now+DOWN"
:local MSGUP "ISP+INFO:+$ISP1NICK+Fiber+is+now+UP"

# Check 2 hosts via ISP1 GATEWAY for monitoring
# Just make sure you ahve static routes for these routes and additionaly blackhole route as well
:global ISP1HOST1 "193.183.98.154"
:global ISP1HOST2 "84.200.70.40"

# FAKE HOST to test script
#:global ISP1HOST1 1.2.3.4
#:global ISP1HOST2 1.2.3.4

:global WAN1STATUS;
:if ([:len $WAN1STATUS] = 0) do={
:set WAN1STATUS UP;
}

# PING each host $PINGCOUNT times
:for i from=1 to=$PINGCOUNT do={
if ([/ping $ISP1HOST1 count=1]=0) do={:set F ($F + 1)}
if ([/ping $ISP1HOST2 count=1]=0) do={:set F ($F + 1)}
:delay 2;
};

:if (($F=$PINGTS)) do={
:if (($ISP1STATUS="UP")) do={
:set ISP1STATUS "DOWN";

# Also add status in global variables to be used as tracking
:set date [/system clock get date];
:set time [/system clock get time];
:set ISP1LASTCHANGESTATUS ($time . " " . $date);

##################################################
####### FOR DOWN STATUS, CHANGE THE RULES ########
##################################################
# If the link is down, then LOG info and warning in Mikrotik LOG window [Zaib]

:log info "$ISP1NICK Gateway Not Responding. Please Check Link Connectivity..."
#:log warning "Switching to Backup Link ($ISP2) ..."
#/ip firewall connection remove [find]
# OR change routes as per your requirements
#/ip firewall nat disable [find comment="Route_SOME_traffic_To_WAN1"]
#/ip firewall nat enable [find comment="Route_SOME_traffic_To_WAN2"]
#/ip route set [find comment="PRIMARY_GATEWAY"] distance=3
:delay 2;

# "Emailing the DOWN status. . . "
/tool e-mail send to="$adminmail1" password=$gmailpass subject="$SUBJECTDOWN" from=$gmailid server=$GMAILSMTP start-tls=yes body="$BODYDOWN"

# SENDING SMS for action DOWN
/tool fetch url="http://$KHOST/cgi-bin/sendsms\?username=$KID&password=$KPASS&to=$CELL1&text=$MSGDOWN"

##################################################
####### FOR UP STATUS, CHANGE THE RULES ########
##################################################

} else={:set ISP1STATUS "DOWN";}
} else={
:if (($ISP1STATUS="DOWN")) do={
:set ISP1STATUS "UP";

# If link is UP, then LOG info and warning in Mikrotik LOG window [Zaib]
:log info "$ISP1NICK Gateway RESTORED ..."
log warning "$ISP1 Restored..."
# OR change routes as per your requirements
#/ip firewall nat disable [find comment="Route_SOME_traffic_To_WAN1"]
#/ip firewall nat enable [find comment="Route_SOME_traffic_To_WAN2"]
#/ip route set [find comment="PRIMARY_GATEWAY"] distance=3
:delay 2;

# Emailing the UP status. . .
/tool e-mail send to="$adminmail1" password=$gmailpass subject="$SUBJECTUP" from=$gmailid server=$GMAILSMTP start-tls=yes body="$BODYUP"

# SENDING SMS for action UP
/tool fetch url="http://$KHOST/cgi-bin/sendsms\?username=$KID&password=$KPASS&to=$CELL1&text=$MSGUP"

:set date [/system clock get date];
:set time [/system clock get time];
:set ISP1LASTCHANGESTATUS ($time . " " . $date);

} else={:set ISP1STATUS "UP";}
}

Scheduler to run script auto

To add scheduler to run script after every 5 minutes (or as required), use following code

/system scheduler
add disabled=no interval=5m name="Monitor WAN connectivity Scheduler / JZ" on-event=wan1mon policy=ftp,reboot,read,write,policy,test,winbox,password,sniff,sensitive,api start-date=jun/12/2014 start-time=\
00:00:00

Don’t forget to change the script name wan1mon in above scheduler to match the name you set for the script.

Example: on-event=wan1mon


Define Static Routes for Monitoring Host – for Route Changing

If  you are using this script to change internet route to backup link, then you must define static routes for the host you are monitoring. So that your monitored hosts should always (forcefully) go via Primary Link.

# Adding routes for the 2 monitoring host to go via primary lin
/ip route
add comment="Force this HOST via Primary Link" disabled=no distance=1 dst-address=193.183.98.154/32 gateway=192.168.1.1 scope=30 target-scope=10
add comment="Force this HOST via Primary Link" disabled=no distance=1 dst-address=84.200.70.40/32 gateway=192.168.1.1 scope=30 target-scope=10

# FORCEFULLY - Adding routes for the 2 monitoring host to go in black hole if primary is not reachable (failsafe) to avoid going pkts vai wan2 for these hosts
add comment="Force this HOST via Primary Link - Black-Hole" distance=2 dst-address=193.183.98.154/32 type=blackhole
add comment="Force this HOST via Primary Link - Black-Hole" distance=2 dst-address=84.200.70.40/32 type=blackhole

Note: Make sure to change gateway 192.168.1.1 to primary internet link gateway.


BE VERY SURE TO READ FOLLOWING

https://aacable.wordpress.com/2015/09/17/monitoring-multiple-wan-links-in-pcc-using-black-hole-route-approach/


Mail Alert Notification Example: (reference purpose)


TIPS: [Internet hosts list for monitoring]

Following is list for some internet hosts that I usually monitor via the script, usefull if you want to monitor multiple wan links.

4.2.2.1
4.2.2.2
84.200.70.40
193.183.98.154
202.12.27.33
202.142.160.2
209.244.0.4
192.5.5.241
199.7.83.42

Regard’s
Syed Jahanzaib

%d bloggers like this: