Monitor/Service Network Latency using SmokePing

---

Recently I was troubleshooting a network where concerned Admin complained that they have delays in internet access particular to DNS resolving. To troubleshoot it I configured Linux base mrtg  which do the Ping base graphs to monitor ping latency. The custom made mrtg ping probe worked fine and can provide an overview on target ping / rtt and Downtime in a nice manner,

BUT . . . . . . . . . . . . . . . . . . .

I was thinking far ahead , I was thinking for much more advanced latency and pin point graphs which can show ping latency / rtt / loss in much more detailed way. I recalled my memory from old days when I used to monitor my old network with variety of tools and scripts and suddenly a name popped in my mind ” SMOKEPING ” , yes this was the tool I was looking for.

SmokePing keeps track of your network latency & latency measurement tool. It sends test packets out to the net and measures the amount of time they need to travel from one place to the other and back. Best displays of latency visualization. It supports Interactive graph explorer. Wide range of latency measurement plugins. Free and OpenSource Software by Tobi Oetiker, the creator of MRTG and RRDtool. SmokePing generates graphs that can reveal the quality (packet loss and latency variability). It uses RRDtool as its logging and graphing back-end, making the system very efficient. The presentation of the data on the Web is done through a CGI with some AJAX capabilities for interactive graph exploration.

For every round of measurement smokeping sends several packets. It then sorts the different round trip times and selects the median, (ie. the middle one). This means when there are 10 time values, value number 5 is selected and drawn. The other values are drawn as successively lighter shades of gray in the background (smoke).

Sometimes a test packet is sent out but never returns. This is called packet-loss. The color of the median line changes according to the number of packets lost.

All this information together gives an indication of network health. For example, packet loss is something which should not happen at all. It can mean that a device in the middle of the link is overloaded or a router configuration somewhere is wrong.

Heavy fluctuation of the RTT (round trip time) values also indicate that the network is overloaded. This shows on the graph as smoke; the more smoke, the more fluctuation.

Smokeping is not limited to testing just the roundtrip time of the packets. It can also perform some task at the remote end (“probe”), like download a webpage. This will give a combined ‘picture’ of webserver availability and network health.

How to Read Smokeping Graphs

• Smokeping sends multiples tests (pings), makes note of RTT, orders these and selects the median.
• The different values of RTT are shown graphically as lighter and darker shades of grey (the “smoke”). This conveys the idea of variable round trip times or jitter.
• The number of lost packets (if any) changes the colour of the horizontal line across the graph.

---

• In this article I will show you howto install smokeping on UBUNTU.

First install required components along with smokeping and apache2 (you can remove Apache or any other component if its not required or already installed)

apt-get -y install smokeping apache2

Enable CGI for apache

sudo a2enmod cgi

When you will try to restart smokeping/apache service, you will encounter following error

ERROR: /etc/smokeping/config.d/pathnames, line 1: File '/usr/sbin/sendmail' does not exist
[ OK ]

To remove the error, you have to COMMENT the sendmail direcitve in smokeping pathnames config file

edit the pathnames file by

nano /etc/smokeping/config.d/pathnames

& comment the first line like

#sendmail = /usr/sbin/sendmail

restart smokeping/apache2 service.  Now you can access the SmokePing Web panel by pointing to

• http://your.ip/smokeping/smokeping.cgi

If you wan to avoid adding smokeping.cgi every time, modify the apache config as below. To add the smokeping.cgi as default page opener index, follow below

Edit Apache Index file

nano /etc/apache2/mods-available/dir.conf

& add smokeping.cgi at endor start under DirectoryIndex directives, example shown below

<IfModule mod_dir.c>
DirectoryIndex smokeping.cgi admin.php index.html index.cgi index.pl index.php index.xhtml index.htm
</IfModule>

Save & Exit & restart apache by

service apache2 reload

 

---

Modifying the SmokePing The configurations

Once the smokeping is installed, we have to modify few configuration files. Mainly includes Probes & Targets  config files. Probes contains services related config, & the Targets contains the hosts we want to monitor.

Config Files locations are located at:

• /etc/smokeping/config.d

First the PROBES config, Edit it by

nano /etc/smokeping/config.d/Probes

Remove older lines & add below,

*** Probes ***
+ FPing
binary = /usr/bin/fping
#forks = 100
pings = 3

+ EchoPingHttp
binary = /usr/bin/echoping
forks = 100
offset = 50%
step = 300

+ EchoPingHttps
binary = /usr/bin/echoping
pings = 3
forks = 100
step = 300
port = 443
url = /

+ DNS
binary = /usr/bin/dig # mandatory
step = 180
lookup = google.com
pings = 3

---

Now the Targets Config File which will contains all the remote hosts we want to monitor, either there ping, services etc, Edit it by

 

nano /etc/smokeping/config.d/Targets

 

Now REMOVE all previous lines , and copy paste following

# You can modify add/remove any line of your chocie particular to TARGETS

*** Targets ***
probe = FPing

menu = Top
title = Network Latency Grapher
remark = "Welcome to the SmokePing stats for <b>SJZ.</b><br>Here you will learn all about the latency of our network.<br><br>This page is maintained by SJZ LTD.<br>Support Email: aacableAThotmailDOTcom<br>Web: https://aacableDotwordpressDotcom"

### YOU CAN CHANGE THE FOLLOWING ACCORDING TO YOUR NETWORK ###

# Remote HTTPS Web server response

+HTTPSchk
menu = Remote HTTPS Response
title = HTTPS Response Remote Hosts

++SJZweb
menu = SJZ WWW
title = cnn.com HTTPS RTT
probe = EchoPingHttps
host = cnn.com
url = /
accept_redirects = yes
++youtube
menu = Youtube WWW
title = youtube.com HTTPS RTT
probe = EchoPingHttps
host = youtube.com
url=/
accept_redirects = yes

# Check WAN Servers PING RTT results

+ Ping
menu = WAN Connectivity
title = WAS Side Network

# Change Below
++ ISP1gw
menu = ISP-1.GW
title = ISP-1 GW RTT
host = 1.2.3.4
++ ISP2
menu = ISP2.GW
title = ISP2 GW Ping RTT
host = 1.2.3.5
++ ISP2status1
menu = ISP2 Internet Hosted Site1 RTT
title = ISP2 Internet Status-1 Ping Report
host = 1.2.3.6
++ ISP2status2
menu = ISP2 Internet Hosted Site2 RTT
title = ISP2 Internet Status-2 Ping Report
host = 1.2.3.7
++ ISP2fiberstatus
menu = ISP2 Fiber RTT
title = ISP2 Fiber GW status Ping Report
host = 1.2.3.8
++ ISP2radiostatus
menu = ISP2 Radio RTT
title = ISP2 Radio status ping report
host = 1.2.3.9
++ googledns
menu = google.dns
title = Google DNS Ping RTT
host = 8.8.8.8
++ onedotonedns
menu = onedotone.dns
title = OneDotOne DNS Ping RTT
host = 1.1.1.1

# Check LAN Servers PING RTT results

+ Ping2
menu = LAN Connectivity
title = LAN Side Network
++ SJZ-Mikrotik
menu = Mikrotik WAN GW
title = SJZ WAN Gateway Mikrotik Local Ping
host = 192.168.10.36
++ dc-10.5
menu = DC01 DC
title = SJZ1.Local DC Ping
host = 192.168.10.5
++ dc-10.6
menu = DC02 DC
title = SJZ2.Local DC Ping
host = 192.168.10.6

# Check DNS Servers delay/latency Results

+ DNS-Latency
probe = DNS
menu = DNS latency
title = Service latency (DNS)

++ DNS-1-1-1-1
title = DNS Latency for Public DNS one.one.one.one
host = 1.1.1.1
++GoogleA
menu = 8.8.8.8
title = DNS Latency for google-public-dns-a.google.com
host = 8.8.8.8
++GoogleB
menu = 8.8.4.4
title = DNS Latency for google-public-dns-b.google.com
host = 8.8.4.4
++OpenDNSA
menu = 208.67.222.222
title = DNS Latency for resolver1.opendns.com
host = 208.67.222.222
++OpenDNSB
menu = 208.67.220.220
title = DNS Latency for resolver2.opendns.com
host = 208.67.220.220

# Create Consolidated / DNS latency Servers Combined Graphs

+ DNSAllin1Graph
menu = All DNS Combined Graph
title = Consolidated DNS Responses
host = /DNS-Latency/GoogleA /DNS-Latency/DNS-1-1-1-1 /DNS-Latency/GoogleB /DNS-Latency/OpenDNSA /DNS-Latency/OpenDNSB

save and exit.

Now restart SmokePing service by

service smokeping restart

& access it via browser.

http://yourip/smokeping/smokeping.cgi

Results should be something like below image…

---

TIPS:

---

Relevance of the light/dark gray entries on the smokeping graphs

• The relevance of the light/dark gray entries on the smokeping graphs? Is this the variance over the 20 requests?

Actually it’s not really the variance but all of the n values scattered around the median. The changing shades of gray show the change in latency from the average, that darker the smoke the greater the number of pings that range away from the average. This gray bar or smoke can either be above the line or below the line. A good circuit would have very little gray and very low loss.

Understand the term “median”. It doesn’t mean average (thus the colored bar would be in the middle), it means “there are an equal number of values above and below”.

Let’s say you have 10 pings that look like this:

• 10 10 10 10 10 10 10 10 10 10 30

Your median is 10 (average is 11). Thus the colored line would be right smack at the bottom and you’d have a very light gray bar going up to 30ms.

That means you have little jitter and that’s a good thing.

Now if your ping is

• 10 100 10 20 30 40 50 60 100 15

the median is 35 (but the average would be 43.5) because at 35 there are equal values below and above it. Then you would see bars of varying darkness above and below the colored line.

---

Customzied basepage.html

Location: /etc/smokeping

Demo:

The Code

<!doctype html><html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="">
<meta name="author" content="<##owner##>">
<META HTTP-EQUIV="Cache-Control" content="no-cache">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Refresh" CONTENT="<##step##>">

<TITLE>SmokePing - <##title##></TITLE>

<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/css/bootstrap.min.css" integrity="sha384-9gVQ4dYFwwWSjIDZnLEWnxCjeSWFphJiwGPXr1jddIhOegiu1FwO5qRGvFXOdJZ4" crossorigin="anonymous">

<!-- Custom styles for this template -->
<style type="text/css">
body {
font-size: .875rem;
padding-bottom: 50px;
}

html {
position: relative;
min-height: 100%;
}
/*
* Sidebar
*/

.sidebar {
position: fixed;
top: 0;
bottom: 0;
left: 0;
z-index: 100; /* Behind the navbar */
padding: 48px 0 0; /* Height of navbar */
box-shadow: inset -1px 0 0 rgba(0, 0, 0, .1);
}

.sidebar-sticky {
position: relative;
top: 0;
height: calc(100vh - 48px);
padding-top: .5rem;
overflow-x: hidden;
overflow-y: auto; /* Scrollable contents if viewport is shorter than content. */
}

@supports ((position: -webkit-sticky) or (position: sticky)) {
.sidebar-sticky {
position: -webkit-sticky;
position: sticky;
}
}

.sidebar-heading {
font-size: .75rem;
text-transform: uppercase;
}

.sidebar #smokeping {
font-weight: 500;
display: block;
padding: .5rem 1rem;
}

#smokeping .menulink {
color: #007bff;
}

#smokeping .menulinkactive, #smokeping .menuactive .menulink {
color: #f9781c;
}

#smokebody img {
padding-bottom: 5px;
}

/*
* Content
*/

[role="main"] {
padding-top: 48px; /* Space for fixed navbar */
}

/*
* Navbar
*/

.navbar-brand {
padding-top: .75rem;
padding-bottom: .75rem;
font-size: 1rem;
background-color: rgba(0, 0, 0, .25);
box-shadow: inset -1px 0 0 rgba(0, 0, 0, .25);
}

.navbar .form-control {
padding: .75rem 1rem;
border-width: 0;
border-radius: 0;
}

.form-control-dark {
color: #fff;
background-color: rgba(255, 255, 255, .1);
border-color: rgba(255, 255, 255, .1);
}

.form-control-dark:focus {
border-color: transparent;
box-shadow: 0 0 0 3px rgba(255, 255, 255, .25);
}

/*
* Utilities
*/

.border-top { border-top: 1px solid #e5e5e5; }
.border-bottom { border-bottom: 1px solid #e5e5e5; }

/*
* Footer
*/
.footer {
position: absolute;
bottom: 0;
width: 100%;
height: 50px;
clear: right;
}

.footer hr {
margin-bottom: 8px;
}

.footer p {
text-align: right;
margin-bottom: 0;
}

</style>
</head>

<body>
<nav class="navbar navbar-dark fixed-top bg-dark flex-md-nowrap p-0 shadow">
<a class="navbar-brand col-sm-3 col-md-2 mr-0" href="?">SmokePing</a>
</nav>

<div class="container-fluid">
<div class="row">
<nav class="col-md-2 d-none d-md-block bg-light sidebar">
<div class="sidebar-sticky">
<h6 class="sidebar-heading d-flex justify-content-between align-items-center px-3 mt-4 mb-1 text-muted">
<span>SmokePing Targets</span>
</h6>
<ul class="nav flex-column mb-2">
<li class="nav-item">
<div id="smokeping">
<##menu##>
</div>
</li>
</ul>
</div>
</nav>

<main role="main" class="col-md-9 ml-sm-auto col-lg-10 px-4">
<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
<h1 class="h2"><##title##></h1>
</div>

<h5><##remark##></h5>
<div id="smokebody">
<P><##overview##></P>
<P><##body##></P>
</div>
</main>
</div>
</div>
<footer class="footer">
<div class="container-fluid">
<hr>
<p class="text-muted"><##smokeping##> by <##author##> - Template by <a href="https://www.simonmott.co.uk">Simon Mott</a></p>
</div>
</footer>
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.0/umd/popper.min.js" integrity="sha384-cs/chFZiN24E4KMATLdqdvsezGxaGsi4hLGOzlXwp5UZB1LY//20VyM2taTB4QvJ" crossorigin="anonymous"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.0/js/bootstrap.min.js" integrity="sha384-uefMccjFJAIv6A+rW+L4AHf99KvxDjWSu1z9VI8SKNVmz4sk7buKt/6v9KI65qnm" crossorigin="anonymous"></script>
<script src="js/prototype/prototype.js" type="text/javascript"></script>
<script src="js/scriptaculous/scriptaculous.js?load=builder,effects,dragdrop" type="text/javascript"></script>
<script src="js/cropper/cropper.js" type="text/javascript"></script>
<script src="js/smokeping-zoom.js" type="text/javascript"></script>
</body>
</html>