Syed Jahanzaib Personal Blog to Share Knowledge !

May 12, 2019

Facilitate CDN traffic with Mikrotik

Filed under: Mikrotik Related — Tags: , , , — Syed Jahanzaib / Pinochio~:) @ 12:49 PM

ncdn_-_cdn.png

Control / Facilitate CDN traffic with

~ Mikrotik Router ~

First some DRY theory !

CDNs replicate content in multiple places. There’s a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network. The general idea of a CDN is to deliver content as fast as possible to the user without compromising the user’s experience. Usually, a CDN have global location servers, called Point of Presence. These PoPs store data as cache. When a user requests for a website, the nearest PoP will handle the request using stored cache.

The BIG players such as Google in order to enhance user experience have tried to get as close to the user as possible by direct peering with the regional service providers and provide contents using CDN (Content delivery network) providers. Google is having its own CDN network branded as a service called Google Global Cache (GGC)

Nowadays all the major ISPs have CDN facility , which tremendously helps them to reduce burden on there internet feed. Without CDN, cost of real internet bandwidth will be a heavy burden for any OP. With CDN user will get better video streaming experience.

I know few ISP’s here in Karachi (& one particularly originated from Gulshan Area) which totally relies on CDN (more than 50-60% of there internet data is routing via CDN) , I have used one of them, there real internet speed is pathetic but if you browse YT/FB they works excellent.


Scenario:

Our upstream ISP have CDN server installed in there data center & traffic going to CDN have no limit. But we want to control the traffic as following

1 Mb package Users break up for bandwidth controlling …

  • 1mb internet bandwidth
  • 1mb CDN bandwidth

So if a user is surfing the internet he will get full 1mb internet speed, & if he uses the traffic going to YOUTUBE CDN ,  He will get another 1 mb (additional).
Virtually he will get 2mb in total.

Using Mikrotik, we can achieve this task by using Firewall Mangle & Queues Tree. Same can be done with Mangle & PCQ base simple queues too. It’s a debatable topic on what to use, & depends on the selection, mangle marking method would also be changed.

Every network is different so one configuration cannot fit all. Number of users & traffic volume plays vital role in selection of marking / queue type to use.

Choose the marking/queue type wisely to save your Mikrotik CPU from becoming Mr. SPIKY 🙂 YKWIM 😀

Disclaimer: This is just an example for sharing purposes ONLY & yes there are many other methods and tuning techniques you can adopt to make this process much more efficient.


Script !

#===================================================
# CDN PACKET MARKING SCRIPT using Mangle/Queue.Tree
# By Syed.Jahanzaib
# Email: aacableAThotmailDOTcom
# https://aacableDOTwordpressDOTcom
# March 2019
#===================================================
# Address list name which is created dynamically by radius or you can go with manual method too
# This is important ... it can be done by varieties of ways, select one that matches your network design
#1Mb
#2Mb

#Create Address List which will contain CDN server's IP addresses
/ip firewall address-list
add address=1.2.3.4/24 list=cdn_list
add address=5.6.7.8/32 list=cdn_list

# Copy paste following rules & make sure to move these MANGLE rules to TOP position,
# so that they can be applied before any other rule, (for cdn)

/ip firewall mangle
add action=mark-packet chain=postrouting dst-address-list=cdn_list new-packet-mark=cdn_1mb_up passthrough=no src-address-list=1Mb
add action=mark-packet chain=postrouting dst-address-list=1Mb new-packet-mark=cdn_1mb_down passthrough=no src-address-list=cdn_list

add action=mark-packet chain=postrouting dst-address-list=cdn_list new-packet-mark=cdn_2mb_up passthrough=no src-address-list=2Mb
add action=mark-packet chain=postrouting dst-address-list=2Mb new-packet-mark=cdn_2mb_down passthrough=no src-address-list=cdn_list

# Define Queue Type & limitation that we want to provide to each package
/queue type
add kind=pcq name=1mb-cdn-download pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=1024k pcq-src-address6-mask=64 pcq-total-limit=1024KiB
add kind=pcq name=1mb-cdn-upload pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=1024k pcq-src-address6-mask=64 pcq-total-limit=1024KiB

add kind=pcq name=2mb-cdn-download pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=2048k pcq-src-address6-mask=64 pcq-total-limit=2048KiB
add kind=pcq name=2mb-cdn-upload pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=2048k pcq-src-address6-mask=64 pcq-total-limit=2048KiB

# Add Queue/Speed Limitation using above Queue Types to firewall mangled/marked packets

/queue tree
add name="CDN - 1mb - upload" packet-mark=cdn_1mb_up parent=global priority=1 queue=1mb-cdn-upload
add name="CDN - 1mb - download" packet-mark=cdn_1mb_down parent=global priority=1 queue=1mb-cdn-download

add name="CDN - 2mb - upload" packet-mark=cdn_2mb_up parent=global priority=1 queue=2mb-cdn-upload
add name="CDN - 2mb - download" packet-mark=cdn_2mb_down parent=global priority=1 queue=2mb-cdn-download

# Script Ends Here.

1mb users CDN usage Graph.

cdn

As shown in above example image, 1mb users are using 227 mb of CDN (YT) bandwidth,  (it was off time with lesser number of users, in peak traffic reaches in Gb’s) & real internet bandwidth is free OR available for other tasks/users,  thus providing relief to the real internet bandwidth pipe.


Regard’s
Syed Jahanzaib

durood

April 5, 2019

Mikrotik with Freeradius/mySQL # Part-22 – Create Dynamic Address List using Mikrotik-Address-List Attribute

Filed under: freeradius, Mikrotik Related — Tags: — Syed Jahanzaib / Pinochio~:) @ 12:28 PM

fre



Disclaimer! This is important!

Every Network is different , so one solution cannot be applied to all. Therefore try to understand logic & create your own solution as per your network scenario. Just dont follow copy paste.

If anybody here thinks I am an expert on this stuff, I am NOT certified in anything Mikrotik/Cisco/Linux or Windows. However I have worked with some core networks and I read , research & try stuff all of the time. So I am not speaking/posting about stuff I am formerly trained in, I pretty much go with experience and what I have learned on my own. And , If I don’t know something then I read & learn all about it.

So , please don’t hold me/my-postings to be always 100 percent correct. I make mistakes just like everybody else. However – I do my best, learn from my mistakes and always try to help others.

Regard’s
Syed Jahanzaib~


Scenario:

  • We have FREERADIUS installed as a AAA system in Ubuntu 16.04 server
  • Mikrotik version 6.44 is acting as PPPoE NAS connected with radius for AAA

Requirement:

When any user connects with our NAS, he should be added to mikrotik dynamic address list under IP > firewall > address list, so that we can manipulate this address list for different tasks, example mark connections/packets/routing and use them in Queues / Routes section or perform different sort of filtering as required.

In this particular task we are dynamically adding user in particular address list using radius attributes, then using this address list packet marking is being made, and then in Queues we are using these marked packets for different sort of bandwidth policies, example for normal internet we will limit 1mb per user , and for CDN traffic we will add addition 2mb for YT & FB. [and vice versa for different packages accordingly]

 


Solution:

We will use Mikrotik-Address-List attribute in radgroupreply section. as shown here.

1# Adding User entry in RADCHECK table so user can authenticate …

mysql> select * from radcheck;
+----+----------+--------------------+----+-------------------+
| id | username | attribute | op | value |
+----+----------+--------------------+----+-------------------+
| 1 | zaib | Cleartext-Password | := | zaib |
+----+----------+--------------------+----+-------------------+
1 rows in set (0.01 sec)

2# Adding Radius Group Reply for 1mb Group, Example 1mb group user will get 1mb dynamic queue plus they will be added dynamically in address list name 1mb

mysql> select * from radgroupreply;
+----+-----------+-----------------------+----+--------------+
| id | groupname | attribute | op | value |
+----+-----------+-----------------------+----+--------------+
| 21 | 1mb | Mikrotik-Rate-Limit | == | 1024k/1024k |
| 22 | 1mb | Mikrotik-Address-List | := | 1mb |
+----+-----------+-----------------------+----+--------------+
2 rows in set (0.00 sec)

2# Adding username ZAIB in the Radius user group & assign him 1mb Group.

 

mysql> select * from radusergroup;
+----+----------+-----------+----------+
| id | username | groupname | priority |
+----+----------+-----------+----------+
| 5 | zaib | 1mb | 1 |
+----+----------+-----------+----------+
1 row in set (0.00 sec)

RADTEST:

Now we will test user via RADTEST cmd …


radtest zaib zaib localhost 1812 testing123

Result:

Sending Access-Request of id 130 to 127.0.0.1 port 1812
User-Name = "zaib"
User-Password = "zaib"
NAS-IP-Address = 101.11.11.254
NAS-Port = 1812
Message-Authenticator = 0x00000000000000000000000000000000
rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=130, length=50

Mikrotik-Rate-Limit = "1024k/1024k"
Mikrotik-Address-List = "1mb"

Freeradius Debug Result:

Sending Access-Accept of id 156 to 127.0.0.1 port 34563
Mikrotik-Rate-Limit == "1024k/1024k"
Mikrotik-Address-List := "1mb"
Finished request 32.

Now try to connect user from your user device, & upon connection you will see new address list entry for this user IP ..

& its 1mb queues have been created as well


# Mikrotik Mangling & Queueing Section !

Now we will move towards Mikrotik related configuration for mangling & queue. in above steps we added DYNAMIC queue for test purposes, & as we will be using simple queues therefore we need to remove the dynamic queue, Do so , then we will move further …

  • Marking upload & download separately for 1mb user address list …

/ip firewall mangle
add action=mark-packet chain=forward comment="1mb users UPLOAD" new-packet-mark=1mb_users_up src-address-list=1mb passthrough=no
add action=mark-packet chain=forward comment="1mb users DOWNLOAD" dst-address-list=1mb new-packet-mark=1mb_users_down passthrough=no
  • Creating PCQ base 1mb download/upload limit variable …
/queue type
add kind=pcq name=download-1mb pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=1024k pcq-src-address6-mask=64
add kind=pcq name=upload-1mb pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=1024k pcq-src-address6-mask=64
  • Creating PCQ base simple Queues to actual limit each user with 1mb download/upload …
/queue simple
add name="1mb user DOWN - PCQ" packet-marks=1mb_users_down queue=upload-1mb/download-1mb target=""
add name="1mb user UP - PCQ" packet-marks=1mb_users_up queue=128k-per-user/128k-per-user target=""

 

PC#1

1st- user - 1mb user test

PC#2

2nd pc 128K 1mb


TIPS:

How to remove all dynamic queues [can be used in script login section]

dynamic queue removal.PNG

/queue simple remove [find where dynamic]

Conclusion:

As we can see that address list have been created successfully, now we can manipulate it for our different tasks using marked packets for customized PCQ base queues for policy base queueing.

I will write more on it later if manage to get some spare time.


 

Regard’s
Syed Jahanzaib

 

March 25, 2019

Mikrotik with Freeradius/mySQL # Part-21 – Weird Trigger for Duplicate Users

Filed under: Mikrotik Related, Radius Manager — Tags: , — Syed Jahanzaib / Pinochio~:) @ 11:13 AM

dup user.jpg

fre



Disclaimer! This is important!

Every Network is different , so one solution cannot be applied to all. Therefore try to understand logic & create your own solution as per your network scenario. Just dont follow copy paste.

If anybody here thinks I am an expert on this stuff, I am NOT certified in anything Mikrotik/Cisco/Linux or Windows. However I have worked with some core networks and I read , research & try stuff all of the time. So I am not speaking/posting about stuff I am formerly trained in, I pretty much go with experience and what I have learned on my own. And , If I don’t know something then I read & learn all about it.

So , please don’t hold me/my-postings to be always 100 percent correct. I make mistakes just like everybody else. However – I do my best, learn from my mistakes and always try to help others.

Regard’s
Syed Jahanzaib~


Scenario:

  • We have DMASOFTLAB radius manager installed as a billing system in Ubuntu 12.04 server
  • Mikrotik version 6.4x.x is acting as Hotspot NAS and connected with radius for AAA

Requirement: [A Weird one really]

As operator demanded

“We are running Hotspot on mikrotik, & client login to hotspot using his mobile/laptop. If logged-in client leaves his primary location without logout, & move to another location, & if he try to login from another device, his request will gets DENY because of Single user limit. We increased it to 2 by using SIM-USE=2 directive in user properties,It allows second session to login, but both sessions can use the bandwidth, therefore we want that once second session is established its old first live session should get kicked. If it was single Hotspot we could have used the script on LOGIN, but there are several NAS spreaded across various location using single radius.”

if the user uses same device then we could have used

if (User-Name){
if("%{sql:UPDATE radacct set AcctStopTime=ADDDATE(AcctStartTime,INTERVAL AcctSessionTime SECOND), AcctTerminateCause='Clear-Stale Session' WHERE UserName='%{User-Name}' and CallingStationId='%{Calling-Station-Id}' and AcctStopTime is NULL}"){
}
}

but things are different in hotspot as I have observed, if devices are different then it will give us already logged-in error, if we use sim-use=2 then second device can be logged-in but old session will also be alive and both ids will suck the bandwidth at a time.

Also using idle-timeout or keep-alive timeout is the simplest way to achieve this , but for some weird reasons and to avoid long arguments dueto accent issues, I made one customized solution for the operator.


Solution:

Login to mysql with root

mysql -uroot -pXXXX

and switch to radius database

use radius;

Now create new table that will hold duplicate users record

MYSQL Table to hold duplicate users list


--
-- Table structure for table `rm_dupusers`
--

DROP TABLE IF EXISTS `rm_dupusers`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `rm_dupusers` (
`dupid` int(9) NOT NULL AUTO_INCREMENT,
`datetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`username` varchar(64) NOT NULL,
`ip` varchar(16) NOT NULL,
`nas` varchar(16) NOT NULL,
`comments` varchar(64) DEFAULT NULL,
KEY `dupid` (`dupid`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `rm_dupusers`
--

MYSQL TRIGGER to check duplicate users sessions

Now we will create a new Trigger that will be executed when any record is inserted in radacct, it will check for existing duplicate session of user and if it found , it will add its entry in the mysql table of rm_dupusers

drop trigger chk_dup_user;
DELIMITER ;;
/*!50003 CREATE*/ /*!50017 DEFINER=`root`@`localhost`*/ /*!50003 TRIGGER `chk_dup_user` BEFORE INSERT ON `radacct` FOR EACH ROW BEGIN
SET @dupuserchk = (SELECT count(*) from radacct where username=New.username and acctstoptime is NULL);
IF (@dupuserchk = 1) THEN
SET @dupusername = (SELECT username from radacct where username=New.username and acctstoptime is NULL);
SET @dupuserip = (SELECT framedipaddress from radacct where username=New.username and acctstoptime is NULL);
SET @dupusernas = (SELECT nasipaddress from radacct where username=New.username and acctstoptime is NULL);
INSERT into rm_dupusers (dupid,username,ip,nas,comments) values ('',@dupusername,@dupuserip,@dupusernas,'Duplicate User');
END IF;
END */;;
DELIMITER ;

Mysql Part is Done.

Now we will create a BASH script that will scheduled to run every minute.

BASH script !

Create bash script in desired folder, in this example I am using /temp folder as default

mkdir /temp
touch /temp/kickdupuser.sh
chmod +x /temp/kickdupuser.sh
nano /temp/kickdupuser.sh

& paste following, make sure to modify credentials

#!/bin/bash
#set -x
# Following script is made specifically for Dmasoftlab radius manager 4.1.x
# When any new user will login, it will simply check if exists session of same user found, it will kick previous session
# it requires custom trigger on radacct table, this script will be schedule to run every minute
# Created: 25-MARCH-2019
# Tested on Ubuntu OS Only
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#################
# CHANGE these
HOSTNAME=`hostname`
SQLID="root"
SQLPASS="XXXXXX"
NAS_COA_PORT="1700"
DB="radius"
SRV="mysql"
DUP_TABLE="rm_dupusers"
INT="1"
RADCLIENT="/usr/local/bin/radclient"
#################
#DATE TIME FUNCTIONS
currenttime=$(date +%H:%M:%S)
# Add Script start execution entry in the /var/log/syslog to see if the script got executed or not
logger "Duplicate User poller script Started @ $currenttime by the CRON scheduler ... Powered by SYED.JAHANZAIB"
echo "- Script Start Time - $currenttime"
echo "- Checking Duplicate Users in $DUP_TABLE table ..."
export MYSQL_PWD=$SQLPASS
CMD="mysql -u$SQLID --skip-column-names -s -e"
#Table which contain main users information
TMPUSRINFO=/tmp/userpass.txt
TEMP="/temp"

# Checking if /temp folder is previously present or not . . .
{
if [ ! -d "$TEMP" ]; then
echo
echo "- INFO: $TEMP folder not found, Creating it now to store logs ..."
mkdir $TEMP
else
echo -e "- INFO: $TEMP folder is already present to store logs."
echo
fi
}

DUP_LIST_FILE=$TEMP/duplicate_users_list.txt
SYSLOG="/var/log/syslog"
> $TMPUSRINFO

# KANNEL DETAILS
KHOST="127.0.0.1:13013"
KID="kannel"
KPASS="KANNEL_PASSWORD"

IPADD=`ip route get 1 | awk '{print $NF;exit}'`
SRVSTATUS=`service $SRV status |grep running |wc -l`
if [ "$SRVSTATUS" -ne 1 ];
#if [ -z "$SRVSTATUS" ];
then
echo "- ALERT: $HOSTNAME - $IPADD - $SRV NOT RESPONDING CHECK - $DATE $DT .Exiting ..."
echo "- ALERT: $HOSTNAME - $IPADD - $SRV NOT RESPONDING CHECK - $DATE $DT .Exiting ..." >> $SYSLOG
echo "- ALERT:

- $HOSTNAME
- $IPADD
- $SRV not responding ***
- $currenttime

Exiting ..."
exit 1
else
echo "- INFO: $SRV service is accessible. Proceeding further ... OK"
fi

# Check if table exists
if [ $($CMD \
"select count(*) from information_schema.tables where \
table_schema='$DB' and table_name='$DUP_TABLE';") -eq 1 ]; then
echo "- INFO: $DUP_TABLE Table exists ..."
else
echo "- WARNING: $DUP_TABLE Table does not exists ..."
fi
########
########
# Enable following line so that it will update all users simultanous-use to '2' so that two sessions can be established
# $CMD "use $DB; UPDATE  radius.radcheck SET value = '2' where Attribute = 'Simultaneous-Use';
#######
#######
# pull user record
$CMD "use $DB; select username,ip,nas from $DUP_TABLE WHERE datetime >= NOW() - INTERVAL $INT MINUTE;" >> $TMPUSRINFO
if [ ! -s $TMPUSRINFO ]
then
endtime=$(date +%H:%M:%S)

echo "
- INFO: No Duplicate User found in DMA RADIUS MANAGER TABLE '$DUP_TABLE' , Sending EXIT signals ...

- Script Ends Here...
- EXITING peacefully...
- Script End Time - $endtime
"
exit 1
fi

# Apply Count Loop Formula while deleting first line which have junk text
num=0
cat $TMPUSRINFO | while read users
do
num=$[$num+1]
username=`echo $users | awk '{print $1}'`
USER_IP=`echo $users | awk '{print $2}'`
ACCTSESID=`$CMD "use $DB; select acctsessionid from radacct where framedipaddress ='$USER_IP' AND acctstoptime is NULL;"`
NAS_IP=`echo $users | awk '{print $3}'`
NAS_SECRET=`$CMD "use $DB; select secret from nas where nasname = '$NAS_IP' ;"`

# Print Info on screen
echo "Duplicate User Found: USER: $username , IP: $USER_IP, ID: $ACCTSESID, $NAS: $NAS+IP @ $currenttime ... KICKING him now ..."
echo "Duplicate User Found: USER: $username , IP: $USER_IP, ID: $ACCTSESID, $NAS: $NAS+IP @ $currenttime ... KICKING him now ..." >> $DUP_LIST_FILE
#echo User-Name=$USERNAME,Acct-Session-Id=$ACCTSESID,Framed-IP-Address=$USER_IP,Mikrotik-Rate-Limit=\"$DN_BWPKG\" | $RADCLIENT -q -c 1 $NAS_IP:$NAS_COA_PORT coa $NAS_SECRET
#for hotspot, enable following line
echo Framed-IP-Address=$USER_IP | radclient -x -c 1 $NAS_IP:$NAS_COA_PORT disconnect $NAS_SECRET
done
# once done, we should delete the tmp files to clear the garbage
rm $TMPUSRINFO

CRON scheduler to run the above script every minute. Edit crontab by

crontab -e

& add following entry

* * * * * /temp/kickdupuser.sh >/dev/null 2>&1

Testing …

Using same credentials, Login to first device, and then on second ,

& run this script,

root@radius:/temp# /temp/kickdupuser.sh
- Script Start Time - 10:52:03
- Checking Duplicate Users in rm_dupusers table ...
- INFO: /temp folder is already present to store logs.
- INFO: mysql service is accessible. Proceeding further ... OK
- INFO: rm_dupusers Table exists ...
Duplicate User Found: USER: test , IP: 172.16.0.253, ID: 81d00057, : +IP @ 10:52:03 ... KICKING him now ...
Sending Disconnect-Request of id 58 to 10.0.0.1 port 1700
Framed-IP-Address = 172.16.0.253
rad_recv: Disconnect-ACK packet from host 10.0.0.1 port 1700, id=58, length=32
NAS-Identifier = "ZAIB_CCR_GW"
root@radius:/temp#

older session will be removed

radclient dc the first user.PNG


Weirdo …. but its fun to learn !

TIPS:

Command to view duplicate users session in freeradius using CLI


mysql -uroot -pMYPASS --skip-column-names -e 'use radius; SELECT username FROM radacct WHERE acctstoptime IS NULL;' > 1.txt && sort 1.txt | uniq -cd

Regard’s
Syed Jahanzaib

February 1, 2019

Forced routing of selective emails to ISP SMTP via Mikrotik Routing

Filed under: IBM Related, Mikrotik Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 10:06 AM

isp.jpeg


Scenario:

We have a LAN environment with our own email server [IBM Lotus Domino] hosted locally. Mikrotik router is acting as our gateway router with /29 public pool & port forwarding from mikrotik public ip to email server is configured. Barracuda Antispam gateway is in place as well.

Problem & Challenges :

Sometimes there are few email servers on the internet that does not accept our emails, either they bounce back or silently drop our emails despite our public IP is not listed in any of blacklisting on the internet[It happens commonly with microsoft hosted email servers as they silently drop our emails without informing any reason]. If we use our ISP SMTP as relay in the DOMINO configuration, then the emails delivers to those particular servers without problem. But we cannot use ISP SMTP for all emails routing/relaying as they have per day sending limit, and we donot get proper reports for delivered or hold emails.

Another BIG problem is that sometimes ISP’s SMTP server IP gets ban/added in the spamhaus or likewise SPAM blacklist database & when this happens 80-90% emails bounces back.

So we needed a solution where we should not use ISP SMTP relay all the time but only particular destination email server’s mails should be routed to ISP smtp. & it should all be controlled by our Mikrotik RouterOS dynamically/centrally.


Solution:

First created a address list which should contain IP addresses of remote email servers [that donot accept our emails directly]

/ip firewall address-list
add address=smtp.remotemail.server.com comment="remote company mail server X IP" list=few_mails_routing_2_primary_ISP_smtp

Now using NAT rule, we will forcefully route all emails [port 25 traffic] going to above address list, will be routed to ISP SMTP , with below rule …

# 1.2.3.4 is the ISP SMTP IP

/ip firewall nat
add action=dst-nat chain=dstnat comment="Few Mails Routing 2 primary ISP smtp" dst-address-list=few_mails_routing_2_primary_ISP_smtp dst-port=25 protocol=tcp to-addresses=1.2.3.4 to-ports=25

It’s done.

BUT next challenge is to overcome issue when ISP changes it’s SMTP IP address for whatsoever reason, so we need to schedule a script that will keep checking the ISP SMTP IP by resolving it via google dns, and update the ISP SMTP IP in the NAT rule. [As per my knowledge we cannot put DNS name in TO-ADDRESS field, this is why putting IP is necessary, & update it dynamically is also essential to avoid bouncing email dueot blacklisting for ISP old SMTP IP]

the Script !

or workaround I suggest for very particular problem?

# Mikrotik routerOS script to resolve ISP SMTP, and add it to variables & in NAT rules
# Useful in scneario where ISP change its smtp IP frequently (to avoid SMTP Blacklisting)
# Script by Syed Jahanzaib / aacable at hotmail dot com / https : // aacable . wordpress . com
# 31-January-2019
# Find rule with following comments
:local COMMENT "few_mails_routing_2_primary_ISP_smtp";
# DNS Name of SMTP for resolving
:local ISP1SMTPDNSNAME "smtp.multi.net.pk";
# Which DNS server to be used for resolving
:local DNSSERVER "8.8.8.8";
# Below is Default IP of SMTP Server, so that if resolving cannot be done for what so ever reason, set this IP as DEFAULT SMTP
:local DEFAULTSMTP "202.141.224.89";
# Destination port that need to be redirected
:local DSTPORT "25";
# Dat time variables
:local i 0;
:local F 0;
:local date;
:local time;
:local sub1 ([/system identity get name])
:local sub2 ([/system clock get time])
:local sub3 ([/system clock get date])
:set date [/system clock get date];
:set time [/system clock get time];
# Set script last execution date time
:global SMTPLastCheckTime;
:set SMTPLastCheckTime ($time . " " . $date);

# Set global variables to store for ISP SMTP & its last resolved status
:global ISP1ACTIVEIP4SMTP;
:global ISP1SMTPLASTRESOLVERESULT;

# Check if resolving is doable, then act accordingly
:local RESOLVELIST {"$ISP1SMTPDNSNAME"}
:foreach addr in $RESOLVELIST do={
:do {:resolve server=$DNSSERVER $addr} on-error={
:set ISP1ACTIVEIP4SMTP "$DEFAULTSMTP";
:set ISP1SMTPLASTRESOLVERESULT "FAILED";
:log error "$ISP1SMTPDNSNAME resolved result: FAILED @ $date $time !";
/ip firewall nat set to-addresses=$DEFAULTSMTP to-ports=$DSTPORT [find comment="$COMMENT"] }}

# If resolving is ok from above results then set resolved address as default SMTP ip
:if ($SP1SMTPLASTRESOLVERESULT !="FAILED") do={
:log warning "$ISP1SMTPDNSNAME resolved result: SUCCESS @ $date $time !";
:set ISP1ACTIVEIP4SMTP [:resolve "$ISP1SMTPDNSNAME"];
:set ISP1SMTPLASTRESOLVERESULT "SUCCESS";
/ip firewall nat set to-addresses=$ISP1ACTIVEIP4SMTP to-ports=$DSTPORT [find comment="$COMMENT"]
}

We can add dynamic names in the ISP SMTP address list.


Regard’s
SYED JAHANZAIB

 

 

 

November 19, 2018

Mikrotik Remote Access via Multiple WAN Links

Filed under: Mikrotik Related — Syed Jahanzaib / Pinochio~:) @ 1:47 PM

how-to-mark-trails-like-a-pro-pin

I wrote about this topic few years back, but forgot where it is now, So adding it again as Note to Self! This solution applies for following particular scenario.


Scenario:

We have 2 wan links configured with policy base routing. As we know that Mikrotik or any device can have only one default route active at a time. So if we will try to access mikrotik via wan2 link it will not work, because when request will arrive on wan2 link, and tries to return to its original requester, it will always route via WAN-1 link dueto default route. At this point remote client will receive packets with a source IP it didn’t initiate traffic with, so it reject that response.

Fair enough !

To sort we need to mark there connections, and make sure every packets should return via same route via which it came IN.

# Mirkotik IP Firewall Mangle Section
/ ip firewall mangle
# Mark traffic coming via WAN-1 link
add chain=input in-interface=WAN1 action=mark-connection new-connection-mark=WAN1_incoming_conn
# Mark traffic coming via WAN-2 link
add chain=input in-interface=WAN2 action=mark-connection new-connection-mark=WAN2_incoming_conn

# Mark traffic routing mark for above marked connection for WAN-1 , so that mikrotik will return traffic via same interface it came in
add chain=output connection-mark=WAN1_incoming_conn action=mark-routing new-routing-mark=to_WAN1
# Mark traffic routing mark for above marked connection for WAN-2, so that mikrotik will return traffic via same interface it came in
add chain=output connection-mark=WAN2_incoming_conn action=mark-routing new-routing-mark=to_WAN2

# Finally Add appropriate routes in ROUTE section
/ ip route
add dst-address=0.0.0.0/0 gateway=1.1.1.2 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=to_WAN2 check-gateway=ping

For other scenario’s, you may want to look into prerouting !

Regard’s
Syed Jahanzaib

 

October 24, 2018

ASCI Fun with Mikrotik Terminal Banner

Filed under: Mikrotik Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 3:46 PM

bat banner

To edit Mikrotik Terminal Welcome Banner, Open Terminal & Issue following command,

/system note edit note

Now Design your graphics / or add texts of your choice, or paste your already copied data in this terminal window.

After Done, Press CTRL+O , & it will save/exit.

Now open Terminal again, and this time you will see your MOTD/Banner smiling 🙂

mikrotik temrinal motd banner

More Info here

September 7, 2018

COA with Radclient workaround for RM 4.1 with Mikrotik 6.4x

Filed under: freeradius, Mikrotik Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 1:12 PM

dealing-with-dynamic-change-2

Scenario:

  • Dmasoftlab Radius Manager 4.1 with multiple services. Some of services have dynamic dynamic bandwidth scheduling for day & night. Example some services have double up mode for day , some for evening, and some for night.
  • Mikrotik 6.42.7 server with hotspot or pppoe authentication services for LAN users

Problem:

DMA Radius Manager 4.1 ‘s API functionality is broken for Mikrotik RouterOS newer versions. The 4.1 code is relying on modifying dynamic queues which had worked on 5.x version (& in some 6.2x series as well e.g: v6.29) . Any circumstances where that was doable were bugs that MikroTik has since fixed. And relying on bugs is generally a bad practice. This can be solved by using CoA instead of modifying dynamic queues which I have used in this post.

It is highly recommended that you must upgrade radius manager to latest 4.2 version which works good with new ROS.


Workaround for RM 4.1:

If for some reasons you want to stick with 4.1 version for whatsoever reason, example 4.2 version have some strict licensing policies so if still wants to use ROS latest series like 6.42.7 (as of writing this post)  , , & if you still wants to avail dynamic bandwidth changes on the fly for particular services , you can schedule following script which will run on hourly basis and will send bandwidth change request to mikrotik according to the service time.



Limitations of the Script:

  • This is a lab testing version of the script. You must modify and tune it for production use. Example the script is doing lots of sql queries, you can minimize it by creating single combined query to fetch all data from the tables, and then read values in next cmd from local file which will be much faster then querying from MySQL.
  • The service must have single time schedule. example from 08:00:00 to 20:00:00 , Multiple times for single service is not supported.
  • The time must consists of single day, it cannot overlap to next day,
  • Script will run as per cron schedule , despite you have selected specific days or not.
  • In lab I have configured it to run every hour , It will query services and its associated users. If the Start time matches , it will send bandwidth change request to the NAS, and if end time matches it will send user original package values to NAS. You can overcome repeating issue by adding additional column in the respective table and update it every time script runs which will check if it have already sent or not.
  • You should disable echoing the outputs, it will save some resources.

the Scheduler!

Either use @hourly in CRONTAB or make separate file under /etc/crond

Create new file name bw in /etc/crond/ with following contents

touch /etc/cron.d/bw
nano /etc/cron.d/bw

& add following line in it,

1 * * * * root /temp/bw.sh

Save & Exit …


the Script!

mkdir /temp
touch /temp/bw.sh
chmod +x /temp/bw.sh
nano /temp/bw.sh

& add following …

WordPress is not letting full code pasting here dueto [ ] limitation. therefore I have copied to my google drive. You can see it here.

https://drive.google.com/drive/folders/1BRIvT6lr9s66nzPP2tRsBV6G-0YA-Zkw


Results:

radius bw poller result.PNG


### ./bw.sh

- Script Start Time - 10:23:39

- INFO: mysql service is accessible. Proceeding further ... OK
- INFO: radius database exist. Proceeding further ... OK
- INFO: Total number of services with Dynamic bandwidth enabled = 11 / No.s ... OK
- INFO: Total number of users with Dynamic bandwidth enabled = 2300 / No.s ...
- INFO: Checking for Dynamic Bandwidth Policies and implement change on the fly for online users , if any ...
- DOWNGRADE ** - XXX1 / 1Mb_day_2mb_night is eligible for DOWNGRADE but NOT online, Ignoring ...
- DOWNGRADE ** - XXX2 / 1Mb_day_2mb_night / 172.16.12.52 / 80702383 is online on NAS 10.0.0.100, and eligible for DOWNGRADE to 1024k/1024k

- DOWNGRADE ** - XXX3 / 1Mb_day_2mb_night is eligible for DOWNGRADE but NOT online,Ignoring...
- DOWNGRADE ** - XXX4 / 1Mb_day_2mb_night / 172.16.6.209 / 80702352 is online on NAS 10.0.0.100, and eligible for DOWNGRADE to 1024k/1024k
- DOWNGRADE ** - XXX5 / 1Mb_day_2mb_night is eligible for DOWNGRADE but NOT online, Ignoring ...
- DOWNGRADE ** - XXX6 / 1Mb_day_2mb_night is eligible for DOWNGRADE but NOT online, Ignoring ...
- DOWNGRADE ** - XXX7 / 1Mb_day_2mb_night is eligible for DOWNGRADE but NOT online, Ignoring ...
^C
- Script Ends Here
- EXITING peacefully ...
- Script Start Time - 10:23:39
- Script End Time - 10:23:40
- Total Number of users packages changed on NAS for UP-GRADE = 0
- Total Number of users packages changed on NAS for DOWN-GRADE = 4

Bandwidth poller script ended @ 10:23:40 ... Powered by SYED.JAHANZAIB

Regard’s
Syed Jahanzaib

March 27, 2018

Separating NATTING from ROUTING in Mikrotik

Filed under: Mikrotik Related — Tags: — Syed Jahanzaib / Pinochio~:) @ 10:57 AM

nattinv and routing brother.jpg

mikrotik natting and routing

Disclaimer:

Every Network is different , so one solution cannot be applied to all. Therefore try to understand logic & create a solution that can match with your network scenario. Do not follow copy paste blindly.

I am NOT certified in anything Mikrotik/Cisco/Linux or Windows. However I have worked with some core networks and I read & research & try stuff all of the time. So I am not speaking/posting about stuff I am formerly trained in, I pretty much go with experience and what I have learned on my own. And , If I don’t know something then I read & learn all about it.

So , please don’t hold me/my-postings to be always 100 percent correct. I make mistakes just like everybody else. However – I do my best, learn from my mistakes and always try to help others

Regard’s
Syed Jahanzaib


Scenario:

We are using Mikrotik CCR as PPPOE NAS. Its a mix match router where large number of users received private ip (via pppoe connection) and other large portion receives IP from routed Public pool as well.

 


Problem:

When we have network outages like light failure in any particular area , in LOG we can see many PPPoE sessions disconnects with ‘peer not responding‘ messages. Exactly at this moments, our NAS CPU usage reaches to almost 100% , which results in router stops passing any kind of traffic. This can continue for a minute or so on.

As showed in the image below …

pppoe high cpu usage

If you are using Masquarade /NAT on the router, that is the problem. When using Masquarade, RouterOS has to do full connection tracking recalculation on EACH interface connect/disconnect. So if you have lots of PPP session connecting/disconnecting, connection tracking will constantly be recalculated which can cause high CPU usage. When interfaces connect/disconnect, in combination with NAT, it gives you high CPU usage.


Solution OR Possible Workaround :

  • If you have lots of PPP users along with some NATTING rules, Stop using Masquarade on same router that have a lot of dynamic interfaces. DO NOT use NAT on any router that have high number of connecting/disconnecting interfaces , like pppoe/vpn. Place an additional router connected with your PPPoE NAS, and route NAT there.
    Example: Add another router & perform all natting on that router by sending marked traffic from private ip series to that nat router. Setup routing between the PPPoE NAS and the NAT router.

Following is an working example.

1# Main CCR as PPPOE NAS

Interface Details:

  • ETHER1-LAN-: 192.168.88.1/24 < User facing interface where pppoe connections establishes
  • PUBLIC-WAN: 101.11.11.254 < WAN Interface for public IP routing
  • 2-NAT-ROUTER: 192.168.100.2/24  < interface connected with another CCR for natting

PPPoE User IP Pool > 172.16.0.1-172.16.0.255
UPSTREAM ISP Core Router Gateway IP >  101.11.11.36

2# Second CCR as NATTING Router

Interface Details:

  • 2-CCR-LAN: 192.168.100.1/24 < interface connected with main CCR [pppoe]
  • NATTING-WAN: 101.11.11.253 < Wan interface for natting users [traffic coming from main CCR for natting]
  • UPSTREAM ISP Core Router Gateway IP >  101.11.11.36

1# Main CCR Configuration for marking traffic

First we will mark traffic for private/public ip and will create routes for them as well.

/ip firewall mangle
add action=mark-routing chain=prerouting comment="Mark routing for Private IP users - zaib" disabled=no new-routing-mark=nat_routing passthrough=yes src-address=172.16.0.1-172.16.0.255
# We really dont need to mark traffic for public ip's because they will simply pass from our default route , but just for the sake of demonstration we are doing it.
add action=mark-routing chain=prerouting comment="Mark routing for PUBLIC IP users - zaib" disabled=no new-routing-mark=public_routing passthrough=yes src-address=1.1.1.1-1.1.1.255

Make sure you dont have any NAT rule in place. [in NAT section]

Now add Routes for marked traffic

/ip route
add comment="Route private ip traffic via second NAT router" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.100.1 routing-mark=nat_routing scope=30 target-scope=10
add comment="Route public ip via this router default Gateway" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=101.11.11.36 routing-mark=public_routing scope=30 target-scope=10
# DEFAULT Gateway for router's own traffic - zaib
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=101.11.11.36 scope=30 target-scope=10

Main CCR configuration part is done. Now moving towards second router where all NATTING will be done.

2# NATTING CCR Configuration for Masquerade

First create Default NAT rule [you may want to add ip series for security purposes.

/ip route
add comment="Default Router for NATTING router " disabled=no distance=1 dst-address=0.0.0.0/0 gateway=101.11.11.36 scope=30 target-scope=10
# Add reverse Route so that NATTING router can see the pppoe user directly
add disabled=no distance=1 dst-address=172.16.0.0/16 gateway=192.168.100.2 scope=30 target-scope=10

Testing !

  • Create TEST user in main CCR pppoe NAS,
  • Assign him private ip series profile,
  • Connect this TEST id from test PC & run TRACEROUTE

As shown in the image below …

ccr pppoe active private.JPG

.

nat-vs-route.JPG

RUN Torch on NATTING Router… as we can see that NATTING router is seeing pppoe users directly dueto reverse route in it.

NATTING CCR torch

.


 

March 21, 2018

FUN with Mikrotik BRIDGE Series# Redirecting Traffic with Mikrotik Bridge – Part#2

Filed under: Mikrotik Related — Tags: , , , — Syed Jahanzaib / Pinochio~:) @ 3:19 PM

link-redirection

  1. FUN with Mikrotik BRIDGE Series#1. Filter PPPoE Requests – Part#1
  2. FUN with Mikrotik BRIDGE Series# Redirecting Traffic with Mikrotik Bridge – Part#2 < You are Here

Disclaimer! This is important!

This post is related to a solution designed specific to cater some local manipulation requirement therefore you may continue to read it as an reference purpose only !

Every Network is different , so one solution cannot be applied to all. Therefore try to understand logic & create your own solution as per your network scenario. Just dont follow copy paste.

Please donot think that I am an expert on this stuff, I am NOT certified in anything including Mikrotik/Cisco/Linux or Windows. However I have worked with some core networks and I read , research & try stuff all of the time. So I am not speaking/posting about stuff I am formerly trained in, I pretty much go with experience and what I have learned on my own. And , If I don’t know something then I read & learn all about it.

So , please don’t hold me/my-postings to be always 100 percent correct. I am human being , I do make mistakes just like everybody else. However – I do my best, learn from my mistakes and always try to help others

Regard's
Syed Jahanzaib~

Scenario & Requirements:

We want to connect Network A & B using Mikrotik Bridge so that we can transparently intercept some traffic for control & redirection purposes. Example we want to make sure that any dns traffic that is traveling from A to B or B to A should be redirected to Mikrotik DNS for manipulation purposes. Also we would like to Block ICMP traffic travelling between both networks.

Solution:

We are using Mikrotik 2011UiAS-2HnD model.

Port-1 is connected with Network A and Port-2 is connected with Network B.

# BRIDGE Configuration

First we will do Bridge configuration & add ports in it,

/interface bridge
add name=bridge1

/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
/interface bridge settings
set use-ip-firewall=yes

As showed in image below …

bridge rules

# DNS Configuration

Now setup Local DNS server

/ip dns
set allow-remote-requests=yes servers=8.8.8.8

# Now we will add static DNS entry for our requirements
/ip dns static
add address=1.2.3.4 name=aacable.wordpress.com

As showed in image below …

bridge dns add static.JPG

# DNS Redirection

Firewall NAT configuration to redirect DNS traffic travelling via BRIDGE interface to Mikrotik local DNS for manipulation purposes

/ip firewall nat
add action=redirect chain=dstnat comment="Redirect DNS Traffic via BRIDGE to local DNS - Zaib" dst-port=53 in-interface=bridge1 protocol=udp to-ports=53

# ICMP Filteration

Firewall Filter configuration to block ICMP protocol

/ip firewall filter
add action=reject chain=forward comment="Block ICMP Rule in BRIDGE - Zaib" in-interface=bridge1 protocol=icmp reject-with=icmp-network-unreachable

Client Testing

Result of testing NSLOOKUP from user PC. [Before vs After]

bridge - dns resolve nslookup result

Result of testing ICMP & PING from user PC.

bridge - icmp block result result


Linux is amazing 🙂 however Mikrotik is handy most of the times 🙂

February 26, 2018

Renew DHCP lease if Gateway not responding

Filed under: Mikrotik Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 8:49 AM

automation

Nothing Fancy , just short notes for script command reference purposes!

Following is a very simple 2 minutes instant cooked noddle type script. It will simply check the gateway status acquired by the dhcp-client using  ARP ping, , if it fails, then it will simply try to renew the ip by release/renew. Its very basic level, but its interesting to see how Mikrotik can really help you in day to day task by facilitated with the Tik level scripting !

Regard’s
Syed Jahanzaib

# Mikrotik Script - Tested with 5.x
# Script to check default gateway acquired by dhcp client on specific interface,
# Lot of room for improvements and modification but following was enough for some particular task,
# You can add BOUND status as well too, but i wanted this particular checking, you can add whatever you like
# Syed Jahanzaib == aacable AT hotmail DOT com - https:// aacable DOT wordpress DOT com
# Feb,2018
# Setting Variables

# Set Interface name which will get DHCP ip , This is the only option you may need to modify
:local INTERFACE "wan1"

# Number of Ping Count, how many times mikrotik should ping the target device
:local PINGCOUNT "5"

# Ping threshold, how many values should set alert, like if 5 out of 5 goes out
:local PINGTS "5"

:local i 0;
#:local i value=0;
:local F 0;
:local date;
:local time;

:log info "Checking default gateway for $INTERFACE interfaces."
:local DHCPGW [ /ip dhcp-client get [/ip dhcp-client find where interface=$INTERFACE ] gateway ]

# IF there is no default gateway on dhcpclient interface or if interface is disabled, then error
:if ([:len $DHCPGW] = 0) do={
:log error "No DEFAULT gateway found on $INTERFACE interface @ $date $time ..."
# Try to renew ip
/ip dhcp-client release $INTERFACE
/ip dhcp-client renew $INTERFACE
# Exit the script without further process ... I found this recently because in mikrotik there is no EXIT 1
error :error
}

# PING host $PINGCOUNT times
:for i from=1 to=$PINGCOUNT do={
:if ([/ping arp-ping=yes interface=$INTERFACE $DHCPGW count=1]=0) do={:set F ($F + 1)}
:delay 1;
};

# If no ping found then give error and do action
:if (($F=$PINGTS)) do={
:log error "PING to $DHCPGW via $INTERFACE is DOWN! @ $date $time "

# Take action if unable to ping gateway
/ip dhcp-client release $INTERFACE
/ip dhcp-client renew $INTERFACE
} else={
:log warning "PING to $DHCPGW via $INTERFACE is UP! @ $date $time "
}
Older Posts »

%d bloggers like this: