Syed Jahanzaib Personal Blog to Share Knowledge !

May 12, 2019

Manipulate CDN Traffic with Mikrotik Mangle & Queue Tree

Filed under: Mikrotik Related — Tags: , , , — Syed Jahanzaib / Pinochio~:) @ 12:49 PM

ncdn_-_cdn.png

Control / Facilitate CDN traffic with

~ Mikrotik Router ~

 

Result

First some DRY theory !

CDNs replicate content in multiple places. There’s a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network. The general idea of a CDN is to deliver content as fast as possible to the user without compromising the user’s experience. Usually, a CDN have global location servers, called Point of Presence. These PoPs store data as cache. When a user requests for a website, the nearest PoP will handle the request using stored cache.

The BIG players such as Google in order to enhance user experience have tried to get as close to the user as possible by direct peering with the regional service providers and provide contents using CDN (Content delivery network) providers. Google is having its own CDN network branded as a service called Google Global Cache (GGC)

Nowadays all the major ISPs have CDN facility , which tremendously helps them to reduce burden on there internet feed. Without CDN, cost of real internet bandwidth will be a heavy burden for any OP. With CDN user will get better video streaming experience.

I know few ISP’s here in Karachi (& one particularly originated from Gulshan Area) which totally relies on CDN (more than 50-60% of there internet data is routing via CDN) , I have used one of them, there real internet speed is pathetic but if you browse YT/FB they works excellent.


Scenario:

Our upstream ISP have CDN server installed in there data center & traffic going to CDN have no limit. But we want to control the traffic as following

1 Mb package Users break up for bandwidth controlling …

  • 1mb internet bandwidth
  • 2mb CDN bandwidth

So if a user is surfing the internet he will get full 1mb internet speed, & if he uses the traffic going to YOUTUBE CDN ,  He will get additional 2 mb.

Using Mikrotik, we can achieve this task by using Firewall Mangle & Queues Tree. Same can be done with Mangle & PCQ base simple queues too. It’s a debatable topic on what to use, & Depends on the selection, mangle marking method would also be changed.

Every network is different so one configuration cannot fit all. Number of users & traffic volume plays vital role in selection of marking / queue type to use.

Choose the marking/queue type wisely to save your Mikrotik CPU from becoming Mr. SPIKY 🙂 YKWIM 😀

Disclaimer: This is just an example for sharing purposes ONLY & yes there are many other methods and tuning techniques you can adopt to make this process much more efficient.


Script !

/ip pool
add name=pppoe_pool ranges=172.16.99.1-172.16.99.255

/ip firewall address-list
add address=172.16.99.0/24 list=1mb
add address=58.27.130.0/24 list=cdn_list
# 58.27.130.0/24 is WATEEN telecom CDN server range. It may change over time or different for each ISP/Network. Pay attention to this part.

# MANGLE Rules orderwise placement is very important. Make sure to mark CDN (or requireD) traffic before other traffic

/ip firewall mangle
# Marking traffic going to CDN (or required sites/ips)
add action=mark-packet chain=forward src-address-list=1mb dst-address-list=cdn_list new-packet-mark=cdn_1mb_up passthrough=no
add action=mark-packet chain=forward src-address-list=cdn_list dst-address-list=1mb new-packet-mark=cdn_1mb_down passthrough=no
# Marking other traffic , normal traffic other then CDN
add action=mark-packet chain=forward src-address-list=1mb new-packet-mark=1mb_up passthrough=no
add action=mark-packet chain=forward dst-address-list=1mb new-packet-mark=1mb_down passthrough=no

# Adding PCQ base queue type for per user distribution

/queue type
add kind=pcq name=2mb-cdn-download pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=2048k pcq-src-address6-mask=64
add kind=pcq name=2mb-cdn-upload pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=2048k pcq-src-address6-mask=64
add kind=pcq name=1mb-upload pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-limit=50KiB pcq-rate=1100k pcq-src-address6-mask=64 pcq-total-limit=2000KiB
add kind=pcq name=1mb-download pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-limit=50KiB pcq-rate=1100k pcq-src-address6-mask=64 pcq-total-limit=2000KiB

# Adding QUEUE  TREE to control bandwidth on per user basis policy using PCQ

/queue tree
# Adding 2mb QUEUE TREE for CDN traffic
add name="CDN - 1mb - upload" packet-mark=cdn_1mb_up parent=global priority=1 queue=2mb-cdn-upload
add name="CDN - 1mb - download" packet-mark=cdn_1mb_down parent=global priority=1 queue=2mb-cdn-download
# Adding 1mb QUEUE TREE for Normal traffic (Other then CDN)
add name=1mb-internet-up packet-mark=1mb_up parent=global queue=1mb-upload
add name=1mb-internet-down packet-mark=1mb_down parent=global queue=1mb-download

 

cdn and net queue tree

 

As shown in above example image, 1mb user is download contents from the internet at full 1 Mb speed (infact I add some extra liberty that is 1100 kb) and on the same side Youtube video (via CDN) is getting 2mb of bandwidth.

With the help of CDN, Real internet bandwidth is saved a lot, and available for other tasks/users,  thus providing relief to the real internet bandwidth pipe.


Personal experience regarding PCQ base queue – Simple Queue vs Tree !

In my personal experience, if you have hundreds of users, then Dynamic simple queues are better as there load spreads over multiple cores.

But for CDN we have to create separate queues for marked packets, therefore we have to use PCQ base queues , either in SIMPLE or TREE. In my experience I have seen that PCQ base queue with Queue Tree puts less load on the CPU.

 


Regard’s
Syed Jahanzaib

durood

March 4, 2016

Lets manipulate ! Part-1 / Traffic base priority via Queue Tree in Mikrotik

Filed under: Mikrotik Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 2:34 PM

 


manipulateBandwidth Management Fast-Track ! No theory, Just coding …

Requirement:

We want to restrict  client with 128 kbps. BUT also want to prioritize it based on traffic type.

Example …

Client-A IP = 101.11.14.1 (zaib-desktop)

Bandwidth Allowed = 128k Total

Priority 1 = ICMP Traffic

Priority 2 = HTTPS Traffic

Priority 3 = HTTP Traffic

Priority 4 = All Other Traffic


Marking traffic from Client-A in MANGLE

First Mark User Traffic in Mangle Section.


/ip firewall mangle

# Mark HTTP 80Traffic Connections/Packets
add action=mark-connection chain=forward comment="Zaib-Desktop - Mark HTTP Port 80" dst-port=80 new-connection-mark=Zaib_Desktop_http_80_Conn protocol=tcp src-address=101.11.14.1
add action=mark-packet chain=forward connection-mark=Zaib_Desktop_http_80_Conn new-packet-mark=Zaib_Desktop_http_80_pkts passthrough=no 

# Mark HTTPS 443 Traffic Connections/Packets
add action=mark-connection chain=forward comment="Zaib-Desktop - Mark HTTPS Port 443" dst-port=443 new-connection-mark=Zaib_Desktop_https_443_Conn protocol=tcp src-address=101.11.14.1
add action=mark-packet chain=forward connection-mark=Zaib_Desktop_https_443_Conn new-packet-mark=Zaib_Desktop_httsp_443_pkts passthrough=no add action=mark-connection chain=forward comment="Zaib Desktop - ICMP" new-connection-mark=Zaib_Desktop_ICMP_Conn protocol=icmp src-address=101.11.14.1

# Mark ICMP TRAFFIC Connections/Packets 
add action=mark-connection chain=forward comment="Zaib Desktop - ICMP" new-connection-mark=Zaib_Desktop_ICMP_Conn protocol=icmp src-address=101.11.14.1
add action=mark-packet chain=forward connection-mark=Zaib_Desktop_ICMP_Conn new-packet-mark=Zaib_Desktop_ICMP_Pkts passthrough=no 

# Mark ALL OTHER Traffic Connections/Packets 
add action=mark-connection chain=forward comment="Zaib Desktop - All Other Traffic" connection-mark=no-mark new-connection-mark=Zaib_Desktop_All_Other_Traffic src-address=101.11.14.1 
add action=mark-packet chain=forward connection-mark=Zaib_Desktop_All_Other_Traffic new-packet-mark=Zaib_Desktop_All_Other_Pkts passthrough=no 

Creating QUEUE TREE to restrict and Prioritize traffic for above marked packets

Now we we will create Parent Queue Tree to restrict 128k then other child queues to prioritize his traffic base on marked packets

# Mark QUEUE TREE
/queue tree 
add limit-at=128k max-limit=128k name="Zaib Dekstop - 128k" parent=global queue=default

# 1st Priority to ICMP Traffic from above 128k Parent Queue 
add name="PRIO 1 - ICMP" packet-mark=Zaib_Desktop_ICMP_Pkts parent="Zaib Dekstop - 128k" queue=default  priority=1

# 2nd Priority to HTTPS 443 Traffic from 128k Parent Queue
add name="PRIO 2 - HTTPS" packet-mark=Zaib_Desktop_httsp_443_pkts parent="Zaib Dekstop - 128k" queue=default priority=2

# 3rd Priority to HTTP Port 80 Traffic from 128k Parent Queue
add name="PRIO 3 - HTTP" packet-mark=Zaib_Desktop_http_80_pkts parent="Zaib Dekstop - 128k" queue=default  priority=3

# 4th Priority to All Other Traffic from 128k Parent Queue
add name="PRIO 4 - All Other Traffic" packet-mark=Zaib_Desktop_All_Other_Pkts parent="Zaib Dekstop - 128k" queue=default priority=8

RESULT # 1

When ICMP have low priority over other protocols

BEFORE ICMP PRIORITY

1- before prio

 


 

RESULT # 2

When ICMP have high priority over others

***   A F T E R   ***

2- after prio


 

Regard’s
Syed Jahanzaib

November 24, 2015

Quick Note: Limit interface total bandwidth by Queue Tree

Filed under: Mikrotik Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 3:12 PM

1

limit.jpg


Requirement:

We want to limit WAN interface upload/download to 300 mb so that it may not cross this limit to avoid any chocking or lets say for our own bandwidth management.

Solution:

We can use Mangle section to mark up/down traffic on particular interface and then using Queue tree we can achieve our required task.

  • Tested with Mikrotik 6.4x.x (It was also working with older versions as well)
# Mark packets for up/down on WAN interface, we are using ether1 for test, change it as per required
/ip firewall mangle
add action=mark-packet chain=prerouting comment="Mark Packets for Upload on WAN interface / jz" in-interface=ether1 new-packet-mark=wan_upload_pkts passthrough=yes
add action=mark-packet chain=postrouting comment="Mark Packets for Download on WAN interface / jz" new-packet-mark=wan_download_pkts out-interface=ether1 passthrough=yes

# Add Parent Queue to define Maximum Limit that WAN can touch
/queue tree
add comment="Limit Total Traffic to 300mb for WAN interface ether1 / zaib" max-limit=300M name=wan-total-traffic parent=global queue=default

# Now add 2 queues to control download/upload , and point it to use above PARENT queue , so that up/down can remain in limit.
add comment="Limit upload Packets marked by mangle / Jz" name=upload packet-mark=wan_upload_pkts parent=wan-total-traffic queue=default
add comment="Limit download Packets marked by mangle / Jz" name=download packet-mark=wan_download_pkts parent=wan-total-traffic queue=default

 

You can use it to distribute specific amount of bandwidth to specific interface, or lets say subnet or number of users. Example if you have 10 mb of bandwidth link and you want to distribute 5 mb to specific number of users. Lot of other queuing trick you can achieve using Marking/Queue combination !

Regard’s
Syed Jahanzaib

%d bloggers like this: