Syed Jahanzaib Personnel Blog to Share Knowledge !

July 21, 2011

MIKROTIK :Howto Redirect HTTP traffic to SQUID with Original Source Client IP


Howto connect SQUID box with Mikrotik and Log user’s original source ip in squid proxy log.

Consider the following Scenario.

Mikrotik + SQUID + Client Source IP Loggin

In this scenario Mikrotik is acting as PPPoE Server. All internet traffic is passed from Mikrotik, except the HTTP PORT 80 data, which is marked and route to SQUID proxy server for caching facility and some other filtering task. We want to preserve source client IP address in squid logs, so that user web activity can be tracked via squid access log, sometimes it can be require by law enforcing agencies or can be required by administration / reporting / management purposes. So how to do it ?

I assume you already have working SQUID and Mikrotik (pppoe server with pppoe user ip pool 172.16.0.0/16 pool in place. however you can change it according to your scenario)

We will divide this article in two sections

1# MIKROTIK RouterOS Configuration
2# SQUID Proxy Server Configuration


1# MIKROTIK  RouterOS CONFIGURATION !

In this example, Mikrotik have 3 LAN interfaces,  Details are as following . . .

[admin@MikroTik] > ip address print
#   ADDRESS            NETWORK         INTERFACE
 0   192.168.0.1/24     192.168.0.0     ether1       >> LAN INTERFACE
 1   192.168.2.1/24     192.168.2.0     ether2       >> PROXY INTERFACE
 2   192.168.1.2/24     192.168.1.0     ether3       >> WAN - DSL INTERFACE

Now we will start with the Mikrotik configuration:

# Add ip addresses for interface cards,

/ip address
add address=192.168.0.1/24 disabled=no interface=ether1 network=192.168.0.0
add address=192.168.2.1/24 disabled=no interface=ether2 network=192.168.2.0
add address=192.168.1.2/24 disabled=no interface=ether3 network=192.168.1.0

# Set DNS for Host Resolving

/ip dns set allow-remote-requests=yes cache-max-ttl=1w cache-size=8000KiB max-udp-packet-size=512 servers=192.168.1.1

# Mark All HTTP Port 80 Traffic, so that we can use these Marked Packets in Route section.

/ip firewall nat
add action=accept chain=srcnat disabled=no dst-port=80 protocol=tcp

/ip firewall mangle
add action=mark-routing chain=prerouting disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp

# Masquerade all traffic (Except http] on ether3 Only, which is connected with DSL Router. This is important to masquerade traffic on WAN Interface only, otherwise http packets will also be masqueraded with mikrotik ip.

add action=masquerade chain=srcnat disabled=no out-interface=ether3

# Define Route for HTTP Marked packets, and set default rule for all other traffic, This is called policy base or pre traffic base routing

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.2.2 routing-mark=http scope=30 target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.1 scope=30 target-scope=10

(Where 192.168.2.2 is the Squid Proxy Server LAN IP , and 192.168.1.1 is our DSL Router IP)

That’s all for MIKROTIK, Now Mikrotik will Redirect HTTP Traffic to Squid Proxy via interface ether2. and all rest of traffic will be masqueraded/nat to WAN (ether3) which is connected with DSL.

Now moving on to SQUID section !

2# SQUID Proxy Server CONFGIURATION !

I assume you already have working squid in transparent mode [its necessary you set this via iptables and some directives in squid.conf

For example: (squid.conf)

#==============================
#Transparent Mode & Example ACL
#==============================
http_port 8080 transparent
acl mylocalnet src 0.0.0.0/0.0.0.0
http_access allow mylocalnet

and IPTABLES example:

# DNAT port 80 request comming from LAN systems to squid 8080 aka transparent proxy .zaib.
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.2.2:8080

In this example, Squid proxy server have 2 Interface cards

eth0: LAN (connected with Mikrotik's PROXY INTERFACE ether2) = 192.168.2.2

eth1: WAN (connected with DSL Router) = 192.168.1.3 /  Default GW = 192.168.1.1  / DNS = 192.168.1.1 , 8.8.8.8

It is necessary that SQUID can directly communicate with the users by setting ROUTE to communicate user subnet via mikrotik, otherwise it won't be able to communicate with the user , Issue the following command ,

route add -net 172.16.0.0 netmask 255.255.0.0 gw 192.168.2.1 dev eth0

172.16.0.0/16 is pppoe user IP pool, After successfully execution, Squid will be able to see the Users ip. Also add the above route command in /etc/rc.local (u must add any command in rc.local before 'exit 0'  if using ubuntu)

Above command Explanation:  172.16.0.0 are pppoe users ip pool and 192.168.2.1 is Mikrotik ether2 which is directly connected to Squid via crossover cable , thus we are telling Squid to look after for users 172.16.0.0 via gateway 192.168.2.1 which is mikrotik, if we don't use this, squid and users wont be able to communicate with each other)

That's it. Now when user will try to use internet, his HTTP traffic will be marked by Mikrotik and then all HTTP marked traffic will be ROUTED to SQUID proxy [192.168.2.2]  with original client ip, instead of Mikrotik ip.

Some useful links for fine tuned working squid.conf and Linux internet sharing script.

http://aacable.wordpress.com/2011/06/01/linux-simple-internet-sharing-script/

http://aacable.wordpress.com/2011/06/01/working-squid-conf-example-fil/

Cheers,
:)

ALLAH  HAFIZ ,
Syed Jahanzaib !

About these ads

122 Comments »

  1. aslaam alikuaam keyse hai sir mikrotik main load balasing karlita hoo par net connct nahi hota aur yeh batao ke connct nahi kariyo nahi kar rahaa aur modem main koi setting nah karsati

    Comment by Aman Manifarooqi — August 1, 2011 @ 4:18 PM

  2. Salam Bhai aap ka tutorial buhat acha hai plzz app mujhy bata sakty hai k squid kis tara configure karna plzzzzzzzzzzz.

    Comment by usman — August 7, 2011 @ 7:03 PM

  3. thanks bhai jaan lkin mujhy 1 bat ki samjh nai a rahi kindly mujhy bata dy mai ny ubuntu11.04 desktop ki hai ur us mai 2 lan card lagay hai lkin 2 lan card akhty enable nai hoty jab 1 ko enable karta ho to dosara disable ho jata hai plzz bhai bata dy yeh kia masla hai?

    Comment by usmans — August 10, 2011 @ 12:22 AM

    • Must be some configuration mistake. or maybe IRQ conflicting issue. try replacing the Lan card to different brand.

      Comment by Pinochio / zaib — August 10, 2011 @ 10:53 AM

  4. ok mai lan card change kar k chk karo ga aur bhai agar ho saky to thunder cache ka tutorial b upload kary……thanks

    Comment by usman — August 10, 2011 @ 4:44 PM

  5. Dear admin,

    Can u explain to me How mikrotik ether 3 interface (192.168.1.2) and squid ether 1 interface (eth 1 = 192.168.1.2) can have the same ip address? please i m really confused

    Comment by Ashraf — August 13, 2011 @ 6:40 PM

  6. Assalamu Walaikum,
    I have implemented the same setup using this tutorial and I very much liked it. But I have a small problem, it seems that after implementing this setup, my clients cannot tracert to any IP. The results are like this:-

    C:\Users\Shishir>tracert -d http://www.yahoo.com

    Tracing route to eu-fp3.wa1.b.yahoo.com [87.248.112.181]
    over a maximum of 30 hops:

    1 * * * Request timed out.
    2 * * * Request timed out.
    3 * * * Request timed out.
    4 * * * Request timed out.
    5 * * * Request timed out.
    6 * * * Request timed out.
    7 * * * Request timed out.
    8 * * * Request timed out.
    9 * * * Request timed out.
    10 * * * Request timed out.
    11 * * * Request timed out.
    12 * * * Request timed out.
    13 * * * Request timed out.
    14 * 280 ms 286 ms 87.248.112.181

    Trace complete.

    On the other hand, normal ping just works fine.

    C:\Users\Shishir>ping http://www.yahoo.com

    Pinging eu-fp3.wa1.b.yahoo.com [87.248.112.181] with 32 bytes of data:
    Reply from 87.248.112.181: bytes=32 time=263ms TTL=49
    Reply from 87.248.112.181: bytes=32 time=280ms TTL=49
    Reply from 87.248.112.181: bytes=32 time=288ms TTL=49
    Reply from 87.248.112.181: bytes=32 time=252ms TTL=49

    Ping statistics for 87.248.112.181:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 252ms, Maximum = 288ms, Average = 270ms

    I have done everything just as said in this tutorial. Mark routing for HTTP (80) and then routing them to the Linux server. All everything else is routed to the default route. Everything is fine, we can browse using the proxy. But only this problem, which I can’t find any solution. I don’t have any other firewall in place.

    Regards,
    Saiful Alam

    Comment by Saiful Alam (@saifulmr) — September 9, 2011 @ 4:26 PM

  7. Guessing is a waste of everybody’s time. Please provide Mikrotik details. There must be some mistake in rules / firewall. Email me your Mikrotik Configuration.

    Comment by Pinochio / zaib — September 10, 2011 @ 10:51 AM

  8. Asslam o Alikum
    Sir Thunder cache bhi sekha de Video cache ke liye thanks

    Comment by Arshad — November 14, 2011 @ 2:13 AM

    • I never worked on Thundercache, Once I installed it but didn’t liked it as its not really an standard solution.
      Use SQUID instead which is open source and works really well if you configure it properly.

      Comment by Pinochio~:) — November 14, 2011 @ 10:34 AM

  9. Dear Syed

    I’ve properly configured squid and testet in the same lan as squid server with google chrome and it works great

    But with mikrotik it doesnt work , i thing it adds everything from mikrotik users to cache but it cannot take anything from cache to the users cuz when i see the squid logs i see everywhere
    TCP_MISS
    and
    TCP_HIT_REFRESH
    TCP_NEGATIVE HIT
    TCP_DENIED

    but just TCP_HIT i can’t see cuz mikrotik cant take anything from squid ?
    if u could just check it with teamviewer i would be more than happy but if not ill try (even im trying it since 2 weeks :S with a couple success )

    MY CONFIGURATION IS SO :

    INTERNET

    switch

    MIKROTIK (CONNECTED TO SWITCH) IP 80.80.171.28 – SQUID (CONNECT TO SWITCH) 80.80.171.29
    THEY ARENT CONNECTED TOGETHER WITH ANY LAN CABLE ) JUST IN THE SAME SWITCH

    WHAT I’VE DONE :
    [nori@Kijeva 1] > ip firewall nat print
    Flags: X – disabled, I – invalid, D – dynamic
    0 ;;; chain=srcnat action=masquerade

    1 ;;; SQUID SERVER
    chain=dstnat action=dst-nat to-addresses=80.80.171.29 to-ports=8080
    protocol=tcp dst-port=80

    /ip firewall mangle> print
    Flags: X – disabled, I – invalid, D – dynamic
    0 ;;; Mark Cache Hit Packets / aacable@hotmail.com
    chain=prerouting action=mark-packet new-packet-mark=proxy-hit
    passthrough=no dscp=12

    [nori@Kijeva 1] /queue tree> print
    Flags: X – disabled, I – invalid
    0 name=”pmark” parent=global-out packet-mark=proxy-hit limit-at=0 queue=default
    priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s

    Comment by Nori — January 7, 2012 @ 10:07 PM

  10. DONE :) thx dead friend :) ur ‘re amazing good and the best IT/Administrator that i’ve ever seen :)

    until now i’ve been trying and reading and reading to build a squid server and to connect with mikrotik , to bypass the limit and others and now everythings working fine thx you…

    i wish u all the best and would like to hear from you soon :)

    Comment by Nori Gashi — January 8, 2012 @ 8:17 AM

  11. Thank you very much (Bark allah veek)
    Your steps guide me to make it work first time ever.
    I need your help in one thing
    I am using mikrotik with hotspot .What i have to do to make it work with it with best speed?and if i use rate limit in user profile ,how i can make the cache comes in full speed?
    Thank you again
    Mohamed

    Comment by Mohamed Fahed — January 12, 2012 @ 9:22 PM

  12. I have followed your tutorial. but one thing I had to change in settings for mangle.
    I have wlan0 PPPOE and Eth1. I had to put exclamation mark in Src. address for squid IP in mangle rule you provided.

    Thank you for this tutorial.
    Badr

    Comment by Badr — January 19, 2012 @ 11:46 PM

  13. hello, can i use dnsmasq with bridge firewall configure. my email id is raaziv@gmail.com

    Comment by raziv ferdous — January 27, 2012 @ 1:51 AM

  14. for less than 200 users,
    like to go with mikrotik based solution.
    have some ideas :
    1.mikrotik router board(with hotspot) + squid server – prefer me good mikrotik model no.
    (or)
    2. mikrotik x86 routeros with cache enabled.

    Im planned to run my network with one pc.
    in area lot of power issues diffcult to run large step.
    please prefer best optimum setup,
    looking for your reply.
    thanks in advance

    Comment by tamilmaran — January 28, 2012 @ 8:58 AM

    • RB450G would be enough for under 200 users.

      You can setup your hotspot and authentication on any mikrotik routerboard, for example RB450G would be enough. and use PC as a squid for caching.

      ROUTERBOARD doesn’t use much power, and restore quickly if any power failure occurs.

      OR if you just wanna use basic level of caching , you can setup all things in one Mikrotik ROUTERBOARD.
      However there are certain benefits of using SQUID, you can highly customize it, cache dynamic contents like youtube and many other features that mikrotik doesn’t even comes near, SQUID is very good if configured properly.

      RB1100 AHx2 is also a very good choice, It can fulfill all your current and future requirements. Its designed to accommodate heavy usage.

      HTH,
      Regard’s

      Comment by Syed Jahanzaib / Pinochio~:) — January 28, 2012 @ 11:13 AM

  15. Can I see how do you shape client BW? Because after I try to implement your tutorial, I cannot limit the bw that flow from user who browse through the squid proxy port.

    Can you show how to mark user who browse from squid proxy?

    Thanks before..

    Comment by Nanda Prima Setiawan — February 3, 2012 @ 6:34 PM

    • Normally I always use RADIUS server along with Mikrotik.
      So user Queue automatically created,
      Even without radius if you create normal queue, User bandwidth will be restricted for all sort of traffic wether its torrent or traffic coming from SQUID

      Comment by Syed Jahanzaib / Pinochio~:) — February 3, 2012 @ 6:51 PM

  16. Thank you for replying sir..

    Actually I just need to log my client traffic trough squid. But I don’t know why the traffic went up after I redirect it to squid. I also can see the clients original ip from the squid. Do i also need to limit the squid IP, or just limit the user that access the web through the squid?
    My network sheme is
    Eth1 : WAN IP
    Eth2 : user IP
    Eth3 : squid IP

    My squid proxy connect to internet via mikrotik Eth1. My user connect to internet via squid. What do i have to do so i can shape the bw from my user ?

    Thank you for your time sir..

    Comment by Nanda Prima Setiawan — February 3, 2012 @ 7:28 PM

  17. Yes sir.. My office just bought mikrotik celoica.
    Eth1: Wan
    Eth9: Squid Proxy
    Eth2-8: User – clients with diffrent location.

    My office only use these ports 21,80,443 and 943.
    if the traffic is redirected to 8181, how I limit the traffic from user?

    Comment by Nanda Prima Setiawan — February 5, 2012 @ 2:01 AM

    • As per this tutorial, mikrotik do redirect http traffic to squid server, but still it goes through he mikrotik, so any restriction and bandwidth policy will be applied on all sort of traffic that goes through Mikrotik. There must be some configuration mistakes in your Queuing section. Check your addressing /routing and queuing section

      Comment by Syed Jahanzaib / Pinochio~:) — February 5, 2012 @ 10:32 AM

  18. Thank You Sir.. it works. Yesterday I did’nt input the destination port. And I change the chain. Before I use forward, now i use prerouting.

    It turns out, that it’s me who was’nt patient enough reading your tutorial..

    Thanks a lot sir..

    May Allah pay your Good Deed..

    Comment by Nanda Prima Setiawan — February 6, 2012 @ 11:47 AM

  19. dear Jahanzaib bhai. I want to use web proxy with with pppoe server and pcc. but i am unable to do so. PPPOE and pcc workin fine but web proxy is not working with pppoe server and pcc. how can i use web proxy with it. pl help

    Comment by khalid iqbal — March 7, 2012 @ 12:46 AM

  20. Dear Jahanzaib bhai thanks for your early response. I have read the forum but unable to understand so much. I got your point that i should have squid for caching. but how can i use squid with this as i already have pcc and pppoe working in a same server. can u please send me script for this.

    Comment by khalid iqbal from Jhang — March 9, 2012 @ 10:54 AM

  21. gud after noon sir i am sanober and i am trying to configure squid. my network setup is like this wan link is on mt 450g ether1 and ether4 is for squid (with single lan card ) and ether5 is for local users.
    wan ip is 10.210.2.5/24 squid ip is 192.168.5.2/24 and ip on mt ether4 is 192.168.5.1 and for lan that is on ether5 (10.0.0.0/8) i have configures hotspot on ether5
    i have configured dstnat
    /ip firewall nat
    add chain=dstnat action=dst-nat to-addresses=192.168.5.2 to-ports=3128 protocol=tcp dst-port=80
    add chain=dstnat action=accept protocol=tcp dst-port=80

    magnle rules
    /ip firewall nat
    add action=accept chain=srcnat disabled=no dst-port=80 protocol=tcp

    /ip firewall mangle
    add action=mark-routing chain=prerouting disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp

    and added route with routing marks
    0.0.0.0/0 10.210.2.1
    0.0.0.0/0 192.168.5.2 with routing marks

    but still that is not working
    can u help me in this topic

    thanks and regards

    Comment by ionlysanober — March 14, 2012 @ 2:24 PM

  22. [...] [...]

    Pingback by Anonymous — May 7, 2012 @ 12:05 AM

  23. hello

    Your manual work for me but I have a problem with my web server because the 80 traffic is directed by the Squid does not rule me out through the Mikrotik, can you help me with this part.

    Note: Other ports if I work like 21, 443 etc by the Mikrotik

    Luis Quispe

    Comment by luis18quispe — July 12, 2012 @ 4:02 AM

  24. plz help 2x wan complete
    setting

    Comment by abdul islam — July 26, 2012 @ 11:33 PM

  25. [...] document.write('[Log in to get rid of this advertisement]'); Hello, I have followed this howto http://aacable.wordpress.com/2011/07…rce-client-ip/ and it has worke perfectly, but when I try with pptp tunnels it does not work. I can ping to google [...]

    Pingback by Anonymous — August 1, 2012 @ 6:34 AM

  26. Hi,

    This worked great with my 2 PPPoE connections setup. Even though it doesn’t merge connections but it really eases the load on one link by adding another.

    I set it up for a small organization of 15 Users and was trying to implement a squid with the setup that it redirects traffic to squid on the same LAN switch Users connected to, but no luck.

    Do you have any advice how I can make mikrotik redirects to squid then squid back to mikrotik with the load balancing not effected. I would like to to make squid in transparent mode.

    I would like also to hint that squid setup works when no load balancing is used and only with one PPPoE Con.

    Here is my network topology:

    PPPoE 2 Internet User1 .
    | 192.168.0.0/24 | .
    Internet Wlan1- MT Router Load Balancing-Ether1 Switch ———- User 2 .
    | | .
    PPPoE 1 Internet 192.168.0.5 .
    Squid in .
    Transparent Mode * .

    * I do load balancing on two connections with the same ISP over two 12 Mbits PPPoE connected by wlan1 and I only have 1 ethernet port on my SXT router so I have to put squid on ether1 too.

    ** With out transparent mode where I put proxy information in User’s browser works just fine but my goal is transparent mode because I am thinking on implementing the same setup with a larger organization where I don’t have to put proxy information.

    I tried these rules on router:

    /ip firewall nat
    add action=accept chain=srcnat disabled=no dst-port=80 protocol=tcp * even disabling or enabling this or changing the order of it had no luck

    /ip firewall mangle
    add action=mark-routing chain=prerouting disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp

    /ip route
    add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.5 routing-mark=http scope=30 target-scope=10

    Rules on the proxy server are:
    # DNAT port 80 request comming from LAN systems to squid 8080 transparent proxy.

    iptables -t nat -A PREROUTING -s 192.168.0.0/24 -p tcp –dport 80 -j DNAT –to 192.168.0.5:8080
    iptables -A INPUT -i eth0 -j ACCEPT
    iptables -A OUTPUT -o eth0 -j ACCEPT
    iptables -A INPUT -j LOG
    iptables -A INPUT -j DROP

    Squid.conf:
    #==============================
    #Transparent Mode
    #==============================
    http_port 8080 transparent
    acl mylocalnet src 0.0.0.0/0.0.0.0
    http_access allow mylocalnet

    Thank you so much,
    Badr

    Comment by Badr — August 18, 2012 @ 7:43 AM

    • squid should be on different subnet. then it will work fine.
      try dst-nat method.

      Comment by Syed Jahanzaib / Pinochio~:) — August 18, 2012 @ 10:06 PM

      • I have been trying with dst-nat method. I even setup 2 network addresses on ether1 with 192.168.0.0/24 dhcp for clients and 192.0.0.0/24 for squid static ip address but had no luck.
        I believe the setup works but I am doing something wrong with dst-nat src-nat rules.

        Any advice?

        Regards,

        Comment by Badr — August 19, 2012 @ 2:37 PM

  27. Sorry Here is my net topology

    Comment by Badr — August 18, 2012 @ 7:47 AM

  28. http://i47.tinypic.com/b3n1o8.png

    Comment by Badr — August 18, 2012 @ 7:48 AM

  29. Hi Sayed,

    I just want to point out that I finally made squid work transparently with one NIC and load balancing not effected and also I would like to share it with this wonderful blog site.

    Scenario:

    ISP Wlan1 Mikrotik Router 2 PPPoE Load balancing internet ether1 Local network 192.168.0.0/24 Switch Users + Squid Proxy

    Wlan1 WDS Link connected to ISP and 2 12 Mbits PPPoE connection established.

    Ether1 DHCP 192.168.0.0/24

    Src NAT masquerading on both PPPoE connections. Mangle rules are set to load balance over the two connections. and finally two mangle rules to mark http traffic for squid.

    Routing is set for load balancing and 2 gateways 192.168.0.5 (squid ip) one for http1 traffic and second for http2 routing marks.

    Squid ip: 192.168.0.5:8080

    Rest of Users share same subnet with squid 192.168.0.1/24 :-)

    PPPoE 1 and 2 : 10.10.10.1

    wlan 1: doesn’t matter.

    Squid options and iptables:

    squid.conf:

    http_port 192.168.0.5:8080

    iptables: Please pay attention** it took me hours to figure it out and it was the only obstacle:

    iptables -t nat -A PREROUTING -s 192.168.0.1/24 -p tcp –dport 80 -j DNAT –to 192.168.0.5:8080
    iptables -t nat -A PREROUTING -s 192.168.0.1 -p tcp –dport 80 -j REDIRECT –to-port 8080

    Look how second rule differs from first rule in which it points to source gateway only.

    Mikrotik RouterOS configuration:

    [iqmaster@iqmaster] > ip address export
    # aug/20/2012 21:20:39 by RouterOS 5.19
    # software id = 7MUK-L4BN
    #
    /ip address
    add address=192.168.0.1/24 disabled=no interface=ether1 network=192.168.0.0
    [iqmaster@iqmaster] > ip firewall nat export
    # aug/20/2012 21:20:45 by RouterOS 5.19
    # software id = 7MUK-L4BN
    #

    /ip firewall nat
    add action=accept chain=srcnat comment=”SRCNAT Option 1″ disabled=no dst-port=80 out-interface=pppoe-out1 protocol=tcp src-address=!192.168.0.5
    add action=masquerade chain=srcnat disabled=no out-interface=pppoe-out1
    add action=accept chain=srcnat comment=”SRCNAT Option 2″ disabled=no dst-port=80 out-interface=pppoe-out2 protocol=tcp src-address=!192.168.0.5
    add action=masquerade chain=srcnat disabled=no out-interface=pppoe-out2

    [iqmaster@iqmaster] > ip firewall mangle export
    # aug/20/2012 21:20:50 by RouterOS 5.19
    # software id = 7MUK-L4BN
    #
    /ip firewall mangle
    add action=mark-connection chain=input comment=”Mark new inbound connection wan1″ connection-state=new disabled=no in-interface=pppoe-out1 new-connection-mark=adsl1 passthrough=yes
    add action=mark-connection chain=input comment=”Mark new inbound connection wan2″ connection-state=new disabled=no in-interface=pppoe-out2 new-connection-mark=adsl2 passthrough=yes
    add action=mark-connection chain=prerouting comment=”Mark established inbound connection wan1″ connection-state=established disabled=no in-interface=pppoe-out1 new-connection-mark=adsl1 passthrough=yes
    add action=mark-connection chain=prerouting comment=”Mark established inbound connection wan2″ connection-state=established disabled=no in-interface=pppoe-out2 new-connection-mark=adsl2 passthrough=yes
    add action=mark-connection chain=prerouting comment=”Mark related inbound connection wan1″ connection-state=related disabled=no in-interface=pppoe-out1 new-connection-mark=adsl1 passthrough=yes
    add action=mark-connection chain=prerouting comment=”Mark related inbound connection wan2″ connection-state=related disabled=no in-interface=pppoe-out2 new-connection-mark=adsl2 passthrough=yes
    add action=mark-routing chain=output comment=”Mark new inbound route wan1″ connection-mark=adsl1 disabled=no new-routing-mark=static-wan1 passthrough=no
    add action=mark-routing chain=output comment=”Mark new inbound route wan2″ connection-mark=adsl2 disabled=no new-routing-mark=static-wan2 passthrough=no
    add action=mark-connection chain=prerouting comment=”Mark traffic that isn’t local with PCC mark rand (2 possibilities) – option 1″ connection-state=new disabled=no dst-address-type=!local in-interface=\
    ether1 new-connection-mark=adsl1_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
    add action=mark-connection chain=prerouting comment=”Mark traffic that isn’t local with PCC mark rand (2 possibilities) – option 2″ connection-state=new disabled=no dst-address-type=!local in-interface=\
    ether1 new-connection-mark=adsl2_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
    add action=mark-connection chain=prerouting comment=”Mark established traffic that isn’t local with PCC mark rand (2 possibilities) – option 1″ connection-state=established disabled=no dst-address-type=\
    !local in-interface=ether1 new-connection-mark=adsl1_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
    add action=mark-connection chain=prerouting comment=”Mark established traffic that isn’t local with PCC mark rand (2 possibilities) – option 2″ connection-state=established disabled=no dst-address-type=\
    !local in-interface=ether1 new-connection-mark=adsl2_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
    add action=mark-connection chain=prerouting comment=”Mark related traffic that isn’t local with PCC mark rand (2 possibilities) – option 1″ connection-state=related disabled=no dst-address-type=!local \
    in-interface=ether1 new-connection-mark=adsl1_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
    add action=mark-connection chain=prerouting comment=”Mark related traffic that isn’t local with PCC mark rand (2 possibilities) – option 2″ connection-state=related disabled=no dst-address-type=!local \
    in-interface=ether1 new-connection-mark=adsl2_pcc_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
    add action=mark-routing chain=prerouting comment=”Mark routing for PCC mark – option 1″ connection-mark=adsl1_pcc_conn disabled=no new-routing-mark=adsl1 passthrough=yes
    add action=mark-routing chain=prerouting comment=”Mark routing for PCC mark – option 2″ connection-mark=adsl2_pcc_conn disabled=no new-routing-mark=adsl2 passthrough=yes
    add action=add-src-to-address-list address-list=Worm-Infected-p445 address-list-timeout=1h chain=prerouting connection-state=new disabled=no dst-port=445 limit=5,10 protocol=tcp
    add action=mark-routing chain=prerouting comment=”Mark traffic to Squid Option 1″ disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp src-address=!192.168.0.5
    add action=mark-routing chain=prerouting comment=”Mark traffic to Squid Option 2″ disabled=no dst-port=80 new-routing-mark=http2 passthrough=yes protocol=tcp src-address=!192.168.0.5
    add action=mark-packet chain=prerouting comment=”Mark Cache Hit Packets” disabled=no dscp=12 new-packet-mark=proxy-hit passthrough=no

    [iqmaster@iqmaster] > ip route export
    # aug/20/2012 21:20:55 by RouterOS 5.19
    # software id = 7MUK-L4BN
    #
    /ip route
    add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.5 routing-mark=http scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=static-wan1 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-mark=static-wan2 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=adsl1 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-mark=adsl1 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-mark=adsl2 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=adsl2 scope=30 target-scope=10
    add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.5 routing-mark=http2 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out2 scope=30 target-scope=10
    add check-gateway=arp disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-out1 scope=30 target-scope=10

    Comment by Badr — August 20, 2012 @ 11:41 PM

    • Thanks for your sharing.

      Comment by Syed Jahanzaib / Pinochio~:) — August 22, 2012 @ 9:33 AM

      • Hi Syed,

        I am facing some strange problem with my setup. an unusual delay in response. it takes like 5 seconds to load or respond to a webpage request, at the same time, when I configure client’s browser to point to squid IP 192.168.0.5 and port 8080 the delay goes away and internet is very responsive.
        I don’t know what is causing the problem. I am sure it is not cache performance or Memory issue since squid is responsive when the browser is configured to use it.

        Do you have any ideas?

        Comment by badr — August 25, 2012 @ 4:57 PM

      • Possibly it could be a DNS related issue. Is your mikrotik acting as DNS Server? check its configuration

        Comment by Syed Jahanzaib / Pinochio~:) — August 26, 2012 @ 10:03 AM

      • I checked for possible dns issue. I disabled mikrotik from acting as dns and tried setting up dnsmasq but still no luck. Why when I point client to squid ip without changing any configuration internet becomes responsive.
        Do you think it is some routing or iptables issue basing on the setup I posted. or may be MTU??

        I tried capturing traffic with tcpdump and examine them with wireshark and there seems nothing wrong.

        It is really intriguing me…

        Comment by Badr — August 27, 2012 @ 3:59 AM

    • What’s the purpose of adding same rule twice?

      /ip firewall mangle
      add action=mark-routing chain=prerouting comment=”Mark traffic to Squid Option 1″ disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp src-address=!192.168.0.5
      add action=mark-routing chain=prerouting comment=”Mark traffic to Squid Option 2″ disabled=no dst-port=80 new-routing-mark=http2 passthrough=yes protocol=tcp src-address=!192.168.0.5

      /ip route
      add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.5 routing-mark=http scope=30 target-scope=10
      add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.5 routing-mark=http2 scope=30 target-scope=10
      ————————————————————————————————————————–

      And you also did not add the post routing rule to bypass the limit for cache hit objects, why is that?

      Comment by Umer Sarwar — October 18, 2013 @ 6:31 PM

      • From where you have got those duplicate rules? its not on my blog.

        Comment by Syed Jahanzaib / Pinochio~:) — October 19, 2013 @ 2:22 PM

      • of course it’s not on your blog. It’s in the script of the “Badr”. The person who posted the whole script above my comment.

        Comment by Umer Sarwar — October 19, 2013 @ 4:38 PM

  30. sir squid proxy direct kese hogi not any browser and idm setting and is ky sath may ek smothwall server lagna chata ho configure kese karo phely sqid phir smoothwall se hota howa afay client ky pas net jai

    Comment by syed salman — August 25, 2012 @ 6:16 AM

  31. sir squid proxy direct kese hogi not any browser proxy setting and idm proxy setting and is ky sath may ek smothwall server lagna chata ho configure kese karo phely sqid phir smoothwall se hota howa agay client ky pas net jai thnx

    Comment by syed salman — August 25, 2012 @ 6:17 AM

  32. AWESOME

    Comment by HRDHKR — August 30, 2012 @ 1:50 AM

  33. salam
    how can limit the cache squid to evry user “user manager+hotspot” like user1 give internet bandwith: 256K/256K and cache limit at: 1M/1M
    thanks

    Comment by khaled — September 5, 2012 @ 8:54 PM

    • why you want to limit the cached contents, they are not suppose to be limited by the package profile.

      Comment by Syed Jahanzaib / Pinochio~:) — September 6, 2012 @ 9:25 AM

      • sir, my isp use hotspot +Radius manager and give me the cache on bandwith 2 MB

        Comment by khaled — September 7, 2012 @ 2:39 AM

  34. salam. Jahanzaib bhai i have followed ur tutorial 100 of times but i failed. Mikrotik showing me that Nat rule is accepting port 80 data and mangle rule also showing me traffic receiving . and also nat rule for WAN is working. but i cant browse http websites. Users can ping squid machine and from squid machiine i can also ping user’s ip. What could be the problem. plzzzzzzzz solve my this problem. i will be very very very thankful to you .

    Comment by Farhan — September 9, 2012 @ 2:36 PM

    • W/Salam.
      The only thing that comes to my mind is that your squid probably not configured in TRANSPARENT mode.
      Check it in squid.conf for ‘transparent’ directive. Also the most important is that on squid, you must redirect port 80 request to port 8080. this would be done using iptables rules. you can use my fw.sh published on blog for assistance.
      I am confident this is the only problem you might be facing for this issue.

      If you use dst-nat method, you won’t be needing iptables because, you can define in dst-nat rule to redirect port 80 request to port 8080, this way you can at least start the journey.

      Comment by Syed Jahanzaib / Pinochio~:) — September 9, 2012 @ 9:28 PM

  35. sir
    squid mikrotik me bi install ho ja ta hai?

    Comment by Qasim Electronics — October 6, 2012 @ 1:08 AM

  36. [...] Redirect HTTP traffic to SQUID with Original Source Client IP by Qasim ElectronicsMikrotik4027 Go to Source | Източник Author: Qasim [...]

    Pingback by Comment on MIKROTIK :Howto Redirect HTTP traffic to SQUID with Original Source Client IP by Qasim Electronics - itcenter-bg.com | itcenter-bg.com — October 6, 2012 @ 6:00 AM

  37. how to redirect traffic to external squid cache with pcc on mikrotik , my pcc is working fine and matching well i have 2 wans and one lan .

    sheraz

    Comment by shiraz — October 29, 2012 @ 2:31 PM

  38. My setup contains 2 wans and 1 local interface for users, pppoe+hotspot are running on same mikrotik ,know my problem is that i want to install squid cache with mikoritk.
    wan1 192.168.6.254
    wan2 192.168.8.254
    local 169.254.241.1
    wan3 reserved for cache.
    i am running pcc on mikrotik and want to install squid with where i use one lan for mikrotik to cache any thinga, kindly help me . for configuration.

    Comment by shiraz — October 31, 2012 @ 8:46 PM

  39. Hi

    Your manual worked great but I have one small problem with my web server because the port80 traffic that needs to go to my website hosted internal(needs to be accessed from outside) is directed to the Squid as well, how can i bypass the redirect rule to go to my website only

    Comment by Jacques — November 28, 2012 @ 12:43 PM

    • In the mangle section where it mark traffic for port 80 traffic, exclude the required ip in dst-address by using exclamation sign ! and define the ip there.

      Comment by Syed Jahanzaib / Pinochio~:) — November 28, 2012 @ 12:54 PM

  40. bagi yang ingin settingan terbaru proxy dengan cache youtube 1hit full (no-range) dan mikrotik bisa add MSN saya : spider84_gun@live.com
    atau FB saya http://www.facebook.com/mikrotik.medan

    Comment by mikrotik — November 29, 2012 @ 8:13 AM

  41. Dear jahanzaib bhai your blog is helping me alot and may Allah bless!!! main nay yeh configuration apply ki hay kindly mje yeh bta den k mikrotik k nat main jo dstnat wala rule hay jis se http traffic ko squid pe redirect krtay hain woh lagana hay ya nahi

    Comment by waqar ahmad — January 21, 2013 @ 6:41 PM

  42. Slam bhai! it is not working, i tried it many time but it is not working as you defined it …… please help me

    Comment by smn4all — January 22, 2013 @ 3:21 PM

  43. Reblogged this on SHERY's BLOG ON COMPUTER NETWORKING/I.T TIPS.

    Comment by Shery — April 4, 2013 @ 10:38 PM

  44. Dear Jehanzaib \
    Asslam-o-Alykum

    How may I use Isa web Proxy Server with Mikrotik as a proxy server.

    Comment by dgnetcables Syed Muhammad Kaleem — April 7, 2013 @ 2:59 AM

    • there are several ways to accomplish it.
      you can configure mikrotik web proxy and use ISA as parent proxy , this is very common configuration and requires no additional rules etc.
      Or you can just redirect user http port 80 traffic from your router to ISA server. On ISA server just create default allow rule and configure cache accordingly :) simple

      Comment by Syed Jahanzaib / Pinochio~:) — April 7, 2013 @ 3:46 PM

  45. hi sir iam aly fouda
    i have pfsense + lusca-cache in hp computer 6 gig ram an 640 hard drive
    and mikrotik on pc
    all like this router to pfsens to mikrotik to switch for coustmer

    i need any reaal way to make all in my network

    router=10.20.30.1
    pfsense=172.20.20.1
    mikrotik = 10.6.6.1/22
    i need really fast web browsing for all

    Comment by aly fouda — May 12, 2013 @ 3:14 PM

    • I really couldn’t understand your query. Please refine your question.

      Comment by Syed Jahanzaib / Pinochio~:) — May 13, 2013 @ 8:37 AM

      • after i did pfsense the prowsing is very slow what can i do syed

        Comment by aly fouda — May 13, 2013 @ 9:41 PM

      • without knowing the network scenario or details how you have implemented things, its hard to suggest any thing. in short, there should be no dramatically impact either positive or negative , either you use mikrotik pfsense, however correct configuration should be implemented.

        Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2013 @ 3:17 PM

  46. i have this storeurl.pl and really cached youtube in pfsense

    #!/usr/bin/perl
    # ==========================================================================
    # $Rev$
    # by chudy_fernandez@yahoo.com
    # Updates at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
    # ==========================================================================
    $|=1;
    while () {
    @X = split;
    # $X[1] =~ s/&sig=.*//;
    $x = $X[0] . ” “;
    $_ = $X[1];
    $u = $X[1];
    # ==========================================================================
    # Speedtest
    # ==========================================================================
    #if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
    # print $x . “http://www.speedtest.net.SQUIDINTERNAL/speedtest/” . $2 . “\n”;
    # ==========================================================================
    # Mediafire
    # ==========================================================================
    #} elseif
    if (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
    print $x . “http://www.mediafire.com.SQUIDINTERNAL/” . $1 .”/” . $2 . “\n”;
    # ==========================================================================
    # Fileserve
    # ==========================================================================
    } elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
    print $x . “http://www.fileserve.com.SQUIDINTERNAL/” . $1 . “./” . $2 . “\n”;
    # ==========================================================================
    # Filesonic
    # ==========================================================================
    } elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
    print $x . “http://www.filesonic.com.SQUIDINTERNAL/” . $1 . “\n”;
    # ==========================================================================
    # 4shared
    # ==========================================================================
    } elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
    print $x . “http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n”;
    # ==========================================================================
    # 4shared preview
    # ==========================================================================
    } elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
    print $x . “http://www.4shared.com.SQUIDINTERNAL/$2\n”;
    # ==========================================================================
    # Photos-X.ak.fbcdn.net where X a-z
    # ==========================================================================
    } elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
    print $x . “http://photos” . $1 . “/” . $2 . $3 . “\n”;
    # ==========================================================================
    # YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
    # ==========================================================================
    } elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
    print $x . “http://photos.ak.fbcdn.net/” . $1 .”/”. $2 . “\n”;
    # ==========================================================================
    # maps.google.com
    # ==========================================================================
    } elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
    print $x . “http://” . $1 . $2 . “\n”;
    # ==========================================================================
    # Compatibility for old cached get_video?video_id
    # ==========================================================================
    } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
    $z = $2; $z =~ s/video_id=/get_video?video_id=/;
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $z . “\n”;
    # ==========================================================================
    # Youtube fix
    # ==========================================================================
    } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
    $p_str = $2;
    $tag = “”;
    $alg = “”;
    $id = “”;
    $range = “”;
    if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
    if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
    if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
    if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
    # ==========================================================================
    # Google Analytic
    # ==========================================================================
    } elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
    print $x . “http://www.google-analytics.com/__utm.gif\n”;
    # ==========================================================================
    # Cache High Latency Ads
    # ==========================================================================
    } elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
    $y = $3;$z = $2;
    for ($y) {
    s/pixel;.*/pixel/;
    s/activity;.*/activity/;
    s/(imgad[^&]*).*/\1/;
    s/;ord=[?0-9]*//;
    s/;&timestamp=[0-9]*//;
    s/[&?]correlator=[0-9]*//;
    s/&cookie=[^&]*//;
    s/&ga_hid=[^&]*//;
    s/&ga_vid=[^&]*//;
    s/&ga_sid=[^&]*//;
    # s/&prev_slotnames=[^&]*//
    # s/&u_his=[^&]*//;
    s/&dt=[^&]*//;
    s/&dtd=[^&]*//;
    s/&lmt=[^&]*//;
    s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
    s/[;&?]ord=[?0-9]*//;
    s/[;&]mpvid=[^&;]*//;
    s/&xpc=[^&]*//;
    # yieldmanager
    s/\?clickTag=[^&]*//;
    s/&u=[^&]*//;
    s/&slotname=[^&]*//;
    s/&page_slots=[^&]*//;
    }
    print $x . “http://” . $1 . $2 . $y . “\n”;
    # ==========================================================================
    # Cache high latency ads
    # ==========================================================================
    } elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
    print $x . “http://” . $1 . “/” . $2 . “\n”;
    # ==========================================================================
    # spicific servers starts here….
    # ==========================================================================
    } elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
    print $x . “http://” . $1 . “\n”;
    # ==========================================================================
    # cdn, varialble 1st path
    # ==========================================================================
    } elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
    @y = ($1,$2,$4,$5);
    $y[0] =~ s/[a-z0-9]{2,5}/cdn./;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;
    # ==========================================================================
    # Rapidshare
    # ==========================================================================
    } elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
    print $x . “http://cdn.” . $3 . “/SQUIDINTERNAL/” . $5 . “\n”;
    } elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
    print $x . “http://” . $1 . “/SQUIDINTERNAL/” . $3 . “\n”;
    # ==========================================================================
    # like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
    # ==========================================================================
    } elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
    print $x . “http://cdn.” . $4 . $6 . “\n”;
    #…spicific servers end here.
    # ==========================================================================
    # Photos-X.ak.fbcdn.net where X a-z
    # ==========================================================================
    } elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
    print $x . “http://photos.ak.fbcdn.net/” . $1 . “\n”;
    # ==========================================================================
    # For yimg.com video
    # ==========================================================================
    } elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
    print $x . “http://cdn.yimg.com//” . $3 . “\n”;
    # ==========================================================================
    # For yimg.com doubled
    # ==========================================================================
    } elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
    print $x . “http://cdn.yimg.com/” . $3 . “\n”;
    # ==========================================================================
    # For yimg.com with &sig=
    # ==========================================================================
    } elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
    @y = ($1,$2);
    $y[0] =~ s/[a-z]+[0-9]+/cdn/;
    $y[1] =~ s/&sig=.*//;
    print $x . “http://” . $y[0] . “.yimg.com/” . $y[1] . “\n”;
    # ==========================================================================
    # Youjizz. We use only domain and filename
    # ==========================================================================
    } elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
    @y = ($1,$2,$4,$5);
    $y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;
    # ==========================================================================
    # General purpose for cdn servers. add above your specific servers.
    # ==========================================================================
    } elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
    print $x . “http://squid-cdn-url//” . $2 . “.” . $3 . “\n”;
    # ==========================================================================
    # Generic http://variable.domain.com/path/filename.”ex” “ext” or “exte” with or withour “? or %”
    # ==========================================================================
    } elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
    @y = ($1,$2,$3,$4);
    $y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
    print $x . “http://” . $y[0] . $y[1] . “/” . $y[2] . “.” . $y[3] . “\n”;
    # ==========================================================================
    # generic http://variable.domain.com/
    # ==========================================================================
    } elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
    print $x . “http://cdn.” . $3 . “.” . $4 . “/” . $5 . “\n”;
    # ==========================================================================
    # spicific extention that ends with ?
    # ==========================================================================
    } elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)(.*)/) {
    print $x . “http://” . $1 . “/” . $2 . “.” . $3 . “\n”;
    # ==========================================================================
    # all that ends with ;
    # ==========================================================================
    } elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
    print $x . “http://” . $1 . “/” . $2 . “\n”;
    } else {
    print $x . $_ . “sucks\n”;
    }
    }

    Comment by aly fouda — May 12, 2013 @ 3:24 PM

  47. it is my config in pfsense

    # Do not edit manually !
    http_port 172.20.20.1:3128 transparent
    http_port 127.0.0.1:80 transparent
    icp_port 0

    pid_filename /var/run/squid.pid
    cache_effective_user proxy
    cache_effective_group proxy
    error_directory /usr/local/etc/squid/errors/English
    icon_directory /usr/local/etc/squid/icons
    visible_hostname localhost
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    logfile_rotate 90
    shutdown_lifetime 0 seconds
    # Allow local network(s) on interface(s)
    acl localnet src 172.20.20.0/255.255.255.0
    uri_whitespace strip
    dns_nameservers 127.0.0.1

    cache_mem 1536 MB
    maximum_object_size_in_memory 128 KB
    memory_replacement_policy lru
    cache_replacement_policy heap LFUDA
    cache_dir coss /var/squid/coss 2048 max-size=65536 block-size=512
    cache_dir aufs /var/squid/cache 309600 16 256 min-size=65536
    minimum_object_size 0 KB
    maximum_object_size 100 MB
    offline_mode off
    cache_swap_low 96
    cache_swap_high 98

    # No redirector configured

    # Setup some default acls
    acl all src 0.0.0.0/0.0.0.0
    acl localhost src 127.0.0.1/255.255.255.255
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 1025-65535
    acl sslports port 443 563
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT
    acl partialcontent_req req_header Range .*
    #acl dynamic urlpath_regex cgi-bin \?
    include /usr/local/etc/squid/include.conf
    #cache deny dynamic
    http_access allow manager localhost

    # Allow external cache managers
    acl ext_manager_1 src 172.20.20.1
    http_access allow manager ext_manager_1

    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports

    # Always allow localhost connections
    http_access allow localhost

    quick_abort_min 0 KB
    quick_abort_max 0 KB
    range_offset_limit 0 MB
    request_body_max_size 0 allow all
    reply_body_max_size 0 deny all

    # Custom options
    zph_mode tos
    zph_local 0×30
    zph_parent 0
    zph_option 136

    # Allow local network(s) on interface(s)
    http_access allow localnet

    # Default block all to be sure
    http_access deny all

    Comment by aly fouda — May 12, 2013 @ 5:12 PM

    • can u see this config good or no syed

      Comment by aly fouda — May 13, 2013 @ 9:27 PM

  48. amin wa’alaikum salam
    Good day, I have just been employed in a company and I don’t know much about Mikrotik. Now I have an issue which invariably is a test as they have said that I have to ensure that I build a cache outside the Mikrotik and that if that improves the speed of browsing tremendously that means I have the job.
    Let me describe the network to you:
    Modem>Gateway(x86 PC)> Mikrotik (Rb1100 or x86)> switch>antennae

    I would like to incorporate both the cache option to do both web-pages and also videos, I would like to do both on the same machine. Also I would appreciate it if you could please give me detailed steps on what to do on each; both on the Linux package and the Mikrotik package. The mikrotik version is 5.22 and the Ubuntu version is 13.04 64 bits.

    Comment by Ahmed Bello — May 30, 2013 @ 12:51 PM

    • As-Salaam-Alaikum

      Thank you very much for your former pointers they have been quite informative!

      Please I would like to ask a question, I am buying bulk bandwidth and
      I am wondering is there any special equipment required on my end minus
      my usual gateway (Clarkconnect is the name of the gateway server)
      that I have used for other T1 connections? Why an asking is that the
      bandwidth is coming as 1024kb and what am used to before is 512up by
      1024down! Would I need any special equipment to break it down to up
      and down? or I should just plug it in?

      Also the above bandwidth was on a 1:3 contention ratio would I doing
      way too much hy going for 1024kb bulk bandwidth? I have a feeling that
      I should go for the 512kb as its bulk which supposedly means pure!
      Thanks.

      Comment by ahmedbello68 — September 17, 2013 @ 12:54 PM

  49. Dear sir, i want all of my internet traffic goes to squid and my squid is clarkconnect and i am just using mikrotik for bandwith managment and proxy load balancing.And secondly how to control https traffic in transparent mode as https traffic bypass transparent squid filter.

    Comment by Nadir — May 31, 2013 @ 8:09 PM

  50. Sir i have search allot about https traffic, some one in blogs said if u install ssl or key on your squid server and redirect https traffice using iptables etc than you will be able to controll https traffic. just like tmg https interception.

    Comment by Nadir — June 4, 2013 @ 11:56 AM

    • SSL or key meas some sort of certificates

      Comment by Nadir — June 4, 2013 @ 11:57 AM

    • Hi
      I dont know what you mean by control. But you can cache it. I have configured it and it is working fine but i’m facing two problems:
      1) I have to configure each client’s browser and give proxy server address & port.
      2) original client source ip does not show. it shows mikrotik box lan ip.

      My scenario>>> Mikrotik (pcc loadbalancing + hotspot) + Squid3.x on ubuntu (not working in transparent mode & cannot. Because of https caching configuration)

      Comment by Umer Sarwar — October 18, 2013 @ 10:11 AM

      • Is there any way or settings so i won’t have to configure each client browser for proxy settings? Client gets ip from mikrotik dhcp but it does not browse until i give proxy server ip & port in browser, It gives error “the url cannot retrieved. Missing hostname etc. contact Administrator”
        And i need to do it without using transparent mode of squid.

        Comment by Umer Sarwar — October 18, 2013 @ 10:39 AM

      • how you are doing port redirection to squid?
        you can use ds-nat rule to redirect port 80 to squid ip:port

        or you can use mark and route method too which will preserve users original ip address in squid logs.
        you can log user original ip in squid logs with dst-nat method too, but make sure squid is on different subnet with 3rd interface, and default masquerade rule should have WAN interface specifically mentioned in OUT INT.

        Comment by Syed Jahanzaib / Pinochio~:) — October 18, 2013 @ 11:12 AM

      • WAN ip: 192.168.1.1
        Mikrtotik (pcc+hotspot): 192.168.3.1/255.255.255.0 attached to squid via cross-over
        Squid ip: 192.168.3.2/255.255.255.0 port 8080 (NOT in transparent mode)

        I have applied below 3 rules in mikrotik as you mentioned above.
        ——————————————————————————————————————————————————————————————–
        1)Mark All HTTP Port 80 Traffic, so that we can use these Marked Packets in Route section.

        /ip firewall nat
        add action=accept chain=srcnat disabled=no dst-port=80 protocol=tcp

        /ip firewall mangle
        add action=mark-routing chain=prerouting disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp
        ————————————

        2) Masquerade all traffic (Except http] on ether3 Only, which is connected with DSL Router. This is important to masquerade traffic on WAN Interface only, otherwise http packets will also be masqueraded with mikrotik ip.

        add action=masquerade chain=srcnat disabled=no out-interface=ether4 (Note: ether4 = squid)
        ————————————-

        3) Define Route for HTTP Marked packets, and set default rule for all other traffic, This is called policy base or pre traffic base routing

        /ip route
        add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.3.2 routing-mark=http scope=30 target-scope=10
        add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.1 scope=30 target-scope=10
        ————————————————————————————————————————————————————————————————–

        But all it shows in squid3 access.log is mikrotik ip 192.168.3.1
        Also i would like to tell you again squid3 is NOT in transparent mode & its caching https contents also.

        Comment by Umer Sarwar — October 18, 2013 @ 12:13 PM

      • in DEFAULT MASQUERADE RULE, specify WAN interface specifically in OUT INTERFACE section. then squid will log users original ip.

        Comment by Syed Jahanzaib / Pinochio~:) — October 19, 2013 @ 2:23 PM

  51. Hi Syed, I have looked through all your documentation but not sure if you will be able to help. Im wanting clients connected using pppoe accounts to access local servers as fast as the cable connection will allow but still limit them on all other data. I have tried several ways like marking the traffic then setting the limit to 1gb in the queue tree, but this does not bypass the simple queue limit when they authenticate using pppoe. Have tried creating a new simple queue and placed it to the top, and marked selected the marked traffic but with not success.
    Please help

    Comment by jacques grundling — June 22, 2013 @ 3:48 PM

  52. As-Salaam-Alaikum

    Thank you very much for your former pointers they have been quite informative!

    Please I would like to ask a question, I am buying bulk bandwidth and
    I am wondering is there any special equipment required on my end minus
    my usual gateway (Clarkconnect is the name of the gateway server)
    that I have used for other T1 connections? Why an asking is that the
    bandwidth is coming as 1024kb and what am used to before is 512up by
    1024down! Would I need any special equipment to break it down to up
    and down? or I should just plug it in?

    Also the above bandwidth was on a 1:3 contention ratio would I doing
    way too much hy going for 1024kb bulk bandwidth? I have a feeling that
    I should go for the 512kb as its bulk which supposedly means pure!
    Thanks.

    Comment by ahmedbello68 — September 17, 2013 @ 12:55 PM

    • You dont need any special equipment other then mikrotik to do the job. Mikrotik can handle all sort of traffic on itself. if configured properly :)

      Comment by Syed Jahanzaib / Pinochio~:) — October 20, 2013 @ 1:49 PM

  53. Thank You brother, its work for me.
    it won’t work before but after I moved srcnat accept port 80 before masquerade rule. now its working fine.
    Very nice tut

    Comment by aghe milano — September 25, 2013 @ 7:47 AM

  54. Hi,

    1) You adviced me in the above reply to put Squid in different subnet.
    Can you write down the diffferent subnet for me? Please.
    My Current situation:
    DSL 192.168.1.1/24
    Mikrotik 192.168.0.0/24
    Squid 192.168.3.1/24

    2) You also said ” in Default MASQUERADE Rule, Out Interface = WAN”
    Im using PCC load-balancing so i have Two Masquerade Rules, so i put Wan1 in MSQ and Wan2 in other MSQ.

    3) I have two interface on Squid box, One for Internet & Second for Mikrotik.
    My Squid 3.2 is NOT in Transparent mode So according to your blog & your replies to other people, I should go for MARK & ROUTE method to Redirect Traffic to Squid, to aviod the configuration of client’s browsers and because it is also the most preffered way, Right?

    4) As i said before, im using PCC Load-Balancing. I got this from the posts of an old topic on Mikrotik forum (http://forum.mikrotik.com/viewtopic.php?f=13&t=45114). What are your views & recommendations? What should i do in Mangle rules?

    A: “You need to do exempt the traffic, that is going to be redirected to Squid, from having the PCC ruleset applied to it (So accept traffic with the same parametes you are going to use to NAT to Squid, as the FIRST STEP (RULE) in the Prerouting Mangle Set.) Traffic makes it to the proxy without being torn apart by PCC, then the proxy takes the traffic & requests the websites, THAT traffic now matches for PCC & is Load Shared”

    B: “Exempting the traffic will kill the purpose of PCC”

    A: “Only exempt the traffic with same source address, interface, prtocol, port, etc. Exempt Exactly the traffic going to the LAN to Squid. The traffic from Squid to the web still goes through PCC”.

    Comment by Umer Sarwar — October 20, 2013 @ 6:11 PM

  55. Thank you for the assistance and sorry for the long comments.
    Problem solved and i have a working squid 3.x ssl shebang dynamic contents caching. No manual configuration of clients computer ip or proxy address:port in client browsers.

    It all came to one word, adding it in front of “http_port 3128″ and that was “intercept” (http_port 3128 intercept).

    All with the help of Mr. Zaib, Thank you.

    Comment by Umer Sarwar — October 21, 2013 @ 1:34 PM

  56. this work great for me, But now i want to use it with One Lan card CPU, because there is not any PCI slot in PC, So kindly please tell me what i need to change and configure to work with single Builtin LAN card
    Thanks in advance

    Comment by hammad hassan — December 8, 2013 @ 12:20 PM

  57. Hello.. Can this be done for HTTPS also ?

    Comment by Parijat Purkayastha — January 15, 2014 @ 4:25 PM

    • yes

      Comment by Syed Jahanzaib / Pinochio~:) — January 17, 2014 @ 7:20 PM

      • Thanks :) I am running out of public IP’s and need to log HTTPS access to sites like facebook and banks… So how do i do it ?? same way but use use port 443 instead ??

        Regards..parijat

        Comment by Parijat Purkayastha — January 19, 2014 @ 11:40 AM

  58. at the moment i am using web proxy with Public IP pool for pppoe… It would be great if you can assist me ..

    Comment by Parijat Purkayastha — January 19, 2014 @ 11:42 AM


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 2,049 other followers

%d bloggers like this: