Youtube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.pl

YOUTUBE CACHING WITH SQUID -by- Syed Jahanzaib

 UPDATED: 10th MAY, 2012

====================================================================

BAD NEWS:

Some research updates: (Someone please confirm and post your comments)

YOUTUBE has split its videos into segments of 1.7 Mb each which is around 51 seconds of video clip length.
I am sure YOUTUBE have taken this step to prevent people from caching  entire videos.
If you have a video which is 100 Mb large, it will be split into about 55-60 segments.
[I consider it as a benefit , it will prevent squid from filling quickly, because many users dont watch video entirely and jump to next one without finishing the video.]
As of right now, storeurl.pl is not be able to cache Youtube Video entirely, it is caching only initial 51 seconds of the clip.
Currently VIDEOCACHE plugin is doing full cache of youtube but at higher  400  $$$ cost yearly [Syed JahanZaib]

====================================================================

If you are caching youtube using storeurl.pl method
Example: http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/ 

and you encounter following error while watching any cached video
“An error occured, Please try again later“
As showed in the image below . . .

Then try using the following updated storeurl.pl

#!/usr/bin/perl
# This script is NOT written or modified by me, I only copy pasted it from the internet.
# It was First originally Writen by chudy_fernandez@yahoo.com
# & Have been modified by various persons over the net to fix/add various functions.
# Like For Example modified by member of comstuff.net to satisfy common and dynamic content.
# th30nly @comstuff.net a.k.a invisible_theater , and possibly other people too.
# For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
# Syed Jahanzaib / aacable @ hotmail . com

$|=1;
while (<>) {
@X = split;
#       $X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];

if ($X[1] =~ /(youtube|google).*videoplayback\?/){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@range = m/[&?](range=[^\&\s]*)/;
@begin = m/[&?](begin=[^\&\s]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@begin@redirect\n";

#speedtest
}elsif (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
print $x . "http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . "\n";

#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . "http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . "\n";

#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
print $x . "http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . "\n";

#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
print $x . "http://www.filesonic.com.SQUIDINTERNAL/" . $1 . "\n";

#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n";

#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/$2\n";

#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
print $x . "http://photos" . $1 . "/" . $2 . $3  . "\n";

#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  ."/". $2 . "\n";

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
print $x . "http://" . $1  . $2 . "\n";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
#...spicific servers end here.

#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn.yimg.com//" . $3 . "\n";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/"  . $3 . "\n";

#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";

#generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %"
} elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

# generic http://variable.domain.com/...
} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";

# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar|rar|cab|amf|swf)(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

} else {
print $x . $_ . "sucks\n";
}
}

I have deployed this storeurl.pl at various different networks, and in my LAB, and so far result is good , although sometimes user still face errors in few videos, but the ratio is very less.
Also try this update:
storeurl_rewrite_children 15
storeurl_rewrite_concurrency 100

Please post your comments regarding updated storeurl.pl

.
.
.

Regard’s
SYED JAHANZAIB

Mikrotik Port Forwarding to Local Server on LAN

If you want to host any web server (or any other service like RDP or Game Server) behind mikrotik server and you want it to be publicly available for all internet users, you can use port forwarding and create one dstnat rule as below.Just make sure this rule comes above any masquerading rule.

Scenario:

DSL MODEM WAN IP = 221.xxx.xxx.xxx
DSL LAN IP = 192.168.1.1

MIKROTIK WAN IP = 192.168.1.2
MIKROTIK LAN IP = 192.168.0.1

WEB SERVER IP = 192.168.0.50

First setup port forwarding in your dsl modem to forward port 80 request to your mikrotik, I am not showing DSL modem config, as its very different for every mode, search for your modem confg page on howto do port forwarding. Just an example here for my Wi.Fi MODEM page.

Then in mikrotik , add an rule to forward port 80 request to your local web server, (one that is hosted behind your mikrotik server, on local user LAN)

MIKROTIK RULE :

/ip firewall nat
add action=dst-nat chain=dstnat disabled=no dst-port=80 in-interface=WAN2-QUBEE protocol=tcp to-addresses=192.168.0.50 \
to-ports=80

The above rule result would be something like below.

Regard's
Syed Jahanzaib

Howto Disabe IPV6 in FEDORA 10

Howto To disable IPV6 in FEDORA 10 ~

Issue the following commands in terminal, (You must be logged in as root)

su -c 'echo "install ipv6 /bin/true
blacklist ipv6" > /etc/modprobe.d/blacklist-ipv6.conf'

su -c 'service ip6tables stop && chkconfig ip6tables off'

After making these changes reboot your system. Then check your interface card by

ifconfig

You will notice that you IPV6 lines have been gone

Regard’s
Syed Jahanzaib

SQUID: Howto Exclude some sites & Exntension from CACHING

Howto Exclude Few Sites from Caching

If you are using SQUID proxy server and you don’t want to cache few sites , use the following directives,

acl NO-CACHE-SITES dstdomain "/etc/squid/not-to-cache-sites.txt"
no_cache deny NO-CACHE-SITES

Now create the file which will contains our sites list which we don’t want to cache.

touch /etc/squid/not-to-cache-sites.txt

and add  your desired web sites name in /etc/squid/not-to-cache-sites.txt
For example
nano /etc/squid/not-to-cache-sites.txt
and add following or your entries


bankalhabib.com
aacable.wordpress.com
wordpress.com
nae.com.pk
jang.com.pk

Now we have to reload squid.conf so changes may apply by following command

squid -k rec
OR
squid -k reconfigure

Howto Exclude Specific Extension from Caching

Similarly if you don’t want to cache *.FLV files , use the following directives

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \? \.flv
no_cache deny QUERY

Now we have to reload squid.conf so changes may apply by following command

squid -k rec
OR
squid -k reconfigure
Regard's
Syed Jahanzaib

UBUNTU rc.local not executing problem and its workaround

Sometimes in ubuntu, rc.local does not execute , specially in fresh installation.
So if you encounter this issue , you can solve it by using two methods.
I have used these methods several time with success. Also make sure you command entries are above ‘exit 0‘ line

Method # 1 , By Using WEBMIN

Method # 2 , By Removing -e from /etc/rc.loacl file

Method # 1 , By Using WEBMIN

Install webmin from http://www.webmin.com/deb.html

After installati0n is done, open webmin and goto
SYSTEM > BOOTUP & SHUTDOWN
Scroll down and and Click on rc.local
Select YES on Start at boot time?

As shown in the image below.

 

 

 

Method # 2 , By Removing -e from /etc/rc.loacl file

Edit rc.local file by

nano /etc/rc.local

at first line remove -e  for example

Change from

#!/bin/sh -e

to

#!/bin/sh

Save & Exit, then Reboot and see, hopefully rc.local will execute on the boot.

Ubuntu Memory Limitation Solution

If you install Ubuntu 32bit , and you have more then 4 GB of  RAM, you won’t be able to use it, The easiest solution is to grab latest 64 bit version and install it (Only if your hardware supports 64bit architecture) otherwise you have to use PAE enabled kernel.

For example if you have 8GB of ram, and you issue the command,

free -m

you will see only 3-4 GB of  total ram. You need to install Physical Address Extension (PAE) aware kernel under 32 bit Ubuntu Linux. It is a feature of x86 and x86-64 processors that allows more than 4 Gigabytes of physical memory to be used in 32-bit systems.

HOWTO Install PAE enabled kernel in UBUNTU

Open terminal and type the following command

If you are using Ubuntu v9.10 (Karmic Koala) and above, enter:

apt-get install linux-generic-pae linux-headers-generic-pae

Once kernel images installed, just reboot your workstation, type:

reboot

After reboot, login into your system and type the following command to verify memory usage:

free -m

if you are using Ubuntu version Ubuntu v9.04 and earlier:

apt-get update
sudo apt-get install linux-headers-server linux-image-server linux-server

Now reboot and check memory by free -m


Youtube caching with SQUID 2.7 [using storeurl.pl]

 UPDATED: 10th MAY, 2012

BAD NEWS:

Some research update (Someone Please confirm & post your comments)

YOUTUBE has split its videos into segments of 1.7 Mb each which is the approximation of the 51 seconds. I am sure YOUTUBE have taken this step to prevent people from caching  entire videos, [In my opinion its good to prevent squid from filling up squickly]. If you are watching a a video which is 100 Mb large, it will be split into about 55-60 segments.
As of right now, storeurl.pl wont be able to cache entire video, only first 51 seconds will be cached.
As of right now VIDEOCACHE plugin is doing full cache of youtube videos but at higher 400 $ Per Year price

I will update if I found out any solution to it. Currently I am busy in my new job, so its very hard to manage time for this.

UPDATED: 2nd April, 2012

This is a Shorten Version of  >  http://aacable.wordpress.com/tag/aacable-howto-cache-youtube/

This is a quick reference guide for SQUID 2.7 installation on Ubuntu Desktop ver 10.4 with youtube caching supported. Make sure you have setup proper internet connection in Ubuntu BOX.

Install SQUID by

apt-get install squid

After installation done, Now edit it’s configuration file

nano /etc/squid/squid.conf

remove current lines and paste all squid.conf as follows.

# SQUID 2.7/ LUSCA TEST CONFIG FILE
# Email: aacable@hotmail.com
# Web  : http://aacable.wordpress.com

# PORT and Transparent Option
http_port 8080 transparent
server_http11 on
icp_port 0

# Cache Directory , modify it according to your system.
# but first create directory in root by mkdir /cache1
# and then issue this command  chown proxy:proxy /cache1
# [for ubuntu user is proxy, in Fedora user is SQUID]
# I have set 500 MB for caching reserved just for caching ,
# adjust it according to your need.
# My recommendation is to have one cache_dir per drive. zzz

#store_dir_select_algorithm round-robin
cache_dir aufs /cache1 500 16 256
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA

# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off

# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

#I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 221.132.112.8
ftp_user anonymous@
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on

#ACL Section
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
http_reply_access allow all
icp_access allow all

#==========================
# Administrative Parameters
#==========================

# I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr aacable@hotmail.com
visible_hostname proxy.aacable.net
unique_hostname aacable@hotmail.com

#=============
# ACCELERATOR
#=============
half_closed_clients off
quick_abort_min 0 KB
quick_abort_max 0 KB
vary_ignore_expire on
reload_into_ims on
log_fqdn off
memory_pools off
cache_swap_low 98
cache_swap_high 99
max_filedescriptors 65536
fqdncache_size 16384
retry_on_error on
offline_mode off
pipeline_prefetch on
# If you want to hide your proxy machine from being detected at various site use following
via off

#============================================
# OPTIONS WHICH AFFECT THE CACHE SIZE / zaib
#============================================
# If you have 4GB memory in Squid box, we will use formula of 1/3
# You can adjust it according to your need. IF squid is taking too much of RAM
# Then decrease it to 128 MB or even less.

cache_mem 8 MB
minimum_object_size 0 bytes
maximum_object_size 100 MB
maximum_object_size_in_memory 128 KB

#============================================================$
# SNMP , if you want to generate graphs for SQUID via MRTG
#============================================================$
#acl snmppublic snmp_community gl
#snmp_port 3401
#snmp_access allow snmppublic all
#snmp_access allow all

#============================================================
# ZPH , To enable cache content to be delivered at full lan speed,
# To bypass the queue at MT.
#============================================================
tcp_outgoing_tos 0x30 all
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

# Caching Youtube
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp)
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.apniisp\.com\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com
acl videocache_deny_dom  dstdomain .download.youporn.com .static.blip.tv
acl dontrewrite url_regex redbot\.org \.php
acl getmethod method GET

storeurl_access deny dontrewrite
storeurl_access deny !getmethod
storeurl_access deny videocache_deny_dom
storeurl_access allow videocache_allow_url
storeurl_access allow videocache_allow_dom
storeurl_access deny all

storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 100

acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \
storeurl_access allow store_rewrite_list
storeurl_access deny all

refresh_pattern -i \.flv$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp3$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp4$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.swf$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.gif$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpeg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth
refresh_pattern -i \.exe$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth

# 1 year = 525600 mins, 1 month = 10080 mins, 1 day = 1440
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern \.(ico|video-stats) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080
refresh_pattern \.etology\?                                     10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern galleries\.video(\?|sz)                         10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern brazzers\?                                      10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern \.adtology\?                                    10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 10080 20% 10080 ignore-no-cache  ignore-private override-expire ignore-reload ignore-auth   negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google  10080 80% 10080 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth  negative-ttl=10080
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)    10080 80% 10080 override-expire ignore-reload ignore-private  negative-ttl=10080
refresh_pattern ytimg\.com.*\.jpg                                       10080 80% 10080 override-expire ignore-reload
refresh_pattern images\.friendster\.com.*\.(png|gif)                    10080 80% 10080 override-expire ignore-reload
refresh_pattern garena\.com                                             10080 80% 10080 override-expire reload-into-ims
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)          10080 80% 10080 override-expire ignore-reload
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                     10080 80% 10080 ignore-no-cache override-expire override-lastmod
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)    10080 80% 10080 reload-into-ims override-expire ignore-private
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                     10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http:\/\/www.onemanga.com.*\/                          10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080

#images facebook
refresh_pattern -i \.facebook.com.*\.(jpg|png|gif)                      10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)        10080 80% 10080 ignore-reload override-expire ignore-no-cache

#All File
refresh_pattern -i \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)       10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar)  10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)         10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims

refresh_pattern -i (/cgi-bin/|\?)  0  0%  0
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern ^ftp:           10080   95% 10080 override-lastmod reload-into-ims
refresh_pattern .               1440    95% 10080 override-lastmod reload-into-ims

Save& Exit.

STOREURL.PL

Now create storeurl.pl which will be used to pull youtube video from cache.

touch /etc/squid/storeurl.pl
chmod +x /etc/squid/storeurl.pl

Now edit this file and paste the following contents.

nano /etc/squid/storeurl.pl

#!/usr/bin/perl
# This script is NOT written or modified by me, I only copy pasted it from the internet.
# It was First originally Writen by chudy_fernandez@yahoo.com
# & Have been modified by various persons over the net to fix/add various functions.
# For Example this ver was modified by member of comstuff.net to satisfy common and dynamic content.
# th30nly @comstuff.net a.k.a invisible_theater ,
# For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
$|=1;
while (<>) {
@X = split;
#       $X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];

#speedtest
if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
print $x . "http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . "\n";

#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . "http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . "\n";

#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
print $x . "http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . "\n";

#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
print $x . "http://www.filesonic.com.SQUIDINTERNAL/" . $1 . "\n";

#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n";

#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/$2\n";

#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
print $x . "http://photos" . $1 . "/" . $2 . $3  . "\n";

#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  ."/". $2 . "\n";

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
print $x . "http://" . $1  . $2 . "\n";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

# youtube fix
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = "";
$alg = "";
$id = "";
$range = "";
if ($p_str =~ m/(itag=[0-9]*)/){$tag = "&".$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = "&".$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = "&".$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = "&".$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $tag . "&" . $alg . "&" . $id . "&" . $range . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
#...spicific servers end here.

#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn.yimg.com//" . $3 . "\n";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/"  . $3 . "\n";

#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";

#generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %"
} elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

# generic http://variable.domain.com/...
} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";

# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

} else {
print $x . $_ . "sucks\n";
}
}

Save & Exit.

Now create cache dir and assign proper permission to proxy user

mkdir /cache1
chown proxy:proxy /cache1
chmod -R  777 /cache1

Now  initialize squid cache directories by

squid -z

You should see Following message

Creating Swap Directories

After this, start SQUID service by

service squid start

From Client end, point your browser to use your squid as proxy server and test any video.

TESTING YOUTUBE CACHING

From client side, View any youtube video, after viewing, clear your cache and play it again (or from another pc) and you will notice that the video loads instantly,

Example:

You can monitor the CACHE TCP_HIT ENTRIES in squid logs, you can view them by

tail -f /var/log/squid/access.log | grep HIT

More information can be found at

http://aacable.wordpress.com/tag/aacable-howto-cache-youtube/

If you face “An error occured” in cached videos , see the following

http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/

Regard’s

Syed Jahanzaib

Howto Recover Mikrotik ADMIN account Forgotten Password

According to information on Mikrotik WIKI and forums, it is not possible to recover the passwords without loosing configuration.
However following are few methods to recover it.

1# Recover password from BACKUP file using a website http://mikrotikpasswordrecovery.com/

2# Recover password from BACKUP file using Linux LIVE CD

3# Recover password by mounting Mikrotik Hard disk in Linux LIVE CD  and do recovery

1# Recover password from BACKUP file using a website http://mikrotikpasswordrecovery.com/

You can easily recover your forgotten password if you have BACKUP file.
Backup file is done by following command.

/system backup save name=zaibmikrotik

Open Following web site.
http://mikrotikpasswordrecovery.com/default.aspx

Click on Browse, and select your backup file,
Now click on ” Shows Passwords “

It will show you your Password in few seconds.
As shown in the image below . . .

Simple enough ??

2# Recover password from BACKUP file using Linux

Login to your Ubuntu / Linux Box,
Download mikrotik password recovery tool and compile it by following commands,

mkdir /temp
cd /temp
wget http://manio.skyboo.net/mikrotik/mtpass-0.7.tar.bz2
tar jxvf mtpass-0.7.tar.bz2
cd mtpass-0.7
make

Now copy your Mikrotik Backup File to /temp folder ( Either using WEBMIN File Manager,  via USB or any other method you like)
Now Issue the following command

 ./mtpass /temp/zaibmikrotik.backup

It will show you all account passwords in few seconds.
As shown in the image below . . .

3# Recover password of x86 PC Version

Boot from Ubuntu LIVE CD
(I used Ubuntu 9.1 Desktop CD in this example, you can download it from following link.

http://old-releases.ubuntu.com/releases/karmic/ubuntu-9.10-desktop-i386.iso

Select “Try Ubuntu“
As Shown in the image below . . .

After you see Desktop, Open TERMINAL from APPLICATION > ACCESSORIES > TERMINAL

Now change to root user by
sudo su

Now check your partitions by issuing
fdisk -l

you will see something like below image.

You partition can be different, use your judgment to see what partition mt is in, either by mounting it one by one.

Now mount it any folder , for example

mkdir /temp
mount -t auto /dev/sda2 /temp

Now check its content by ls /temp and you will something like below image

*********************************************************

*********************************************************

Now Copy the ‘mikrotik password file’, (in newer ROS , it is located in /rw/store/user.dat) to USB flash drive, It will be used to decode the password.
(The USB flash drive should be ‘plug and play’ in Ubuntu Live. Just plug it in usb port and it will appear on desktop in few seconds , OR you can also copy it your network pc via configuring interface lan card accordingly)
(Note: In older ROS it is /nova/store/user.dat)

Now shutdown live cd.

then Boot from your Ubuntu box, and use the Method # 2 , described earlier in this article to decode password using mtpass tool  from this file.

Copy the user.dat file where you have decompressed mt password tool . e.g /temp , now use the mt pass tool to recover password from this file.

./mtpass user.dat

and you will see your admin password.

As Now you have seen the password now, you can login into mikrotik pc router  with your Old Password

If your architecture is ROUTERBOARD RB series, Please follow the below guide.

http://manio.skyboo.net/mikrotik/

Also it’s possible for you to send an SMS to your router, tell it to run a script (parameters supported) and the router can even respond, as it also supports SMS sending! More here:
http://wiki.mikrotik.com/wiki/Sms

You can setup an script just in case you forgot your password , and via sending sms, it can reset it to default. or send you return the new/original password.

Regard’s
SYED JAHANZAIB

http://aacable.wordpress.com

Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue

UPDATED: 10th MAY, 2012

 

BAD NEWS:

Some research update (Someone please confirm and post your comments)
YOUTUBE has split its videos into segments of 1.5 Mb each which is the approximation of the 51 seconds. I am sure YOUTUBE have taken this step to prevent people from caching  entire videos. If you have a video which is 100 Mb large, it will be split into about 55-60 segments.
As of right now, storeurl.pl wont be able to cache it. Currently VIDEOCACHE plugin is

doing full cache of youtube but at higher 400 $ Per Year

What is LUSCA/SQUID ?

LUSCA is an advance version or Fork of  SQUID 2. The Lusca project aims to fix the shortcomings in the Squid-2. It also supports a variety of clustering protocols. By Using it, you can cache some dynamic contents that you previously can’t do with the squid.

For example
#  Video Cachingi.e Youtube / tube etc . . .
#  Windows / Linux Updates / Anti-virus , Anti-Malware i.e. Avira/ Avast / MBAM etc . . .
#  Well known sites i.e. facebook / google / yahoo etch. etch.
#  Download caching mp3′s/mpeg/avi etc . . .

Advantages of Youtube Caching   !!!

In most part of the world, bandwidth is very expensive, therefore it is (in some scenarios) very useful to Cache Youtube videos or any other flash videos, so if one of user downloads video / flash file , why again the same user or other user can’t download the same file from the CACHE, why he sucking the internet pipe for same content again n again?
Peoples on same LAN ,sometimes watch similar videos. If I put some youtube video link on on FACEBOOK, TWITTER or likewise , and all my friend will  watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my lan users / friends.

This is the reason why I wrote this article. I have implemented Ubuntu with LUSCA/ Squid on it and its working great, but to achieve some results you need to have some TB of storage drives in your proxy machine.

Disadvantages of Youtube Caching   !!!

The chances, that another user will watch the same video, is really slim. if I search for something specific on youtube, i get more then hundreds of search results for same video. What is the chance that another user will search for the same thing, and will click on the same link / result? Youtube hosts more than 10 million videos. Which is too much to cache anyway. You need lot of space to cache videos. Also accordingly you will be needing ultra modern fast hardware with tons of RAM to handle such kind of cache giant. anyhow Try it

We will divide this article in following Sections

1#  Installing SQUID / LUSCA in UBUNTU
2#  Setting up SQUID / LUSCA Configuration files
3#  Performing some Tests, testing your Cache HIT
4# Using ZPH TOS to deliver cached contents to clients vai mikrotik at full LAN speed, Bypassing the User Queue for cached contents.

1#  Installing SQUID / LUSCA in UBUNTU

I assume your ubuntu box have 2 interfaces configured, one for LAN and second for WAN. You have internet sharing already configured. Now moving on to LUSCA / SQUID installation.

Download LUSCA source and compile it using,

mkdir /temp
cd /temp
wget http://lusca-cache.googlecode.com/files/LUSCA_HEAD-r14809.tar.gz
tar xzvf LUSCA_HEAD-r14809.tar.gz

Update & Install some tools for ubuntu

sudo apt-get update
sudo apt-get install gcc build-essential sharutils ccze libzip-dev automake1.9
cd LUSCA_HEAD-r14809

Now compile LUSCA with following options

./configure '--prefix=/usr/local/squid' '--enable-removal-policies=heap,lru' '--disable-dependency-tracking' '--disable-arp-acl' '--disable-cache-digests' '--enable-cachemgr-hostname=localhost' '--disable-delay-pools' '--enable-epoll' '--enable-external-acl-helpers=ip_user' '--disable-ident-lookups' '--enable-linux-netfilter' '--disable-referer-log' '--enable-removal-policies=heap,lru' '--disable-snmp' '--disable-ssl' '--enable-storeio=aufs,null,coss' '--disable-useragent-log' '--disable-wccpv2' '--with-aio' '--with-maxfd=1048576' '--with-dl' '--with-pthreads' 'build_alias=i686-redhat-linux-gnu' 'host_alias=i686-redhat-linux-gnu' 'targe_alias=i686-redhat-linux-gnu''--enable-truncate' '--disable-unlinkd' '--with-large-files' '--disable-htcp'
sudo make all
sudo make install

All of Lusca/Squid configuration files can be found at

/usr/local/squid/etc/
and squid executable can be found at
/usr/local/squid/sbin/

Now We will edit squid.conf file to make it customize according to our requirements by . . .

nano /usr/local/squid/etc/squid.conf

Delete all previously lines , and paste the following lines.

Remember following squid.conf is not very neat and clean , you will find many un necessary junk entries in it, but as I didn’t had time to clean them all, so you may clean them as per your targets and goals.

# SQUID 2.7/ LUSCA TEST CONFIG FILE
# Email: aacable@hotmail.com
# Web  : http://aacable.wordpress.com

# PORT and Transparent Option
http_port 8080 transparent
server_http11 on
icp_port 0

# Cache Directory , modify it according to your system.
# but first create directory in root by mkdir /cache1
# and then issue this command  chown proxy:proxy /cache1
# [for ubuntu user is proxy, in Fedora user is SQUID]
# I have set 500 MB for caching reserved just for caching ,
# adjust it according to your need.
# My recommendation is to have one cache_dir per drive. zzz

#store_dir_select_algorithm round-robin
cache_dir aufs /cache1 500 16 256
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA

# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off

# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

#I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 221.132.112.8
ftp_user anonymous@
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on

#ACL Section
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
http_reply_access allow all
icp_access allow all

#==========================
# Administrative Parameters
#==========================

# I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr aacable@hotmail.com
visible_hostname proxy.aacable.net
unique_hostname aacable@hotmail.com

#=============
# ACCELERATOR
#=============
half_closed_clients off
quick_abort_min 0 KB
quick_abort_max 0 KB
vary_ignore_expire on
reload_into_ims on
log_fqdn off
memory_pools off
cache_swap_low 98
cache_swap_high 99
max_filedescriptors 65536
fqdncache_size 16384
retry_on_error on
offline_mode off
pipeline_prefetch on
# If you want to hide your proxy machine from being detected at various site use following
via off

#============================================
# OPTIONS WHICH AFFECT THE CACHE SIZE / zaib
#============================================
# If you have 4GB memory in Squid box, we will use formula of 1/3
# You can adjust it according to your need. IF squid is taking too much of RAM
# Then decrease it to 128 MB or even less.

cache_mem 8 MB
minimum_object_size 0 bytes
maximum_object_size 100 MB
maximum_object_size_in_memory 128 KB

#============================================================$
# SNMP , if you want to generate graphs for SQUID via MRTG
#============================================================$
#acl snmppublic snmp_community gl
#snmp_port 3401
#snmp_access allow snmppublic all
#snmp_access allow all

#============================================================
# ZPH , To enable cache content to be delivered at full lan speed,
# To bypass the queue at MT.
#============================================================
tcp_outgoing_tos 0x30 all
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

# Caching Youtube
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp)
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.apniisp\.com\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com
acl videocache_deny_dom  dstdomain .download.youporn.com .static.blip.tv
acl dontrewrite url_regex redbot\.org \.php
acl getmethod method GET

storeurl_access deny dontrewrite
storeurl_access deny !getmethod
storeurl_access deny videocache_deny_dom
storeurl_access allow videocache_allow_url
storeurl_access allow videocache_allow_dom
storeurl_access deny all

storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 100

acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \
storeurl_access allow store_rewrite_list
storeurl_access deny all

refresh_pattern -i \.flv$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp3$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp4$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.swf$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.gif$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpeg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth
refresh_pattern -i \.exe$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth

# 1 year = 525600 mins, 1 month = 10080 mins, 1 day = 1440
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern \.(ico|video-stats) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080
refresh_pattern \.etology\?                                     10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern galleries\.video(\?|sz)                         10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern brazzers\?                                      10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern \.adtology\?                                    10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 10080 20% 10080 ignore-no-cache  ignore-private override-expire ignore-reload ignore-auth   negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google  10080 80% 10080 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth  negative-ttl=10080
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)    10080 80% 10080 override-expire ignore-reload ignore-private  negative-ttl=10080
refresh_pattern ytimg\.com.*\.jpg                                       10080 80% 10080 override-expire ignore-reload
refresh_pattern images\.friendster\.com.*\.(png|gif)                    10080 80% 10080 override-expire ignore-reload
refresh_pattern garena\.com                                             10080 80% 10080 override-expire reload-into-ims
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)          10080 80% 10080 override-expire ignore-reload
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                     10080 80% 10080 ignore-no-cache override-expire override-lastmod
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)    10080 80% 10080 reload-into-ims override-expire ignore-private
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                     10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http:\/\/www.onemanga.com.*\/                          10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080

#images facebook
refresh_pattern -i \.facebook.com.*\.(jpg|png|gif)                      10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)        10080 80% 10080 ignore-reload override-expire ignore-no-cache

#All File
refresh_pattern -i \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)       10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar)  10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)         10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims

refresh_pattern -i (/cgi-bin/|\?)  0  0%  0
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern ^ftp:           10080   95% 10080 override-lastmod reload-into-ims
refresh_pattern .               1440    95% 10080 override-lastmod reload-into-ims

Now create cache dir and logs files , and assign them necessary permissions.

mkdir /cache1
chown proxy:proxy /cache1
mkdir /var/log/squid
chmod 777 /var/log/squid

Now initialize cache dir by

/usr/local/squid/sbin/squid -z

SOTEURL.PL

Now We have to create an important file name storeurl.pl , which is very important and actually it does the
main job to pull video from cache.

mkdir /etc/squid
touch /etc/squid/storeurl.pl
chmod +x /etc/squid/storeurl.pl
nano /etc/squid/storeurl.pl

Now paste the following lines, then Save and exit.

#!/usr/bin/perl
# This script is NOT written or modified by me, I only copy pasted it from the internet.
# It was First originally Writen by chudy_fernandez@yahoo.com
# & Have been modified by various persons over the net to fix/add various functions.
# For Example this ver was modified by member of comstuff.net to satisfy common and dynamic content.
# th30nly @comstuff.net a.k.a invisible_theater ,
# For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
$|=1;
while (<>) {
@X = split;
#       $X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];

#speedtest
if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
print $x . "http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . "\n";

#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . "http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . "\n";

#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
print $x . "http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . "\n";

#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
print $x . "http://www.filesonic.com.SQUIDINTERNAL/" . $1 . "\n";

#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n";

#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/$2\n";

#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
print $x . "http://photos" . $1 . "/" . $2 . $3  . "\n";

#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  ."/". $2 . "\n";

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
print $x . "http://" . $1  . $2 . "\n";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

# youtube fix
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = "";
$alg = "";
$id = "";
$range = "";
if ($p_str =~ m/(itag=[0-9]*)/){$tag = "&".$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = "&".$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = "&".$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = "&".$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $tag . "&" . $alg . "&" . $id . "&" . $range . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
#...spicific servers end here.

#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn.yimg.com//" . $3 . "\n";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/"  . $3 . "\n";

#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";

#generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %"
} elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

# generic http://variable.domain.com/...
} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";

# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

} else {
print $x . $_ . "sucks\n";
}
}

Now start SQUIDServer by

/usr/local/squid/sbin/squid

TIP:
To start SQUID Server in Debug mode, to check any erros, use

/usr/local/squid/sbin/squid -d1

3# TESTING YOUR CACHE HIT

It’s time to hit the ROAD and do performing some tests.

YOUTUBE TEST

Open Youtube and watch any Video. After complete download, Check the same video from another client. You will notice that it download very quickly , you can watch the bar moving fast.
As Shown in the example Below . . .

monitor the Squid access LOG. You will see cache hit TPC_HIT for this video.
As Shown in the example Below . . .

MUSIC DOWNLOAD TEST

Now test any music download. For example Go to
http://www.apniisp.com/songs/indian-movie-songs/ladies-vs-ricky-bahl/690/1.html
As Shown in the example Below . . .

and download any song , after its downloaded, goto 2nd client pc, and download the same song, and monitor the Squid access LOG. You will see cache hit TPC_HIT for this song.

As Shown in the example Below . . .

EXE / PROGRAM  DOWNLOAD TEST

Now test any .exe file download.
Goto http://www.rarlabs.com and download any package. After Download completes, goto 2nd client pc , and download the same file again. and monitor the Squid access LOG. You will see cache hit TPC_HIT for this file.

As Shown in the example Below . . .

SQUID LOGS

If you face “An error occured” in cached videos , see the following

http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/

Regard’s
SYED JAHANZAIB

http://aacable.wordpress.com

Using WEBMIN to create Forward/Reverse ZONE in BIND / UBUNTU

For some people , installing and configuring BIND for local network using CLI can be a typical task. However it can be done easily in few steps via using GUI base controller known as WEBMIN which is a very nice and powergul GUI to manage  almost every service of your Linux box via your browser.

In this article we will discuss following topics.

1# Install WEBMIN (Using 2 different methods)
2# Install BIND (Using WEBMIN)
3# Create Forward Zone for our domain
4# Create Reverse Zones for our domain
5# Test Zones via nslookup

1# Install WEBMIN Using DKPKG

First we need to install webmin. you can either download it from webmin site and install it by issuing following commands.

wget http://prdownloads.sourceforge.net/webadmin/webmin-1.570_all.deb
# then run the following command
dpkg --install webmin_1.570_all.deb

You should now be able to login to Webmin via your browser using the URL

http://192.168.2.1:10000/

OR
https://192.168.2.1:10000/

IF the above method fails to install WEBMIN on your system, Follow this alternate method to install it.

1# Install WEBMIN Using the Webmin APT repository [Alternate Way]

install and update Webmin via APT, edit the /etc/apt/sources.list file by using the following command

nano /etc/apt/sources.list
and add these lines at end.

deb http://download.webmin.com/download/repository sarge contrib
deb http://webmin.mirror.somersettechsolutions.co.uk/repository sarge contrib

Now Save the file and exit.

You should also fetch and install my GPG key with which the repository is signed, with the commands :

cd /root
wget http://www.webmin.com/jcameron-key.asc
apt-key add jcameron-key.asc

You will now be able to install with the commands :

apt-get update
apt-get install webmin

For more information on other distribution installation , please visit http://www.webmin.com

2# INSTALLING BIND via webmin.

After logging in to Webmin ,
Goto SERVERS > BIND DNS Server

It will inform you that bind is not installed, install it via click on “Click Here“
As shown in the image below . . .

Now it will download BIND and install it. Make sure your Linux box is connected with the internet. After installation it will show you something
As shown in the image below . . .

Now that BIND is installed, We have to create Forward and Reverse zones for our new domain. In last we will check it by doing nslookup.

3# Adding FORWARD ZONE for our domain

Goto SERVERS > BIND DNS Server

Now you will see BIND DNS SERVER and various icons.
First we have to create forward zone,
Click on CREATE MASTER ZONE
As shown in the image below . . .

> in Zone Type : Select Forward
> in Domain name / Network , type the name you want to configure your domain with.
As shown in the image below . . .

4# Adding REVERSE ZONE for our domain

Goto SERVERS > BIND DNS Server
Now we have to create Reverse Zone,
Click on CREATE MASTER ZONE
As shown in the image below . . .

Now on your TOP RIGHT, click on APPLY CHANGES.

5# TESTING YOUR NEWLY CREATED ZONE using nslookup

Login to your client pc, open command prompt and issue the following command.
nslookup – 192.168.2.1

now type zaib.com and you should see your Linux box ip in response.
As shown in the image below . . .

All Done. Now you can careate hosts record / MX record etc using GUI base DNS MANAGEMENT CONSOLE

BIND TIP’S N TRICKS

*** Can’t find server name *** ERROR

When you do NSLOOKUP, you see an error something like.
*** Can’t find server name for address 192.168.2.1: Non-existent domain, Default ServeR: UnKnown

To solve this , jsut add reverse lookup entry for your domain.
Goto SERVERS > BIND DNS Server
Open your newly created zone in EXISTING DNS ZONES
As shown in the image below . . .

After opening zaib.com or your zone,
Click on ADDRESS
Here create an address
In Name: zaib.com
In Address: 192.168.2.1
Click on CREATE and then APPLY CHANGES.
Now from you client side, try to do nslookup, and this time you wont see any error.

Create MX record for MAIL Server

Creating an MX Record (Mail Exchange Record).

1) Open your zone , for example zaib.com ,
Click on Mail Server
2) For the name, type in the domain e.g zaib.com
3) For the mail server, type in zaib.com
4) Set the priority to what you like.. 10 is usually default.
5) Click on SAVE and aplpy changes.

Make sure your Linux box have your BIND Server IP at top.

Edit your resolv.conf and set dns name server search order.

# Generated by NetworkManager
search zaib.com
nameserver 127.0.0.1
nameserver 192.168.2.4

Adding Forwarding Entry (For resolving External Hosts)

Goto SERVERS > BIND DNS Server
Click on Forwarding and Transfers
In Servers to forward queries to, type your ISP’s DNS Server IP
As shown in the image below . . .

Now when your client will do query , BIND will first check in zaib.com, if it doesn’t fin any entry, it will forward this query to ISP’S DNS Server.

Have Fun

 SYED JAHANZAIB