Syed Jahanzaib Personnel Blog to Share Knowledge !

January 11, 2012

Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue


UPDATED: 10th MAY, 2012

BAD NEWS:

.

.

Some research update (Someone please confirm and post your comments)
YOUTUBE has split its videos into segments of 1.5 Mb each which is the approximation of the 51 seconds. I am sure YOUTUBE have taken this step to prevent people from caching  entire videos. If you have a video which is 100 Mb large, it will be split into about 55-60 segments.
As of right now, storeurl.pl wont be able to cache it. Currently VIDEOCACHE plugin is

doing full cache of youtube but at higher 400 $ Per Year :)

Following is an update version of youtube caching

http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/

.

http://aacable.wordpress.com/2012/08/13/youtube-caching-with-squid-nginx/

.

.

.

 

What is LUSCA/SQUID ?

LUSCA is an advance version or Fork of  SQUID 2. The Lusca project aims to fix the shortcomings in the Squid-2. It also supports a variety of clustering protocols. By Using it, you can cache some dynamic contents that you previously can’t do with the squid.

For example
#  Video Cachingi.e Youtube / tube etc . . .
#  Windows / Linux Updates / Anti-virus , Anti-Malware i.e. Avira/ Avast / MBAM etc . . .
#  Well known sites i.e. facebook / google / yahoo etch. etch.
#  Download caching mp3′s/mpeg/avi etc . . .

Advantages of Youtube Caching   !!!

In most part of the world, bandwidth is very expensive, therefore it is (in some scenarios) very useful to Cache Youtube videos or any other flash videos, so if one of user downloads video / flash file , why again the same user or other user can’t download the same file from the CACHE, why he sucking the internet pipe for same content again n again?
Peoples on same LAN ,sometimes watch similar videos. If I put some youtube video link on on FACEBOOK, TWITTER or likewise , and all my friend will  watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my lan users / friends.

This is the reason why I wrote this article. I have implemented Ubuntu with LUSCA/ Squid on it and its working great, but to achieve some results you need to have some TB of storage drives in your proxy machine.

Disadvantages of Youtube Caching   !!!

The chances, that another user will watch the same video, is really slim. if I search for something specific on youtube, i get more then hundreds of search results for same video. What is the chance that another user will search for the same thing, and will click on the same link / result? Youtube hosts more than 10 million videos. Which is too much to cache anyway. You need lot of space to cache videos. Also accordingly you will be needing ultra modern fast hardware with tons of RAM to handle such kind of cache giant. anyhow Try it

We will divide this article in following Sections

1#  Installing SQUID / LUSCA in UBUNTU
2#  Setting up SQUID / LUSCA Configuration files
3#  Performing some Tests, testing your Cache HIT
4# Using ZPH TOS to deliver cached contents to clients vai mikrotik at full LAN speed, Bypassing the User Queue for cached contents.

1#  Installing SQUID / LUSCA in UBUNTU

I assume your ubuntu box have 2 interfaces configured, one for LAN and second for WAN. You have internet sharing already configured. Now moving on to LUSCA / SQUID installation.

Download LUSCA source and compile it using,

mkdir /temp
cd /temp
wget http://lusca-cache.googlecode.com/files/LUSCA_HEAD-r14809.tar.gz
tar xzvf LUSCA_HEAD-r14809.tar.gz

Update & Install some tools for ubuntu

sudo apt-get update
sudo apt-get install gcc build-essential sharutils ccze libzip-dev automake1.9
cd LUSCA_HEAD-r14809

Now compile LUSCA with following options

./configure '--prefix=/usr/local/squid' '--enable-removal-policies=heap,lru' '--disable-dependency-tracking' '--disable-arp-acl' '--disable-cache-digests' '--enable-cachemgr-hostname=localhost' '--disable-delay-pools' '--enable-epoll' '--enable-external-acl-helpers=ip_user' '--disable-ident-lookups' '--enable-linux-netfilter' '--disable-referer-log' '--enable-removal-policies=heap,lru' '--disable-snmp' '--disable-ssl' '--enable-storeio=aufs,null,coss' '--disable-useragent-log' '--disable-wccpv2' '--with-aio' '--with-maxfd=1048576' '--with-dl' '--with-pthreads' 'build_alias=i686-redhat-linux-gnu' 'host_alias=i686-redhat-linux-gnu' 'targe_alias=i686-redhat-linux-gnu''--enable-truncate' '--disable-unlinkd' '--with-large-files' '--disable-htcp'
sudo make all
sudo make install

All of Lusca/Squid configuration files can be found at

/usr/local/squid/etc/
and squid executable can be found at
/usr/local/squid/sbin/

Now We will edit squid.conf file to make it customize according to our requirements by . . .

nano /usr/local/squid/etc/squid.conf

Delete all previously lines , and paste the following lines.

Remember following squid.conf is not very neat and clean , you will find many un necessary junk entries in it, but as I didn’t had time to clean them all, so you may clean them as per your targets and goals.

# SQUID 2.7/ LUSCA TEST CONFIG FILE
# Email: aacable@hotmail.com
# Web  : http://aacable.wordpress.com

# PORT and Transparent Option
http_port 8080 transparent
server_http11 on
icp_port 0

# Cache Directory , modify it according to your system.
# but first create directory in root by mkdir /cache1
# and then issue this command  chown proxy:proxy /cache1
# [for ubuntu user is proxy, in Fedora user is SQUID]
# I have set 500 MB for caching reserved just for caching ,
# adjust it according to your need.
# My recommendation is to have one cache_dir per drive. zzz

#store_dir_select_algorithm round-robin
cache_dir aufs /cache1 500 16 256
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA

# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off

# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log

#I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 221.132.112.8
ftp_user anonymous@
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on

#ACL Section
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
http_reply_access allow all
icp_access allow all

#==========================
# Administrative Parameters
#==========================

# I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr aacable@hotmail.com
visible_hostname proxy.aacable.net
unique_hostname aacable@hotmail.com

#=============
# ACCELERATOR
#=============
half_closed_clients off
quick_abort_min 0 KB
quick_abort_max 0 KB
vary_ignore_expire on
reload_into_ims on
log_fqdn off
memory_pools off
cache_swap_low 98
cache_swap_high 99
max_filedescriptors 65536
fqdncache_size 16384
retry_on_error on
offline_mode off
pipeline_prefetch on
# If you want to hide your proxy machine from being detected at various site use following
via off

#============================================
# OPTIONS WHICH AFFECT THE CACHE SIZE / zaib
#============================================
# If you have 4GB memory in Squid box, we will use formula of 1/3
# You can adjust it according to your need. IF squid is taking too much of RAM
# Then decrease it to 128 MB or even less.

cache_mem 8 MB
minimum_object_size 0 bytes
maximum_object_size 100 MB
maximum_object_size_in_memory 128 KB

#============================================================$
# SNMP , if you want to generate graphs for SQUID via MRTG
#============================================================$
#acl snmppublic snmp_community gl
#snmp_port 3401
#snmp_access allow snmppublic all
#snmp_access allow all

#============================================================
# ZPH , To enable cache content to be delivered at full lan speed,
# To bypass the queue at MT.
#============================================================
tcp_outgoing_tos 0x30 all
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

# Caching Youtube
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp)
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.apniisp\.com\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com
acl videocache_deny_dom  dstdomain .download.youporn.com .static.blip.tv
acl dontrewrite url_regex redbot\.org \.php
acl getmethod method GET

storeurl_access deny dontrewrite
storeurl_access deny !getmethod
storeurl_access deny videocache_deny_dom
storeurl_access allow videocache_allow_url
storeurl_access allow videocache_allow_dom
storeurl_access deny all

storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 100

acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \
storeurl_access allow store_rewrite_list
storeurl_access deny all

refresh_pattern -i \.flv$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp3$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.mp4$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.swf$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.gif$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private ignore-auth
refresh_pattern -i \.jpeg$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth
refresh_pattern -i \.exe$ 10080 80% 10080  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache  ignore-private  ignore-auth

# 1 year = 525600 mins, 1 month = 10080 mins, 1 day = 1440
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?)    10080 80% 10080 ignore-no-cache  ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern \.(ico|video-stats) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080
refresh_pattern \.etology\?                                     10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern galleries\.video(\?|sz)                         10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern brazzers\?                                      10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern \.adtology\?                                    10080 80% 10080 override-expire ignore-reload ignore-no-cache
refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 10080 20% 10080 ignore-no-cache  ignore-private override-expire ignore-reload ignore-auth   negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google  10080 80% 10080 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth  negative-ttl=10080
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)    10080 80% 10080 override-expire ignore-reload ignore-private  negative-ttl=10080
refresh_pattern ytimg\.com.*\.jpg                                       10080 80% 10080 override-expire ignore-reload
refresh_pattern images\.friendster\.com.*\.(png|gif)                    10080 80% 10080 override-expire ignore-reload
refresh_pattern garena\.com                                             10080 80% 10080 override-expire reload-into-ims
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)          10080 80% 10080 override-expire ignore-reload
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                     10080 80% 10080 ignore-no-cache override-expire override-lastmod
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)    10080 80% 10080 reload-into-ims override-expire ignore-private
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                     10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http:\/\/www.onemanga.com.*\/                          10080 80% 10080 reload-into-ims ignore-no-cache  ignore-reload override-expire
refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 10080 80% 10080 override-expire ignore-reload ignore-no-cache  ignore-private ignore-auth override-lastmod  negative-ttl=10080

#images facebook
refresh_pattern -i \.facebook.com.*\.(jpg|png|gif)                      10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png)                 10080 80% 10080 ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)        10080 80% 10080 ignore-reload override-expire ignore-no-cache

#All File
refresh_pattern -i \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)       10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar)  10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)         10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 10080 80% 10080 ignore-no-cache   override-expire override-lastmod reload-into-ims

refresh_pattern -i (/cgi-bin/|\?)  0  0%  0
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern ^ftp:           10080   95% 10080 override-lastmod reload-into-ims
refresh_pattern .               1440    95% 10080 override-lastmod reload-into-ims

Now create cache dir and logs files , and assign them necessary permissions.

mkdir /cache1
chown proxy:proxy /cache1
mkdir /var/log/squid
chmod 777 /var/log/squid

Now initialize cache dir by

/usr/local/squid/sbin/squid -z

SOTEURL.PL

Now We have to create an important file name storeurl.pl , which is very important and actually it does the
main job to pull video from cache.

mkdir /etc/squid
touch /etc/squid/storeurl.pl
chmod +x /etc/squid/storeurl.pl
nano /etc/squid/storeurl.pl

Now paste the following lines, then Save and exit.

#!/usr/bin/perl
# This script is NOT written or modified by me, I only copy pasted it from the internet.
# It was First originally Writen by chudy_fernandez@yahoo.com
# & Have been modified by various persons over the net to fix/add various functions.
# For Example this ver was modified by member of comstuff.net to satisfy common and dynamic content.
# th30nly @comstuff.net a.k.a invisible_theater ,
# For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
$|=1;
while (<>) {
@X = split;
#       $X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];

#speedtest
if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
print $x . "http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . "\n";

#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . "http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . "\n";

#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
print $x . "http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . "\n";

#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
print $x . "http://www.filesonic.com.SQUIDINTERNAL/" . $1 . "\n";

#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n";

#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
print $x . "http://www.4shared.com.SQUIDINTERNAL/$2\n";

#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
print $x . "http://photos" . $1 . "/" . $2 . $3  . "\n";

#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  ."/". $2 . "\n";

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
print $x . "http://" . $1  . $2 . "\n";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

# youtube fix
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = "";
$alg = "";
$id = "";
$range = "";
if ($p_str =~ m/(itag=[0-9]*)/){$tag = "&".$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = "&".$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = "&".$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = "&".$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $tag . "&" . $alg . "&" . $id . "&" . $range . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
#...spicific servers end here.

#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn.yimg.com//" . $3 . "\n";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/"  . $3 . "\n";

#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";

#generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %"
} elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

# generic http://variable.domain.com/...
} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";

# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2  . "\n";

} else {
print $x . $_ . "sucks\n";
}
}

Now start SQUIDServer by

/usr/local/squid/sbin/squid

TIP:
To start SQUID Server in Debug mode, to check any erros, use

/usr/local/squid/sbin/squid -d1

3# TESTING YOUR CACHE HIT

It’s time to hit the ROAD and do performing some tests.

YOUTUBE TEST

Open Youtube and watch any Video. After complete download, Check the same video from another client. You will notice that it download very quickly , you can watch the bar moving fast.
As Shown in the example Below . . .

monitor the Squid access LOG. You will see cache hit TPC_HIT for this video.
As Shown in the example Below . . .

MUSIC DOWNLOAD TEST

Now test any music download. For example Go to
http://www.apniisp.com/songs/indian-movie-songs/ladies-vs-ricky-bahl/690/1.html
As Shown in the example Below . . .

and download any song , after its downloaded, goto 2nd client pc, and download the same song, and monitor the Squid access LOG. You will see cache hit TPC_HIT for this song.

As Shown in the example Below . . .

EXE / PROGRAM  DOWNLOAD TEST

Now test any .exe file download.
Goto http://www.rarlabs.com and download any package. After Download completes, goto 2nd client pc , and download the same file again. and monitor the Squid access LOG. You will see cache hit TPC_HIT for this file.

As Shown in the example Below . . .

SQUID LOGS

If you face “An error occured” in cached videos , see the following

http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/

Regard’s
SYED JAHANZAIB
http://aacable.wordpress.com

About these ads

281 Comments »

  1. ur amazing gr8 , thx for everything …

    Comment by Nori — January 11, 2012 @ 1:39 PM

    • Hi,suddenly lusca stops responding but still running.i have to restart in-order to work again.
      please help

      Comment by Tarek El Ali — March 16, 2012 @ 6:42 PM

  2. awesome!! great sharing

    Comment by faizan — January 11, 2012 @ 3:49 PM

  3. Nice project zaib bhai….

    Comment by adeel — January 11, 2012 @ 3:52 PM

  4. spectacular! thanks man

    Comment by benk — January 12, 2012 @ 3:05 PM

  5. assalamualaikum
    hi brother I’ll try your tutorial, thank’s ;-)

    Comment by bambang — January 13, 2012 @ 9:33 AM

  6. plz yar aap mary number per kar karain aap bohat sari baatian kar ni hai

    Comment by ashraf — January 14, 2012 @ 12:23 AM

  7. number hai 3217701039

    Comment by ashraf — January 14, 2012 @ 12:24 AM

  8. nice post mr. zaib… i have squid 3.1 on slitaz (small linux distro with 30MB iso and serve repository for squid 3.1). is that possible if i used your lusca conf to my quid? or i must compile lusca source on slitaz by my self? thx…

    Comment by mbahsimak — January 18, 2012 @ 12:13 PM

  9. Everything works fine, however, I found 1 thing … May be its a squid bug…I am not sure…But files downloaded with browser download manager is cached, and files downloaded with Internet Download Manager is not cached, even though I am using transparent proxy for both. Please check this in your server, and see if the same happens for u as well….

    Comment by Saiful — January 18, 2012 @ 10:44 PM

    • I have observed this behavior.
      But currently I don’t have any solution for this.

      Comment by Syed Jahanzaib / Pinochio~:) — January 18, 2012 @ 11:14 PM

      • No browser addon issue.
        Its not CDN in this case, I am trying to download a simple static mp3 file. Its actually because IDM uses multiple parts downloads … and therefore its not caching. But is there any solution other than that?

        Comment by Saiful — January 19, 2012 @ 2:10 AM

      • I have observed this behavior.
        But currently I don’t have any solution for this.

        Comment by Syed Jahanzaib / Pinochio~:) — January 19, 2012 @ 11:17 AM

  10. Nice job! only thing which I dont se is file in /etc/init.d …
    So how to autostart lusca ?

    Comment by Damir — January 25, 2012 @ 12:00 AM

    • Various Solutions:

      # You can have some caching functionality even on simple squid 2.7 (even with default installation in ubuntu by apt-get install squid), You don’t need hard coded LUSCA to achieve youtube or dynamic contents, Use the following guide.
      http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/

      # For simply running LUSCA after the computer started but before a user logs in, you can simply edit the script /etc/rc.local which is meant to solve exactly this task.
      For example: use this to start squid on start-up
      echo “/usr/local/squid/sbin/squid -D” >> /etc/rc.local

      # Add it in your CRON cron daemon, one of the predefined cron time hooks is @reboot, which naturally runs when the system starts/reboot. Run crontab -e to edit your crontab file, and add a line:
      @reboot /your/command/here

      # Add LUSCA as a service , then you can control it as normal squid service. There are lot of guides available for this purpose. hit the road, i mean google it :)

      Comment by Syed Jahanzaib / Pinochio~:) — January 25, 2012 @ 11:08 AM

      • Thanks ! I make cron job.
        anyway I was notice one error for example when waching direct video like this everythin is fine :

        but when try to wach same video with apendix like this:

        I have error …
        This is probably related to storeurl.pl …. do you have some update ?

        Comment by Damir — January 29, 2012 @ 1:18 AM

      • I also faced this issue.
        I googled and found out that This is some what related to TTL. Google it , and if you found some working solution, do let me know :)

        Comment by Syed Jahanzaib / Pinochio~:) — January 29, 2012 @ 9:46 AM

  11. This is really a great explanation
    But I have a question
    Why take squid speed Internet, despite all of the client take only half the speed of the Internet

    thanks for help me

    Comment by malak — January 25, 2012 @ 2:18 AM

  12. sorry Syed Jahanzaib

    iam not good in english

    i have mikrotik loadbalance —— squid —— mikrotik hotspot
    and i have limted download in 512K bytes and iam only in my network
    when i download and take 512K speed squid have all triffice

    see this PIC
    http://www.m5zn.com/uploads2/2012/1/25/photo/012512090112eqizr1ccy79442j2dogjew.png

    this mean the squid server get all triffice :(

    can you help me

    Comment by malak — January 25, 2012 @ 10:02 PM

    • The Mikrotik is showing that wlan1 pppoe client is taking 4 mbps bandwidth. but the IDM is showing its taking only 512 kbps.

      Check your queuing section.

      Comment by Syed Jahanzaib / Pinochio~:) — January 26, 2012 @ 11:01 AM

  13. Hey thanks for the tutorial, I have not really make the squid to connect to the mikrotik could guide me to make the connection you make would be greatly appreciated

    Comment by esteban — January 26, 2012 @ 9:11 AM

  14. Dear Syed Jahanzaib,
    Do configuraton files need to be changed if iam using squid through one nic ?

    Comment by Mohamad Hassan — January 26, 2012 @ 9:12 AM

  15. Dear Syed Jahanzaib,

    lam very very very sorry

    Is it normal squid take all internet speed ?????? when user download any file and user have limted speed ????

    i see when one only in my network and do download he take 512K only but in the same time squid take 4M

    i want squid take same speed of client

    explanation

    i want to do this

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan , wan have speed = 512 K
    in Load balace Local = 512
    ————————————————————–

    but when i setup squid

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan = 512K , wan have speed = 4M ???
    in Load balace Local = 4M ???

    squid download the file in all speed internet it`s make Browsing soooooo slowly

    see this Pic
    http://www.m5zn.com/uploads2/2012/1/26/photo/012612020158bqjbygj2rjke4frhygvzk.png

    this pic Immediately when i download squid take allllllllll speed but i have only 512K???

    thanks for help

    Comment by malak — January 26, 2012 @ 3:33 PM

  16. Please mikrotik setting…..thanks

    Comment by slams — January 26, 2012 @ 6:41 PM

  17. Dear Syed Jahanzaib,

    lam very very very sorry

    Is it normal squid take all internet speed ?????? when user download any file and user have limted speed ????

    i see when one only in my network and do download he take 512K only but in the same time squid take 4M

    i want squid take same speed of client

    explanation

    i want to do this

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan , wan have speed = 512 K
    in Load balace Local = 512
    ————————————————————–

    but when i setup squid

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan = 512K , wan have speed = 4M ???
    in Load balace Local = 4M ???

    squid download the file in all speed internet it`s make Browsing soooooo slowly

    see this Pic
    http://www.m5zn.com/uploads2/2012/1/26/photo/012612020158bqjbygj2rjke4frhygvzk.png

    this pic Immediately when i download squid take allllllllll speed but i have only 512K???

    thanks for help

    plzzzzz help mee

    Comment by malak — January 28, 2012 @ 2:48 AM

  18. Dear Sayed,
    I’m using squid 2.7 with storeurl.pl
    its not caching youtube and the dynamic pages its giving this error :

    root@proxy:~# squid -z
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).*’: ignore-auth  
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^http:\/\/images|pics|thumbs[0-9]\.                    ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^http:\/\/www.onemanga.com.*\/                         ‘: ignore-no-cache 
    2012/02/01 06:12:08| parseConfigFile: squid.conf:209 unrecognized: ‘refresh_pattern ‘
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)      ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar) ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)        ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob)’: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv)’: ignore-no-cache 

    please help me, i appreciate it
    Best regards . Mohamad

    Comment by Mohamad Hassan — February 1, 2012 @ 9:14 AM

  19. Search and remove following directive
    ignore-no-cache
    ignore-auth

    squid 2.7 dont support few refresh pattern.
    LUSCA is better in some cases.
    any how try it

    Comment by Syed Jahanzaib / Pinochio~:) — February 1, 2012 @ 12:55 PM

  20. Dear Syed Jahanzaib,

    I just tried caching using SQUID / Lusca, as described above, but the videos are not even playing for the first time!
    Could you help me fix this?

    Best regards

    Comment by laziz — February 7, 2012 @ 6:58 PM

    • If not playing even for the first time, there could be problem with the storerul.pl
      try to start lusca/squid without storeurl.pl and check if it works.

      Comment by Syed Jahanzaib / Pinochio~:) — February 7, 2012 @ 11:17 PM

      • I was also notice this behavior … after few refeshes … somethimes is open … maybe is smarter to contact this original author of storeurl to ask him for some update ?

        Comment by damir — February 8, 2012 @ 1:35 AM

      • There is no absolute fix for this problem at a moment, At least I am not aware of it :)
        If you find one, do let me know.

        and yes contacting author of storeurl.pl is the best method to get the authentic solution or reply.

        Comment by Syed Jahanzaib / Pinochio~:) — February 8, 2012 @ 9:02 AM

  21. Well I was sent email to autor … without sucess … they was working now with videocashe and sugested me to swich to videocache which is to much expensive for my requierments cca 300 $ :(

    Comment by damir — February 9, 2012 @ 1:17 AM

    • Yes that’s the problem that videocache is paid solution and costly one too, also after 6 months, you have to pay half of the amount for the renewal or updated version :s

      Comment by Syed Jahanzaib / Pinochio~:) — February 9, 2012 @ 8:56 AM

  22. sir video cache ka lya pay karni para gi kya

    Comment by sheeraz — February 9, 2012 @ 9:55 AM

  23. Jahanzaib bahi

    Lusca install kar nay kay liyeh kiya phelay say muj ko squid install cahiyeh hoga ya nahii

    Comment by Asif Lai — February 15, 2012 @ 9:17 PM

    • You don’t need SQUID to install LUSCA
      LUSCA works like SQUID, infact it is squid’s more advance version you can say.

      Comment by Syed Jahanzaib / Pinochio~:) — February 15, 2012 @ 10:51 PM

      • Hi do you have a version for windows xp???? or can you make a version of squid/lusca for windows xp thx. and more power….

        Comment by tapitz — February 16, 2012 @ 12:47 PM

      • There are win32 base squid version available on internet.
        But You can’t get real performance from windows base SQUID.
        So its better if you do it in Linux.
        Ubuntu is good for beginners and starters.

        Comment by Syed Jahanzaib / Pinochio~:) — February 17, 2012 @ 8:55 AM

  24. jahanzeb bhai

    main nay app kay way try kiya magar jab main youtube ka koi bhi clip play karta hoon to kuch tume baad woh error show kar deta hain
    aisa kiyon…

    Comment by Asif Lai — February 16, 2012 @ 4:43 PM

  25. jahanzeb bahi

    eik aur baat jab mian cache server ko restart karta hoon to sari config remove hojatii is ka koi hall batain

    Comment by Asif Lai — February 16, 2012 @ 4:55 PM

  26. jahanzaib bhai i have dual core CPU with 2.50GHz processor. 4 GB RAM and 1 TB harddisk. to kya main easily use kr skta hun jo ap nay caching btaei hai ?

    Comment by SHAFQAT FARHAN — February 26, 2012 @ 9:46 PM

  27. thank your for this useful tutorial syed, im just wondering how you could install cache manager and/or webmin

    Comment by cervelo — February 29, 2012 @ 3:28 PM

  28. Hi Sir can you make a delaypools options… thx and more power….

    Comment by tapitz07 — March 5, 2012 @ 3:04 PM

  29. hi dear
    hope that you are doing fine

    thanks for this good job , but there is a problem with youtube.com videos

    please see this picture and let me know how to fix it

    http://www.m5zn.com/uploads2/2012/3/6/photo/030612010328iy6z9uw8s5wbnmfna1c50i.jpg

    waiting for the replay ASAP

    Comment by hamid — March 6, 2012 @ 2:43 PM

  30. Hi,

    i’m very thankful for this deep explanation.

    I managed to install it and it is working fine, but wanted to enable SNMP, so I can draw some graphs.
    First, uncommented the lines in the squid.conf that were # before, but it give error when trying to start it, then I realized that it is compiled with disable-snmp, so i made ./configure –enable-snmp but when tried to compile again I got errors

    /temp/LUSCA_HEAD-r14809/src/snmp_core.c:323: undefined reference to `theOutSnmpConnection’
    collect2: ld returned 1 exit status
    make[3]: *** [squid] Error 1
    make[3]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make[2]: *** [all-recursive] Error 1
    make[2]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make[1]: *** [all] Error 2
    make[1]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make: *** [all-recursive] Error 1

    Have you tried to compile it with enable-snmp?

    Comment by x-man — March 10, 2012 @ 1:54 PM

  31. range_offset_limit 512 KB
    can you define it please and how can it be good for improving the performance??

    Comment by mohamed koubar — March 11, 2012 @ 7:05 AM

  32. Can you help me… Im configure Lucas Squid , almost all is ok , my problem is with the use ACL , i need use filter arp , filter by hours , filter by contents , in this case i create a files with all mac address of the lan , and do the same for the other files , but i dont have nothing answer , with mac in the file or within a can navagate, in other version of squid all that is ok, im compile de Lucas package with arp support.What is my mistake

    Comment by Elfoman — March 13, 2012 @ 8:32 AM

  33. Hello Sir,

    I have try your tutorial but .. i have some faced problem below
    10.201.21.10 – - [14/Mar/2012:10:27:44 +0700] “GET http://cdn.api.twitter.com/1/urls/count.json? HTTP/1.1″ 200 513 TCP_MISS:DIRECT
    10.201.21.10 – - [14/Mar/2012:10:27:46 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:27:47 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:27:51 +0700] “POST http://ubuntu-indonesia.com/forums/ubbthreads.php HTTP/ 1.1″ 200 655 TCP_MISS:DIRECT
    10.201.21.10 – - [14/Mar/2012:10:27:51 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:27:53 +0700] “GET http://0-38.channel.facebook.com/pull? HTTP/1.1″ 200 159 1 TCP_MISS:DIRECT
    10.201.21.10 – - [14/Mar/2012:10:27:56 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:27:56 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:27:58 +0700] “GET http://0-38.channel.facebook.com/pull? HTTP/1.1″ 200 159 2 TCP_MISS:DIRECT
    10.201.21.10 – - [14/Mar/2012:10:27:59 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:28:00 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE
    10.201.21.10 – - [14/Mar/2012:10:28:02 +0700] “HEAD error:invalid-request HTTP/0.0″ 400 236 TCP_DENIED:NONE

    But proxy is running and can cache would you like to help me please

    regards

    Kamb33ng

    Comment by kamb33ng — March 14, 2012 @ 8:37 AM

  34. Thanks for such information…. Greet

    Comment by mp4 videos — March 19, 2012 @ 2:37 AM

  35. thank you very much

    Comment by sara — March 22, 2012 @ 6:12 PM

  36. Too few store_rewriter processes are running
    FATAL: The store_rewriter helpers are crashing too rapidly, need help!

    whenever i start
    /usr/local/squid/sbin/squid -d1

    Comment by aws — March 27, 2012 @ 5:37 PM

    • This usually happens when there is a typo / paste mistake in storeurl.pl

      Try to use newer version and make sure you copy the right source code.

      Comment by Syed Jahanzaib / Pinochio~:) — March 27, 2012 @ 10:28 PM

      • i have same problem too. wriiten like this :
        WARNING: store_rewriter #1 (FD 7) exited
        WARNING: store_rewriter #2 (FD 8) exited
        WARNING: store_rewriter #3 (FD 9) exited
        WARNING: store_rewriter #10 (FD 10) exited
        Too few store_rewriter processes are running
        FATAL: The store_rewriter helpers are crashing too rapidly, need help!

        I use Ubuntu server 11.04. And i copy paster the squid.conf and storeurl.pl files using view source and then paste it to a file in notepad in windows. Then i copy them to the directory as written above.

        Is that causing this problem ? Please help

        Comment by djynny — June 30, 2012 @ 10:40 AM

    • I think i found the problem, but still don’t know how to solve it. If you open the /var/log/squid/cache.log you will find the problem.
      I got this error :
      helperOpenServer: Starting 7 ‘storeurl.pl’ processes
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      I think it related with file permission. Because i am a linux newbie, I hope someone could help.

      Syed, I hope when you are not busy you can help us..

      Best Regards,

      Djemmy

      Comment by djynny — July 13, 2012 @ 12:50 PM

      • The problem is solved. After try to reinstalling the ubuntu server 5 times i found that the problem because of i copy paste the squid.conf and storeurl.pl via notepad in windows. Although I save as it as all files and put the file extension with .conf and .pl it seem that ubuntu didn’t recognize the .pl file that created by notepad. That was my own mistake…..
        Syed thanks for all the share knowledge here.

        Cheers,
        Djemmy

        Comment by djynny — July 16, 2012 @ 10:15 AM

      • You are welcome !

        Comment by Syed Jahanzaib / Pinochio~:) — July 16, 2012 @ 11:29 AM

  37. Thanks for the post…….

    I’m using Fedora for forwarding my mikrotik port 80 request of browsing…. Wil I use all these stuff in Fedora or I have to use debain for this especially….. I thought zph methos not works in fedora

    Comment by kashif khan — March 27, 2012 @ 10:02 PM

    • Most of my posts related to Linux are Ubuntu base, because I found Ubuntu most easy to use Linux flavor. You or other may found any other flavor more suitable for various reasons, As I have my own :)
      By default FEDORA installs SQUID 3 which deoesn’t support ZPH by default, you have to compile Squid 2.7 on Fedora in order to use ZPH or Use LUSCA on FED , as LUSCA supports ZPH well.

      On the other hand UBUNTU installs squid 2.7 which supports ZPH so no extra work is required.

      It’s all about personnel preference, in past, FEDORA was my first choice, but from past 2 year,s UBUNTU have become First :) but don’t worry, You can achieve your goal by using any flavor of Linux, doesn’t matter its Ubuntu , Fedora, Centos , FreeBSD , or any OS with Linux as its base OS /. Kernel , it will work :)

      Comment by Syed Jahanzaib / Pinochio~:) — March 27, 2012 @ 10:19 PM

      • Thanks Syed Jhanzaib.. I install Ubuntu 10.4 and Everything works fine…. I ‘m using Mikrotik OS and zph working fine tooo………. Thanks a lot

        Comment by kashif — April 21, 2012 @ 11:49 PM

      • I am glad it helped you.

        Comment by Syed Jahanzaib / Pinochio~:) — April 22, 2012 @ 12:01 AM

  38. i’m using your source code !!!!

    and what’s new version you are talking about ????

    thanks for your reply

    Comment by aws — March 28, 2012 @ 3:52 AM

  39. Buenas noches, lo instale en mi equipo portátil para realizar varias pruebas pero soy novato, como hago para ver si esta funcionando y mi equipo tiene dirección ip asignada por dhcp, y también como detengo squid?

    Comment by tonyvzla — April 1, 2012 @ 5:17 AM

  40. hi im ask what is different lusca fmi and lusca head? so it lusca works on ubuntu server 10.10 64bit?

    Comment by Rinocomp Ajust — April 2, 2012 @ 4:33 PM

  41. i am new to the forum is very interesting your project and followed it step by step but when redirected to Lusca / squid will not let me navigate PORSIACASO mikrotik use which may be the error I would greatly appreciate it

    Comment by nickeliott — April 6, 2012 @ 9:41 AM

  42. what is PORSIACASO?

    Comment by Syed Jahanzaib / Pinochio~:) — April 9, 2012 @ 4:37 PM

  43. i have not seen yet somone adapt Content-Type: video/webm youtube using it now wide on most of the new video

    i hoop you cancame up with solution

    Comment by joe — April 11, 2012 @ 4:46 AM

  44. cache webm seen a little dificult.. can you help us?? by the way.. thanks for the solution for youtube.. works great!

    Comment by Daniel Echizen — April 13, 2012 @ 6:30 AM

  45. some news of webm .. need to test more but… for anyone to want.. try this.. set in squid.conf range_offset_limit -1, also change your refresh pattern to ensure that squid will cache webm file format

    Comment by Daniel Echizen — April 13, 2012 @ 6:52 AM

    • Have you tried it ?

      Comment by Syed Jahanzaib / Pinochio~:) — April 13, 2012 @ 9:20 AM

      • yeap.. working like a charm .. seens like video/webm keep a NaN range

        Comment by Daniel Echizen — April 13, 2012 @ 8:03 PM

      • just a tip.. to work correctly try to avoid caching redirect text/html content

        Comment by Daniel Echizen — April 13, 2012 @ 8:06 PM

      • Hmmm what impact you have seen for not to cache text/html contents ?

        Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 8:00 PM

      • redirect content from webm has no Cache-Control:no-cache so instead of webm video you’ll cache the redirect file which is empty. Which will also loop back to redirect content.

        Comment by Daniel Echizen — April 14, 2012 @ 10:07 PM

      • news.. the fix in here is exactly what im says.. http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube tested direct on source and working great! dont need to worry about text/html.. just patch with the fix in client_side.c

        Comment by Daniel Echizen — April 14, 2012 @ 10:15 PM

  46. jahanzaib bhai ubuntu main lan and wan kesay configure krtay hain ??

    Comment by Shafqat Farhan — April 13, 2012 @ 3:13 PM

  47. im having problems with sourceforge.net … when i try to redownload a file in another browser, the counter restarts and the download do not start properly, anyone hav this issue too?

    Comment by Daniel Echizen — April 13, 2012 @ 9:34 PM

    • Nops. However going through same WAN IP every time, it can cause access deny sometime.

      Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 7:59 PM

      • will see about that! thanks for the tip.. by the way.. did you get working 4shared preview files? i only get working download files.. but preview.mp3 dont..

        Comment by Daniel Echizen — April 14, 2012 @ 10:08 PM

  48. jahanzaib bhai mujh say ubuntu mainn lan and wan configure nhi ho raha . will u plz tell me .

    Comment by Shafqat Farhan — April 15, 2012 @ 12:34 PM

  49. Everything is working fine. but video cach nhi ho rahi . Kya masla ho skta hai?? youtube, metacafe etc etc. koi b cach nhi ho rahi . baqi softwares, mp3 files etc cach ho rahi hain

    Comment by Shafqat Farhan — April 16, 2012 @ 6:43 PM

  50. one big problem in youtube.. if you get a cached video and try to change dpi the video doesnt play anymore.. only the original dpi.. anyone hav a fix for this?

    Comment by Daniel Echizen — April 16, 2012 @ 9:52 PM

    • ps.: the problem is with the webm content.. range_offset to -1 cause this issue

      Comment by Daniel Echizen — April 16, 2012 @ 10:24 PM

      • These are the bugs with the storeurl.pl
        or you can say ” It comes with the Package ” ;)

        Comment by Syed Jahanzaib / Pinochio~:) — April 17, 2012 @ 8:45 AM

      • fixed forcing cache with quick_abort_min -1 in squid.conf.. for anyone that hav the same issue!

        Comment by Daniel Echizen — April 17, 2012 @ 10:00 AM

      • forcing cache from beginning* with quick_abort_min -1

        Comment by Daniel Echizen — April 17, 2012 @ 10:02 AM

  51. jahanzaib bhai u did’nt replye me

    Comment by Shafqat Farhan — April 17, 2012 @ 6:52 PM

  52. My cache is working now. But yay btaein k yay small videos ki caching ho jati hai but bari videos ki nhi hoti . Means jo videos 30 ya phr 35 seconds ki hon on ki to cache ho jati hai but jo 1 minute say above hun on ki caching nhi hoti . same with softwares . jo softwares approx 15 MB say baray hon on ki caching nhi hoti . Help Plz

    Comment by Shafqat Farhan — April 17, 2012 @ 9:11 PM

    • Check File Size limitation in squid.

      Comment by Syed Jahanzaib / Pinochio~:) — April 18, 2012 @ 12:42 PM

      • From where i can check the file size limitaion ? I mean in which variable in squid code ?

        Comment by Shafqat Farhan — April 18, 2012 @ 2:58 PM

      • minimum_object_size and maximum_object_size in squid.conf

        Comment by Daniel Echizen — April 19, 2012 @ 2:32 AM

  53. i have already tried this already minimum_object_size 0 bytes and maximum_object_size is 100 MB . but still i get the problem.

    Comment by Shafqat Farhan — April 19, 2012 @ 9:33 AM

  54. jahanzaib bhai waiting for your reply

    Comment by Shafqat Farhan — April 19, 2012 @ 6:14 PM

    • Hi, If you copy the content to you clipboard from above tutorial, you need to make sure that you remove all the %nbsp”. The code will not read correctly. I had the same problem at first.

      Comment by A.J. Hart — May 1, 2012 @ 7:31 PM

  55. HI, i have follow this tutorial but i have this error “root@debian:/home/debian# /usr/local/squid/sbin/squid -d1
    2012/04/19 01:46:16| ACL name ‘videocache_deny_dom’ not defined!
    FATAL: Bungled squid.conf line 160: storeurl_access deny videocache_deny_dom
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.”

    Comment by bmwfrs — April 22, 2012 @ 6:28 AM

    • remvoe all lines which have videocache_deny_dom

      Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 4:27 PM

    • Hi, If you copy the content to you clipboard from above tutorial, you need to make sure that you remove all the %nbsp”. The code will not read correctly. I had the same problem at first. Sorry about the previous comment, it was ment for bmwfrs

      Comment by A.J. Hart — May 1, 2012 @ 7:34 PM

  56. Is there any way to graph the squid cache perfomance???

    Comment by kashif khan — April 25, 2012 @ 3:57 PM

    • Yes I worked on MRTG graphs in depth, I made graphs for squid cache hit also, but unfortunately, I did this long time ago and I don’t have its cfg files. but its simple. Make sure your squid is snmp enabled, then by querying squid snmp oid, you can create graph for every instance of it. I will try to search for it on my local repositories.

      Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 4:37 PM

      • jahanzaib bhai ap nay btaya nhi k squid large file kesay cache krta hai ?

        Comment by Shafqat Farhan — April 26, 2012 @ 9:27 AM

  57. i have done the settings but IDM page k ilaawa kuch kuch pages ki caching ho rahi hay aur wo bhi all objects ko cache nahi karta kuch DIRECT khultay hien aur kuch HIT hotay hien…. Youtube se bhi yehee problem hay, kuch aik videos k ilaawa kisi video ki caching nahi ho rahi…. please jahanzaib bhai iska solution bataaein kia problem hay… sab commands without error install hui hay kisi mein koi error nahi aaya tha…

    Looking for you reply Sir….

    Comment by Ijaz Ahmed — April 27, 2012 @ 2:47 PM

    • Most IDM downloads will not fit into cache due to multipart chunks downloads.
      I haven’t tested squid youtube caching since few weeks. I will test it again and will let you know the results.

      Comment by Syed Jahanzaib / Pinochio~:) — April 28, 2012 @ 4:22 PM

      • sir i need to make cache of youtube and facebook. please tell me the method… and this script is not caching all the site I mean it saves some objects of a site and dont save many objects…

        Comment by Ijaz Ahmed — April 28, 2012 @ 5:27 PM

  58. from about 2 month ago youtube chache always tcp_miss, anyone have the solution?

    Comment by sevensvr — May 6, 2012 @ 7:07 AM

  59. Hi Syed,
    congrats for the excellent work!
    One question if I may…
    If I want to use an apache as a proxy for utube..and squid as cache..
    How can I tell squid to always try to find cached content first before trying to proxy?
    These rules are enough for apache I guess(I am only interested for embedded videos):
    RewriteRule ^/get_video_info(.*)$ http://www.youtube.com%{REQUEST_URI} [P,L]
    RewriteRule ^(.*)$ http://www.youtube.com/embed%{REQUEST_URI} [P,L]

    -Thanks

    Comment by Thanassis Zakopoulos — May 7, 2012 @ 3:16 AM

    • I have no idea for apache as youtube caching server.

      Youtube caching can be done using etiher storeurl.pl file, or python base scripts (which uses apache for youtube cache distribution) , Also this method is used in VIDEOCACHE plugin which cache youtube using pytin scripts with apache which is a paid addon.

      Comment by Syed Jahanzaib / Pinochio~:) — May 8, 2012 @ 3:18 PM

      • Hello Syed,

        Do you have any fix about the “51% youtube caching”, when you play a certain video in youtube then cached it, the next time it will be played is delayed or stucks at 51% then buffers again, but when i checked that certain video, its been already cached, the problem here is the delay.

        Thank you :)

        Comment by Incognito79 — May 12, 2012 @ 9:49 AM

      • No solution for it yet.

        Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2012 @ 8:57 AM

      • this is not a problem at all.. look at access.log.. youtube work with range files .. so you’ll cache alot of parts of the same video, you’ll buffer another part when playing part is about 90%, thats why u hav the delay on preload..but if you got hit on access.log its all fine.

        Comment by Daniel Echizen — May 14, 2012 @ 11:33 AM

  60. i success make cache youtube .. :D

    Comment by fahri — May 12, 2012 @ 1:18 PM

    • Syed,

      Hi again, do you have the new solution for the youtube partial caching problem?…

      thank you

      Comment by Incognito79 — May 15, 2012 @ 10:49 AM

  61. no hard work cache youtube :D

    Comment by fahri — May 12, 2012 @ 1:21 PM

  62. I did all sir. per squid server ka status check karta hu to squid stop/waiting py hi hota hy…… jub ky usy kafi dafa start kiya hy

    Comment by Muhammad Abdullah Butt — May 12, 2012 @ 8:07 PM

  63. Problem Solved! main ny /var/log/squid/store.log-access.log-cache.log ko chmod 777 ki to squid server start ho gai. eb baqi testing start karny laga hu.

    Comment by Muhammad Abdullah Butt — May 13, 2012 @ 1:01 AM

  64. Dear Sir,

    @karthik : can u please help me, if we check the log it is blank ” tail -f /var/log/squid/access.log | grep o-o”

    root@karthik:/var/log/squid# du -s -h *
    0 access.log
    12K cache.log
    0 store.log

    in microtik dns ip given same has squid ip.

    Comment by karthik — May 17, 2012 @ 7:29 PM

  65. ~ # tail -f /var/log/squid/access.log | grep o-o
    127.0.0.1 – - [22/May/2012:13:14:06 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/generate_204? HTTP/1.1″ 204 217 TCP_MISS:DIRECT
    127.0.0.1 – - [22/May/2012:13:14:06 +0700] “GET http://o-o.preferred.excelcom-cgk1.v17.lscache3.c.youtube.com/videoplayback? HTTP/1.1″ 200 873541 TCP_MISS:DIRECT
    127.0.0.1 – - [22/May/2012:13:14:08 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782153 TCP_HIT:NONE
    127.0.0.1 – - [22/May/2012:13:14:09 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782166 TCP_HIT:NONE
    127.0.0.1 – - [22/May/2012:13:14:27 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782166 TCP_HIT:NONE

    whats wrong TCP_MISS:DIRECT ?
    im install lusca on linux mint 12

    Comment by paydoel — May 22, 2012 @ 11:19 AM

  66. One billion thanks

    Comment by mahgobsoft — May 28, 2012 @ 4:27 PM

  67. After downloading the Ubuntu 10.4 LTS desktop and the work of keeping track of all the wonderful Arif unfortunately not out of Ubuntu to another computer science does not work with Squid on the same Ubuntu
    Please utilize a how the output from the Ubuntu desktop to another computer with Cache
    thanks very .

    Comment by mahgobsoft — May 28, 2012 @ 4:39 PM

  68. thanks for your amazing explication
    but its not working correctly with me
    i mean when i stop the line in squid.conf “storeurl_rewrite_program /etc/squid/storeurl.pl ” the proxy work but not cached youtube…
    when i run this line the proxy stop working
    can you help me plz

    Comment by Ismail — June 9, 2012 @ 1:01 AM

  69. its saying squid is already running but when i run youtube nothing is cached yet
    Do i have to configure squid.conf and put my ips??

    Comment by mungaz — June 13, 2012 @ 2:33 AM

  70. Dear Jhanzeb Bahi !!!!!!!

    I have 1 TB of HDD and it almost full with the cache drive for approx. 89%, what to do now.??? could you suggest me

    Comment by kashif khan — June 14, 2012 @ 10:25 AM

    • Well, either add another drive for caching support, or clear the current cache.

      Comment by Syed Jahanzaib / Pinochio~:) — June 14, 2012 @ 10:57 AM

      • After i plug a new device how to add or format this drive… well I dont even fromat after installation.. thats y im asking

        Comment by kashif khan — June 14, 2012 @ 3:31 PM

      • Jhanzaib bahi …. I added a new hard disk are reinstall the Ubuntu 10.4 again and after 4 days I saw this error and after every 2 hours my squid stops working…… Could you help me or suggest me….

        IN=eth1 OUT= MAC=00:19:db:55:52:7d:94:0c:6d:c8:4e:f3:08:00 SRC=66.220.149.94 DST=192.168.250.9 LEN=40 TOS=0×00 PREC=0×00 TTL=232 ID=19055 DF PROTO=TCP SPT=80 DPT=43107 WINDOW=0 RES=0×00 ACK RST URGP=0

        Comment by kashif khan — July 2, 2012 @ 8:11 PM

  71. help me please about youtube

    Comment by vickyajah — June 14, 2012 @ 2:25 PM

  72. sir , how to make autostart lusca head service on ubuntu 12.04 ?

    Comment by beruang — June 22, 2012 @ 12:46 PM

  73. AS-SALAM-WALEYKUM
    THIS IS REHMAT ALI GULWATING, HAVE TO INFORM THAT I INSTALL LUSCA/SQUID WITH MY MIKROTIK OS ALL WORK FINE ONLY 1 ISSUE IS THAT:
    WHEN ANY PC HIT YOUTUBE VIDEO IT PLAYS WELL AND COMPLETED NO ERROR BUT WHEN I REFRESH PAGE TO CHECK CACHE ONLY 1 SEGMENT OF VIDEO IS UTILIZED FROM LUSCA CACHE. DO U GET ANY UPDATE STOREURL.PL TO CACHE COMPLETE SEGMENTS OF YOUTUBE VIDEOS BECAUSE I USED UR LAST STOREURL.PL SCRIPT LINK: http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/ IF YES THEN PLEASE PLZ PLZ MAIL ME A LINK ON MY MAIL E-MAIL=BIG.BANG.NOW@GMAIL.COM.
    GOOD JOB KEEP IT UP ALLAH BLESS YOU. U R TRUELY DOING A PERFECT JOB.
    THANKING YOU,

    Comment by rehmat ali — June 26, 2012 @ 5:30 PM

  74. thanks alot sir i configure it is working great sir, but i have some problem as i try many times that i configure it but i can not get any results sir,
    i want to make transparent proxy,
    i can do browsing and everything by defining the proxy but i can not download anything by internet download manager.

    Comment by aliakber — July 16, 2012 @ 5:07 PM

  75. to put your (0×30 tcp_outgoing_tos all). By doing this the ZPH has no effect, ie mark everything as 0×30 and not just hit squid. And if you remove all the tcp_outgoing_tos 0×30 does not work. Any ideas?

    Comment by sadamssh — July 16, 2012 @ 10:53 PM

  76. I forgot to say that use DEBIAN

    Comment by sadamssh — July 16, 2012 @ 11:02 PM

  77. hi thank for your configuration
    squid will support form 3000 users?
    maximun how much it will support?

    Comment by karthik — August 1, 2012 @ 10:55 PM

    • Yes SQUID can support this number of users, but it really depend on the hardware you use. For 3000 users, you would be needing some latest hardware machine with quadx2 core cpus and lot of RAM.
      But putting big load on single machine, is not a wise idea, and consider to be a dad network design.
      Its better to divide proxy load on at least two machines.

      Comment by Syed Jahanzaib / Pinochio~:) — August 3, 2012 @ 7:30 AM

  78. Syed Jahanzaib Sir please help me mera ubuntu squid 5 ,6 hour bad cache khtm kar deta hai mery squid wly pc ki ye requirment hain Dell GX 280 Tower
    Processor 2.8 ghz
    Ram 1 GB
    Hard 120 GB Sata

    Comment by ahmad subhani — August 9, 2012 @ 12:14 PM

  79. assalamualaikum
    sir maine ye puchna tha k ye squid 10mb se zyada ki exe file cache nahi kar raha plz help me ……

    Comment by shani — August 12, 2012 @ 7:38 AM

  80. hello sir ,
    it work with squid3 ??

    Comment by Hussien — August 13, 2012 @ 2:52 AM

  81. assalamualaikum…
    please help me solved this problem
    2012/08/17 12:35:39| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-redhat-linux-gnu…
    2012/08/17 12:35:39| Process ID 1334
    2012/08/17 12:35:39| NOTICE: Could not increase the number of filedescriptors
    2012/08/17 12:35:39| With 1024 file descriptors available
    2012/08/17 12:35:39| Using epoll for the IO loop
    2012/08/17 12:35:39| Performing DNS Tests…
    2012/08/17 12:35:39| Successful DNS name lookup tests…
    2012/08/17 12:35:39| Adding nameserver 127.0.0.1 from squid.conf
    2012/08/17 12:35:39| Adding nameserver 221.132.112.8 from squid.conf
    2012/08/17 12:35:39| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2012/08/17 12:35:39| logfileOpen: opening log /var/log/squid/access.log
    2012/08/17 12:35:39| Swap maxSize 512000 + 8192 KB, estimated 40014 objects
    2012/08/17 12:35:39| Target number of buckets: 2000
    2012/08/17 12:35:39| Using 8192 Store buckets
    2012/08/17 12:35:39| Max Mem size: 8192 KB
    2012/08/17 12:35:39| Max Swap size: 512000 KB
    2012/08/17 12:35:39| logfileOpen: opening log /var/log/squid/store.log
    2012/08/17 12:35:39| AUFS: /cache1: log ‘/cache1/swap.state’ opened on FD 18
    2012/08/17 12:35:39| AUFS: /cache1: tmp log /cache1/swap.state.new opened on FD 18
    2012/08/17 12:35:39| Rebuilding storage in /cache1 (DIRTY)
    2012/08/17 12:35:39| Using Least Load store dir selection
    2012/08/17 12:35:39| Current Directory is /root
    2012/08/17 12:35:39| Loaded Icons.
    2012/08/17 12:35:39| commBind: Cannot bind socket FD 20 family 2 to 192.168.3.2 port 3128: (98) Address already in use
    FATAL: Cannot open HTTP Port

    Comment by widi — August 17, 2012 @ 10:43 AM

    • It shows that port 3128 is already occupied by another application, possibly another instance of squid or lusca is already running
      check it with

      ps aux |grep squid

      or

      netstat -tulpn | grep :8080

      first kill previous instance, then start new one,
      you can kill it by

      killall -9 squid (or app name)

      Comment by Syed Jahanzaib / Pinochio~:) — August 18, 2012 @ 10:17 PM

      • Thank you sir
        the problem has been found, for the problematic proxy port (3128) I changed the port 8080 results instantly

        Comment by widi — August 21, 2012 @ 9:04 AM

      • gr8

        Comment by Syed Jahanzaib / Pinochio~:) — August 22, 2012 @ 9:30 AM

  82. i am try squid configuration in centos 5.5, (sudo apt-get install gcc build-essential sharutils ccze libzip-dev automake1.9) i am install gcc and build-essential but other packages not found . even tho it was compiled every steps are work, but i am stuck up in this command (chown proxy:proxy /cache1 ) error chown: `proxy:proxy’: invalid user any solution.

    i have give root instead of proxy
    /usr/local/squid/sbin/squid -z
    FATAL: getpwnam failed to find userid for effective user ‘proxy’
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.
    CPU Usage: 0.006 seconds = 0.003 user + 0.003 sys
    Maximum Resident Size: 6816 KB
    Page faults with physical i/o: 0
    Aborted

    please help me, in my server Ubuntu is not installing(after language section it was thronging error CD ROM is not stable)system x3850 m2 server, so that installed centos but in Ubuntu every thin is working fine IBM xeon

    Comment by karthik — August 21, 2012 @ 1:59 PM

  83. thank a lot sir…it’s work even i join with coovachilli in 1pc

    Comment by jadiakbar — August 26, 2012 @ 5:45 AM

  84. Salam alkm

    i just installed on my ubuntu box, and the youtube is cached 100%….. thank you

    i want to know make the the youtube open as default same ( 240/ video quality ) to all users, how i can make that ????

    Comment by SAT LINK — August 29, 2012 @ 3:44 PM

    • Currently there is no function to force 240p streamed to user.
      somewhere at videocache forum, I read that they were talking about implementing this feature that by default, 240p video should be served to user as default. I don’t know if they have implemented it on there software, check there forum or ask there support.

      Comment by Syed Jahanzaib / Pinochio~:) — August 30, 2012 @ 9:09 AM

  85. hello mr syed..
    i have 4 harddisks 1Tb and 3 500gb
    i want to use 1 of 500 gb for system
    and 1tb for youtube cahce
    and the 2 other for files

    can you tell me how can do this..thanks you

    Comment by ismail — August 30, 2012 @ 4:00 PM

  86. Is youtube caching working with this guide as of right now or still have to refer to nginx method.

    Comment by Badr — August 30, 2012 @ 5:52 PM

    • Some people claims that its still working at there end.
      I know that youtube didnt changed the this range behavior for the whole
      globe because as for now few people emailed me that storeurl.pl still working at there end. So in the same country you can still get two different url patterns.

      Comment by Syed Jahanzaib / Pinochio~:) — August 31, 2012 @ 8:31 AM

  87. Hello….
    I have successful installation of squid video caching but I am facing some issues like serving of old/ stale pages…. Happens mostly with flckr.com, news websites and few blogs…. Could there be any solution to the same….? Thanks….

    Comment by Dinesh Bhardwaj — September 3, 2012 @ 8:33 PM

  88. sayed can you tell me how can i do that

    i have RB 1100AH2 and i have PPPOE setup on 10 interface each interface have
    defrant ip addresses and i have squid cash set up and evrything working find with all PPPOE interface
    the issue is i want to
    enable the cash rule for just one PPPOE interface

    and im using this nat rule
    add action=dst-nat chain=dstnat disabled=yes dst-port=80 protocol=tcp \
    to-addresses=10.3.0.100 to-ports=800

    10.3.0.100 my squid ip
    i had try to put the in interface = the PPPOE interface i need it work with cash but wont work
    and also i tried to put the interface ip on src address and did not work

    Comment by thegoogl — September 14, 2012 @ 12:58 AM

    • in src-address, add the ip range which is assigned to specific interface. this way only clients with the specific pool will be redirected to squid, rest of clients will go directly, as per default route.

      Comment by Syed Jahanzaib / Pinochio~:) — September 16, 2012 @ 12:56 PM

  89. If you are using centos tiens to change the file
    line “proxy” for squid and ready

    Comment by zeus — September 17, 2012 @ 10:33 AM

  90. sir mena ap ka blog sey bahoot kuch hasel kiyah hai ma ser bahoot bahoot khuch ho but kuch din chelna ka bad mera computer ki cache memmry pyll ho gai hai sir plz is ko delled karna ki bhi command bata dey

    Comment by hafizfidaali — October 5, 2012 @ 12:04 AM

  91. sir mari total disk space 72 gb hai jes ma sey 60 gb cache ka liyah rahki hai plz help me sir

    Comment by hafizfidaali — October 6, 2012 @ 2:04 AM

  92. sir mena ap ka blog sey bahoot kuch hasel kiyah hai ma sir bahoot bahoot khuch ho but kuch din chelna ka bad mera ubuntu server ki cache memmry full ho gai hai sir plz is ko delled karna ki bhi command bata dina sir mari total disk space 72 gb hai jes ma sey 60 gb cache ka liyah rahki hai plz help me sir

    Comment by hafizfidaali — October 7, 2012 @ 12:08 AM

  93. Assalamualaikum..
    Dear Jahanzaib..
    Please help me solved this problem
    root@ubuntu:~# 2012/10/10 00:33:37| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-pc-linux-gnu…
    -bash: 2012/10/10: No such file or directory
    2012/10/10 00:33:37| NOTICE: Could not increase the number of filedescriptors
    2012/10/10 00:33:37| With 1024 file descriptors available
    2012/10/10 00:33:37| Using epoll for the IO loop
    2012/10/10 00:33:37| Performing DNS Tests…
    2012/10/10 00:33:37| Successful DNS name lookup tests…
    2012/10/10 00:33:37| Adding nameserver 192.168.3.254 from squid.conf
    2012/10/10 00:33:37| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2012/10/10 00:33:38| User-Agent logging is disabled.
    2012/10/10 00:33:38| Referer logging is disabled.
    2012/10/10 00:33:38| logfileOpen: opening log /var/log/lusca/access.log
    2012/10/10 00:33:38| Unlinkd pipe opened on FD 18
    2012/10/10 00:33:38| Swap maxSize 61891584 + 524288 KB, estimated 4801220 objects
    2012/10/10 00:33:38| Target number of buckets: 240061
    2012/10/10 00:33:38| Using 262144 Store buckets
    2012/10/10 00:33:38| Max Mem size: 524288 KB
    2012/10/10 00:33:38| Max Swap size: 61891584 KB
    2012/10/10 00:33:38| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
    2012/10/10 00:33:38| logfileOpen: opening log /var/log/lusca/store.log
    2012/10/10 00:33:38| AUFS: /cache: log ‘/cache/swap.state’ opened on FD 20
    2012/10/10 00:33:38| AUFS: /cache: tmp log /cache/swap.state.new opened on FD 20
    2012/10/10 00:33:38| Rebuilding storage in /cache (DIRTY)
    2012/10/10 00:33:38| Using Least Load store dir selection
    2012/10/10 00:33:38| Current Directory is /root
    2012/10/10 00:33:38| Loaded Icons.
    2012/10/10 00:33:38| Accepting transparently proxied HTTP connections at 0.0.0.0, port 3128, FD 22.
    2012/10/10 00:33:38| WCCP Disabled.
    2012/10/10 00:33:38| Ready to serve requests.
    2012/10/10 00:33:38| WARNING: store_rewriter #1 (FD 7) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #2 (FD 8) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #3 (FD 9) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #4 (FD 10) exited
    2012/10/10 00:33:38| Too few store_rewriter processes are running
    FATAL: The store_rewriter helpers are crashing too rapidly, need help!

    Comment by Habib Yunus — October 9, 2012 @ 10:40 PM

    • The error occurs when there is any syntax mistake in the storeurl.pl , its the culprit
      Try copying it to notepad then see for any invalid character, or try copy pasting it from another source.

      what is your aim ? are you trying to caching youtube? if yes , then use NGINX method, its much better then this one. Search my blog I have written an article on it too.

      Comment by Syed Jahanzaib / Pinochio~:) — October 10, 2012 @ 9:16 AM

  94. Sorry for my noob and bad english. I want to ask wether the facebook app (like zynga, ninja saga, etc) is dynamic content? how we can cache it?

    Comment by Al Kurnia — October 16, 2012 @ 6:15 PM

    • Try not to cache FB contents, they some times becomes headache. avoid caching dynamic content specially content that updates very frequently :)

      Comment by Syed Jahanzaib / Pinochio~:) — October 17, 2012 @ 8:43 AM

  95. Thank u. Now, my proxy be good

    Comment by haloteknisi — October 26, 2012 @ 4:36 PM

  96. [...] buat install lusca na di mari [...]

    Pingback by install lusca « Tikus Server — October 27, 2012 @ 10:09 AM

  97. Hello there! I simply wish to give you a huge thumbs up for the excellent information you have got here on this post. I am returning to your web site for more soon.

    Comment by filme xxx — October 31, 2012 @ 3:07 AM

  98. i want script redirect port 80 to port 8080

    Comment by malak — November 7, 2012 @ 6:05 AM

  99. Syed , what’s the way to add more than one hdd to save the files …
    i’ve 3hdd * 2TB =6TB

    Comment by darkng — November 21, 2012 @ 3:27 PM

  100. Thanks syed , just a last question about adding a hdd .
    for example if i have 4HDD’s
    1xSATA II – 90GB
    3x2TB Hardisk’s =6TB in total

    if i mount every disk and add just the cashe_dir of all the disk .
    would the disk be balanced so if the first is full to add it to the second or if the first is filled with 1Mb then the second becomes so the third … and so on .

    thanks for every help from you

    Comment by darkng — November 24, 2012 @ 3:05 AM

  101. Hiya, i’m having problem when i finish isntalling iàve got that error trying to run squid:

    FATAL: Bungled (null) line 170: http_access deny all
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.

    Could you help me?

    cheers
    Rafa

    Comment by Rafa TI — November 24, 2012 @ 5:52 PM

  102. hi sir,

    i m getting this msg after installing

    ” squidaio_queue_request: WARNING – Queue congestion”

    any suggestion!

    millions of thanks

    Comment by fajja — November 26, 2012 @ 2:10 PM

  103. hi, sir.
    how can disable caching the video from youtube only ..

    Comment by khaled — November 26, 2012 @ 11:33 PM

  104. Thank u. Now, my proxy be verygood

    Comment by tikkee — December 12, 2012 @ 8:36 PM

  105. Thank before, my problem why not stored in chrome update my proxy. consequently chrome update every time a connection is slow.
    Help me…

    Comment by Habib Yunus — December 13, 2012 @ 10:26 AM


  106. how to do like this?

    Comment by tom — December 21, 2012 @ 1:03 PM

  107. hi sir,
    i m getting this msg after installing
    ” squidaio_queue_request: WARNING – Queue congestion”
    any suggestion!
    millions of thanks

    Comment by fajja — December 21, 2012 @ 11:47 PM

  108. Nice project zaib bhai….

    Comment by Syed Azhar Hussain — December 27, 2012 @ 3:43 AM

  109. syed bhai after installing squid it works fine but after sometime i cant access cachelog and hits it gives this error.

    ./cachelog
    tail: cannot open `/var/log/squid/cache.log’ for reading: No such file or director
    y
    tail: no files remaining

    tail -f /var/log/squid/access.log
    tail: cannot open `/var/log/squid/access.log’ for reading: No such file or directo
    ry
    tail: no files remaining

    after rebooting squid those commands start working but after sometime like 12 hours or more it start giving same error.

    can u please help me y this happens?

    Comment by fajja — December 27, 2012 @ 5:26 PM

  110. 10080 minute is actually 1 week , not 1 month. Anyway, good project. Thanks

    Comment by Indra Gunawan — December 27, 2012 @ 5:34 PM

    • :)
      Configuration changes rapidly as per requirement, I forgot to update the heading :)

      Comment by Syed Jahanzaib / Pinochio~:) — December 28, 2012 @ 8:08 AM

      • Hi Bro!
        I will be so Grateful if you could please guide me for the line to add some lines: in order to cache from this site ” filehippo.com”

        Alex

        Comment by speedwaves — January 3, 2013 @ 7:23 AM

  111. I am having this problem with my sitemap. i got a video sharing website and its sitemap have some error in it. can anyone give me a solution to that.

    Comment by Sana zahra — January 9, 2013 @ 4:33 PM

  112. Hi
    I was wondering,since i decided to install proxy server in my network,could you advice me what kind of hardware should i buy for proxy machine.I have about 150 clients and average consume about 30mbits.
    I would appreciate your answer.
    10x in advance

    Comment by Fibonacci — January 19, 2013 @ 3:56 AM

    • Fro 150 Users, Any Intel Core2Duo base PC with 4-8 GB RAM, 320/500 GB HDD would be enough.
      XEON base processors are however good and stable as compare to desktop PC. Its all depend on your budget.

      It’s a good idea to have 64 bit architecture hardware/OS support. It works very good and reliable.

      Comment by Syed Jahanzaib / Pinochio~:) — January 19, 2013 @ 11:25 AM

      • Many thanks for quick reply

        Comment by Fibonacci — January 19, 2013 @ 2:19 PM

  113. can not start up automatically after a reboot
    is there a way?

    Comment by rama — January 23, 2013 @ 2:03 PM

  114. Hello ,
    did anyone find a fix for the youtube segmentation issue in order to be downloaded as a single file in the zph tos
    i am using LUSCA_HEAD-r14733
    thank you

    Comment by cr — January 26, 2013 @ 11:29 PM

  115. Good Work !!! Zaib

    Comment by Chetan Muneshwar — January 30, 2013 @ 10:11 PM

  116. Hello
    thank you for all tut.
    I have problem after make install and setup cache work fine with images , exe to by pass queue from mikrotik but youtube and video dnt by pass please bro if can help me with my request and I want you to advise me which better between nginx the one I installed and LUSCA ,
    thank you very much

    Comment by alfanet1 — February 11, 2013 @ 1:52 PM

  117. hi
    how can i use laptop with one LAN interface as a cache server

    Comment by dh — February 13, 2013 @ 9:19 PM

    • on squid, set its default gateway and dns pointing to mikrotik interface, and in mikrotik create a masquerade rule that masquerade all traffic coming from squid pc.
      move this rule on TOP. (if using HOTSPOT you can bypass squid box mac in BYPASSED macs)

      Comment by Syed Jahanzaib / Pinochio~:) — February 14, 2013 @ 4:12 PM

      • it will work with dstnat the one interface will work as input to squid and output.? dstnat in mikrotik with out change squid roule

        Comment by dh — February 15, 2013 @ 2:21 AM

  118. if can the script for one LAN squid with mikrotik nat

    Comment by dh — February 17, 2013 @ 4:51 PM

    • yes you can use one lan card with squid also.
      in squid, configure default gateway and dns pointing to mikrotik interface which is connected with the squid box.
      also create a NAT rule for squid ip.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:42 PM

  119. Hello

    i make 2 squid as peer in 1 box. the peer is working perfect between ( Lusca as video cache ) and ( squid2.7.stabel7 as data files cache )

    the problem is, we can mangle in mikrotik the DSCP TOS of squid only, but not the peer data that coming from Lusca.

    any solution ???

    Comment by Bilal Mahdi — February 18, 2013 @ 3:39 AM

    • I have not worked in such scenario, but TOS wont work in default config i am sure.
      Search google there might be some workaround for it.
      I recommend you to post your query at squid mailing list.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:41 PM

  120. Hello All
    i have install ( lusca – video cache ) and ( squid 2.7 stable 7 ) in same box as peer.
    i Tos between the mikrotik and squid is working fine but the peer data is not include in the Tos for some reason .

    any suggesting !!!

    Comment by Bilal Mahdi — February 18, 2013 @ 3:44 AM

    • I have not worked in such scenario, but TOS wont work in default config i am sure.
      Search google there might be some workaround for it.
      I recommend you to post your query at squid mailing list.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:40 PM

  121. pcc loadbalacsinig main squid settinig kese karo plz help me

    you help me

    Comment by noman — February 20, 2013 @ 11:35 PM

  122. hello, i wanna ask
    i’ve just use ur storeurl and conf. im forget to backup my old config.
    but now i want to disable youtube cache, which one i should disable?
    i tried # in youtube fix and compability old cached, but cache still work
    thanks

    Comment by merry — February 24, 2013 @ 1:14 AM

  123. Thank you very much for this tutorial. It’s great. But I have two problems. 1. My cache is always full then the squid will stop. how can I automatically remove the old cache object? 2. Internet call is always disconnected?

    Comment by Rolando Casinillo — March 2, 2013 @ 6:14 PM

  124. Assalamu’alaikum im from indonesia, i have finishe doing the instalation, then i try it, its work fine, but while im playing the other video from youtube, in the screen stiil apear and play the video that that i played before, what is th eproblem ,thanks before
    Wassalam

    Comment by Dedy Availatz — March 22, 2013 @ 1:53 PM

  125. dear Brother

    we have install the ( Cache Youtube with SQUID / LUSCA )

    and its work perfect, and after 671 GB cached used, the squid start make a big traffic on our upload, it tack all the upload bandwidth.

    any suggesting ???

    Comment by Bilal Mahdi — March 23, 2013 @ 3:38 PM

    • Hmmm its to understandable that how cache is increasing upload traffic. However first try to clear cache , and see if it helps, then we can do further digging what could be possible wrong.

      Comment by Syed Jahanzaib / Pinochio~:) — March 25, 2013 @ 9:43 AM

  126. Is there a way to test storurl.pl if it works??
    I test the storeurl.pl as the above posted, and fed the youtube url:
    http://r6—sn-5njj-u2xe.c.youtube.com/videoplayback?algorithm=throttle-factor&burst=40&cp=U0hVSVRSVF9HTkNONV9MTFhDOmZYOUhGeEFicHoy&cpn=ifHLBv3rILC7UrfL&expire=1364476224&factor=1.25&fexp=916807%2C914083%2C916623%2C901478%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C904830%2C919373%2C930803%2C906836%2C920201%2C929602%2C930101%2C930603%2C926403%2C900824%2C910223&id=o-AJDf3TDHyp2H0FMpLSoCKM0akW1_kumyjOAtem_dY-_V&ip=203.71.84.209&ipbits=8&itag=34&keepalive=yes&key=yt1&ms=au&mt=1364452453&mv=m&newshard=yes&range=13-1783807&ratebypass=yes&signature=D2474D7D411E60FC505DBF4CBDC6A2BDF1AE32B9.45C16D3AA02E5BBC38F373993A3688480EDA139C&source=youtube&sparams=algorithm%2Cburst%2Ccp%2Cfactor%2Cid%2Cip%2Cipbits%2Citag%2Csource%2Cupn%2Cexpire&sver=3&upn=ZRNE9iDw7gM

    the result is :
    http://r6—sn-5njj-u2xe.c.youtube.com/videoplayback?algorithm=throttle-factor&burst=40&cp=U0hVSVRSVF9HTkNONV9MTFhDOmZYOUhGeEFicHoy&cpn=ifHLBv3rILC7UrfL&expire=1364476224&factor=1.25&fexp=916807%2C914083%2C916623%2C901478%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C904830%2C919373%2C930803%2C906836%2C920201%2C929602%2C930101%2C930603%2C926403%2C900824%2C910223&id=o-AJDf3TDHyp2H0FMpLSoCKM0akW1_kumyjOAtem_dY-_V&ip=203.71.84.209&ipbits=8&itag=34&keepalive=yes&key=yt1&ms=au&mt=1364452453&mv=m&newshard=yes&range=13-1783807&ratebypass=yes&signature=D2474D7D411E60FC505DBF4CBDC6A2BDF1AE32B9.45C16D3AA02E5BBC38F373993A3688480EDA139C&source=youtube&sparams=algorithm%2Cburst%2Ccp%2Cfactor%2Cid%2Cip%2Cipbits%2Citag%2Csource%2Cupn%2Cexpire&sver=3&upn=ZRNE9iDw7gM sucks

    I think the storeurl.pl is not work correctly.

    If I cancel “storeurl_rewrite_program /etc/squid3/storeurl.pl”, and replay the youtube movie again, I can find “TCP_HIT” in the access.log with the same movie.

    Comment by YCC — March 28, 2013 @ 12:29 PM

  127. bahi ager aap ko 400$ pay kr do tu kia aap bna do gy

    Comment by Shahid — March 31, 2013 @ 2:41 PM

  128. few months ago i follow your tutorial, squid can save video youtube HIT good .. but yesterday i play youtube videos and when I play another video why video is my first turn, rotated back, I’ve been trying to clear squid cache but return again like what I described above
    I even re-install the server sdh still as above
    may be updated squid.conf & store.url you

    Comment by hary sasmito — April 1, 2013 @ 10:03 PM

    • I have the same problem. I think the storeurl.pl is useless which should be rewrited.

      Comment by YCC — April 3, 2013 @ 1:52 PM

  129. biar dikata nyontek,, tapi hasil oprek Syed Jhanzaib. hasilnya lebih maknyuzzzzzzz… :-)

    Comment by ming — April 9, 2013 @ 12:42 PM

  130. few months ago i follow your tutorial, squid can save video youtube HIT good .. but yesterday i play youtube videos and when I play another video why video is my first turn, rotated back, I’ve been trying to clear squid cache but return again like what I described above
    I even re-install the server sdh still as above
    may be updated squid.conf & store.url you

    Comment by Muhammad Ardiansyah Putra — April 20, 2013 @ 10:57 AM

    • This method no longer cache the youtube cache AFAIK.
      Try with NGINX method, possibly it may b working. youtube is blocked in our country from past one year therefore I cant do any testing on it,

      Comment by Syed Jahanzaib / Pinochio~:) — April 24, 2013 @ 10:23 AM

  131. assalamualaikum…..
    I am from indonesia, i’ve tried the tutorial above. but the results I could not browsing.

    sorry, I am not so fluent in English :D

    Comment by Aji Diyantoro — April 20, 2013 @ 6:23 PM

    • You need to provide more detail. Without knowing you goals and what you have done so far, its hard to assist you. Please provide details how you have setup squid and how it is attached to mikrotik or what? describe your network scenario and goals you want to achieve.

      Comment by Syed Jahanzaib / Pinochio~:) — April 24, 2013 @ 10:22 AM

  132. hello syed ,
    i have install the nginx with YouTube and cache work fine except the download the YouTube video from real player or internet download manger its not any help for that?

    Comment by husam — May 4, 2013 @ 12:27 AM

    • Can’t help you in this regard anymore, youtube got banned by the authorities in our country last year, so i cant test anything on it.

      Comment by Syed Jahanzaib / Pinochio~:) — May 4, 2013 @ 10:20 AM

  133. High Performance Cache HIT Proxy Lusca on Ubuntu Server + Configurasi Router MikroTik

    Sumber: MR-EKOAPRIADI – http://mr-ekoapriadi.blogspot.com/2013/03/high-performance-lusca-squid-proxy.html

    Comment by Adam — May 5, 2013 @ 10:54 PM

  134. hello sayed
    in this case that parameter should modify according to server memory (ram ) and capacity ? and all videos and files are not cashed ?

    Comment by Ali — May 27, 2013 @ 7:51 PM

  135. new youtube any help to cache this
    http://www.youtube.com/ptracking?
    plid=AAThOYd6J3xLA9WC
    oid=SlODfwbK82f2lVHy5P82EA.UnsFrT3f3Ne9E0pAoQfCsQ.VyEtjbeN302yXkP3Qulrog.9b8naKhAjYHfDP3yKaWKtg.GoednyIs730UDQtrvI_P-Q.FvXsj2Mdo_hj4nnNnOxKzg.0YSVm7yx3KFfY2JfQ2_1CA
    pltype=contentugc
    ptk=youtube_multi
    video_id=N-kuHq5rJYU
    cpn=pJmS-oUxyadHDGM-

    GET /ptracking?plid=AAThOYd6J3xLA9WC&oid=SlODfwbK82f2lVHy5P82EA.UnsFrT3f3Ne9E0pAoQfCsQ.VyEtjbeN302yXkP3Qulrog.9b8naKhAjYHfDP3yKaWKtg.GoednyIs730UDQtrvI_P-Q.FvXsj2Mdo_hj4nnNnOxKzg.0YSVm7yx3KFfY2JfQ2_1CA&pltype=contentugc&ptk=youtube_multi&video_id=N-kuHq5rJYU&cpn=pJmS-oUxyadHDGM- HTTP/1.1
    Host: http://www.youtube.com
    User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Referer: http://s.ytimg.com/yts/swfbin/watch_as3-vfldOoVEA.swf
    Cookie: VISITOR_INFO1_LIVE=yv8v0rxZnvI; PREF=fv=11.7.700; YSC=x2u8uvMXD8Y; ACTIVITY=1373537164597
    Connection: keep-alive

    HTTP/1.0 204 No Content
    Date: Thu, 11 Jul 2013 10:05:14 GMT
    Server: gwiseguy/2.0
    Content-Type: video/x-flv
    Expires: Tue, 27 Apr 1971 19:44:06 EST
    X-YouTube-Other-Cookies: VISITOR_INFO1_LIVE=yv8v0rxZnvI
    X-Content-Type-Options: nosniff
    X-XSS-Protection: 1; mode=block
    X-Frame-Options: SAMEORIGIN
    Content-Length: 0
    X-Cache: MISS from cache
    X-Cache-Lookup: MISS from cache:8080

    Comment by joe — July 11, 2013 @ 3:11 PM

  136. FATAL: cache_dir /usr/local/squid/var/cache: (2) No such file or directory
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.
    CPU Usage: 0.004 seconds = 0.004 user + 0.000 sys
    Maximum Resident Size: 6192 KB
    Page faults with physical i/o: 0
    Aborted (core dumped)

    what the meaning of error

    Comment by AHMAD FARUQ — July 15, 2013 @ 12:25 PM

    • It says that the squid cannot found the cache2 folder in your system.

      Adjust the cache_dir folder in the squid.conf , point to to valid folder, or create that folder in the respective location.

      Comment by Syed Jahanzaib / Pinochio~:) — July 16, 2013 @ 8:21 AM

      • apakah untuk folder cache harus dibuatkan partisi sendiri… atau tidak

        Comment by AHMAD FARUQ — July 16, 2013 @ 8:57 AM

      • whether the cache folder should be created for its own partition … or not

        Comment by AHMAD FARUQ — July 16, 2013 @ 8:59 AM

      • you can use same drive for caching folder, or you can also use separate drive for the cache , actually it depends on the number of users, load and some other factors.

        Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:06 AM

    • change the folder of log

      Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:06 AM

  137. 2013/07/16 07:30:32| aclParseIpData: WARNING: Netmask masks away part of the specified IP in ’192.168.3.1/8′
    2013/07/16 07:30:32| WARNING: ’0.0.0.0/0.0.0.0′ is a subnetwork of ’0.0.0.0/0.0.0.0′
    2013/07/16 07:30:32| WARNING: because of this ’0.0.0.0/0.0.0.0′ is ignored to keep splay tree searching predictable
    2013/07/16 07:30:32| WARNING: You should probably remove ’0.0.0.0/0.0.0.0′ from the ACL named ‘all’
    2013/07/16 07:30:32| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-redhat-linux-gnu…
    2013/07/16 07:30:32| Process ID 12774
    2013/07/16 07:30:32| NOTICE: Could not increase the number of filedescriptors
    2013/07/16 07:30:32| With 1024 file descriptors available
    2013/07/16 07:30:32| Using epoll for the IO loop
    2013/07/16 07:30:32| Performing DNS Tests…
    2013/07/16 07:30:32| Successful DNS name lookup tests…
    2013/07/16 07:30:32| Adding nameserver 192.168.3.1 from squid.conf
    2013/07/16 07:30:32| Adding nameserver 8.8.8.8 from squid.conf
    2013/07/16 07:30:32| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2013/07/16 07:30:32| logfileOpen: opening log /var/log/squid/access.log
    2013/07/16 07:30:32| logfileOpen: opening log /usr/local/squid/var/logs/access.log

    FATAL: Cannot open ‘/usr/local/squid/var/logs/access.log’ for writing.
    The parent directory must be writeable by the
    user ‘proxy’, which is the cache_effective_user
    set in squid.conf.
    i found the error again
    please syed jahanzaib help me…..

    Comment by AHMAD FARUQ — July 16, 2013 @ 4:34 PM

    • Do you have the correct ownership of files/directories? It says that it cannot open ‘/usr/local/squid/var/logs/access.log’ for writing and that the parent directory must be writeable by the user ‘proxy’ OR ‘squid’. You could try chown -R squid /usr/local/squid

      Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:05 AM

      • salam…. syed
        I am very curious about the tutorial that you share

        I have several times tried to follow all the tutorial and I have failed but to this day I continue to try to understand because I still do not know much about network …………

        whether you willing help me … provide installation instructions ubuntu 10:04 server especially on the partition corresponding to the appropriate setting squid with this blog

        This specification yng computer I use for external proxies:
        = 1 Gb ram
        = 80 Gb HDD
        192.168.3.2 ip proxy.

        email = bouojog@gmail.com
        or through your blog is because I am waiting for your answer on a daily basis.

        I am from Indonesia and could not speak English well
        apologize if there is a greeting that is less true

        Comment by AHMAD FARUQ — July 18, 2013 @ 12:45 PM

  138. […] There might be some problem with the storeurl.pl content, either its not copy pasted correctly. First try without storeurl.pl IF it works ok , then Try to create storeurl.pl from following URL. http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-video&#8230; […]

    Pingback by Youtube caching with SQUID 2.7 [using storeurl.pl] | Syed Jahanzaib Personnel Blog to Share Knowledge ! — September 17, 2013 @ 2:46 PM

  139. Dear All,

    I am getting following error when typing “lusca -d1″

    /cache-1//swap.state: (13) Permission denied

    FATAL: storeAufsDirOpenSwapLog: Failed to open swap log.

    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.

    CPU Usage: 0.012 seconds = 0.004 user + 0.008 sys

    Maximum Resident Size: 16160 KB

    Page faults with physical i/o: 0

    (squid)[0x80b4a84]

    (squid)[0x80b543f]

    (squid)[0x80c25e7]

    (squid)[0x80bf6e3]

    (squid)[0x80af1fe]

    (squid)[0x80aba34]

    (squid)[0x804bdaa]

    (squid)[0x804cb13]

    /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75584d3]

    (squid)[0x804d1b9]

    Comment by Siraj — September 28, 2013 @ 9:42 PM

    • If using Ubuntu

      chown proxy:proxy /cache-1

      Erase the swap.state file
      squid -z
      service squid start

      OR
      squid -d1N

      Comment by Syed Jahanzaib / Pinochio~:) — September 29, 2013 @ 10:17 AM

      • whe i am typing “service squid3 restart:
        i am getting following error:
        stop: Unknown instance:
        start: Rejected send message, 1 matched rules; type=”method_call”, sender=”:1.103″ (uid=1000 pid=18302 comm=”start squid3 “) interface=”com.ubuntu.Upstart0_6.Job” member=”Start” error name=”(unset)” requested_reply=”0″ destination=”com.ubuntu.Upstart” (uid=0 pid=1 comm=”/sbin/init”)

        Comment by Siraj — September 29, 2013 @ 5:59 PM

      • Dear All,
        Cant we just use one interface. I mean same interface is connecting cache server to internet and also receiving request from LAN/user? if it possible then how can we change perimeters in your mentioned configs to achieve this goal.

        Comment by Siraj — September 29, 2013 @ 6:16 PM

  140. Salam Bhai,
    Mikrotik proxy can able to chche youtube video ????

    Comment by rabby — October 7, 2013 @ 9:33 AM

  141. i’ve tested yesterday but seems it doesn’t work anymore, can anyone re-confirm this tut still works or not ?

    Comment by Albert — November 23, 2013 @ 9:34 AM

  142. Mantaf gan atriclenya sangat membantu, ini sy experiment sendiri pake ini lumayan mantaf gan.
    Sumber: http://mediacomptech.blogspot.com/2013/11/cara-baypass-hit-proxy.html

    Comment by Tuan Lang — November 28, 2013 @ 6:35 AM

  143. Mantaf gan atriclenya sangat membantu, ini sy experiment sendiri pake ini lumayan mantaf gan.
    Sumber: http://mediacomptech.blogspot.com/2013/11/high-performance-cache-hit-proxy-lusca.html

    Comment by Tuan Lang — November 28, 2013 @ 6:39 AM

  144. thanks for the tutorial,
    i’m newbie who learn microtik :-)

    Comment by bersihwajah — December 19, 2013 @ 6:19 PM

  145. […] by Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue « Syed Jah… — March 28, 2012 @ 10:38 […]

    Pingback by Youtube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.pl | Syed Jahanzaib Personnel Blog to Share Knowledge ! — January 20, 2014 @ 1:48 PM

  146. hati – hati … perhatikan baik2…. banyak configurasi yg aneh …. jangan di serap 100% … dia ga mungkin share ilmu 100% logikanya gitu , tp kasian 1000′an orang awan jd korban ….

    Comment by Bajak Laut — February 1, 2014 @ 7:04 PM

  147. aoa. kya ap torrent ke ports forward kar sakty hy

    Comment by Naeem Ahmed — April 10, 2014 @ 2:50 AM


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 2,047 other followers

%d bloggers like this: