This is a Shorter Version of > http://aacable.wordpress.com/tag/aacable-howto-cache-youtube/
This is a quick reference guide for SQUID 2.7 installation on Ubuntu Desktop ver 10.4 with youtube caching supported. Make sure you have setup proper internet connection in Ubuntu BOX.
Install SQUID by
apt-get install squid
After installation done, Now edit it’s configuration file
nano /etc/squid/squid.conf
remove current lines and paste all squid.conf as follows.
# SQUID 2.7 TEST CONFIG FILE # Email: aacable@hotmail.com # Web : http://aacable.wordpress.com # PORT and Transparent Option http_port 8080 transparent server_http11 on icp_port 0 # Cache Directory , modify it according to your system. # but first create directory in root by mkdir /cache1 # and then issue this command chown proxy:proxy /cache1 # [for ubuntu user is proxy, in Fedora user is SQUID] # I have set 10 GB for caching reserved just for caching , # adjust it according to your need. # My recommendation is to have one cache_dir per drive. zzz #store_dir_select_algorithm round-robin cache_dir aufs /cache1 1000 16 256 cache_replacement_policy heap LFUDA memory_replacement_policy heap GDSF # If you want to enable DATE time n SQUID Logs,use following emulate_httpd_log on logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt log_fqdn off # How much days to keep users access web logs # You need to rotate your log files with a cron job. For example: # 0 0 * * * /usr/local/squid/bin/squid -k rotate logfile_rotate 14 debug_options ALL,1 cache_access_log /var/log/squid/access.log cache_log /var/log/squid/cache.log cache_store_log /var/log/squid/store.log #I used DNSAMSQ service for fast dns resolving #so install by using apt-get install dnsmasq first dns_nameservers 127.0.0.1 192.168.2.4 192.168.2.3 ftp_user anonymous@ ftp_list_width 32 ftp_passive on ftp_sanitycheck on #ACL Section acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 # https, snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow all http_reply_access allow all icp_access allow all #========================== # Administrative Parameters #========================== # I used UBUNTU so user is proxy, in FEDORA you may use use squid cache_effective_user proxy cache_effective_group proxy cache_mgr aacable@hotmail.com visible_hostname proxy.aacable.net unique_hostname aacable@hotmail.com #============= # ACCELERATOR #============= half_closed_clients off quick_abort_min 0 KB quick_abort_max 0 KB quick_abort_pct 100 vary_ignore_expire on reload_into_ims on pipeline_prefetch on range_offset_limit 512 KB read_timeout 30 minutes log_fqdn off memory_pools off # If you want to hide your proxy machine from being detected at various site use following via off #============================================ # OPTIONS WHICH AFFECT THE CACHE SIZE / zaib #============================================ # If you have 4GB memory in Squid box, we will use formula of 1/3 # You can adjust it according to your need. IF squid is taking too much of RAM # Then decrease it to 128 MB or even less. cache_mem 8 MB minimum_object_size 512 bytes maximum_object_size 256 MB maximum_object_size_in_memory 64 KB #============================================================$ # SNMP , if you want to generate graphs for SQUID via MRTG #============================================================$ #acl snmppublic snmp_community gl #snmp_port 3401 #snmp_access allow snmppublic all #snmp_access allow all #============================================================ # ZPH , To enable cache content to be delivered at full lan speed, # To bypass the queue at MT. #============================================================ tcp_outgoing_tos 0x30 all zph_mode tos zph_local 0x30 zph_parent 0 zph_option 136 # Caching Youtube acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\? acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\? acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\? acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\? acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\? acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\? acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/ acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/ acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4) acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]? acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp) acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v) acl videocache_allow_url url_regex -i \.apniisp\.com\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v) acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4) acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com acl videocache_deny_dom dstdomain .download.youporn.com .static.blip.tv acl dontrewrite url_regex redbot\.org \.php acl getmethod method GET storeurl_access deny dontrewrite storeurl_access deny !getmethod storeurl_access deny videocache_deny_dom storeurl_access allow videocache_allow_url storeurl_access allow videocache_allow_dom storeurl_access deny all storeurl_rewrite_program /etc/squid/storeurl.pl storeurl_rewrite_children 7 storeurl_rewrite_concurrency 10 acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id) acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \ storeurl_access allow store_rewrite_list storeurl_access deny all refresh_pattern -i \.flv$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.mp3$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.mp4$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.swf$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.gif$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.jpg$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.jpeg$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims refresh_pattern -i \.exe$ 43200 9980% 43200 override-expire override-lastmod reload-into-ims # 1 year = 525600 mins, 1 month = 43200 mins, 1 day = 1440 refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 43200 80% 43200 override-expire override-lastmod reload-into-ims refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 43200 80% 43200 override-expire override-lastmod reload-into-ims refresh_pattern \.(ico|video-stats) 43200 80% 43200 override-expire override-lastmod refresh_pattern \.etology\? 43200 80% 43200 override-expire ignore-reload ignore-no-cache refresh_pattern galleries\.video(\?|sz) 43200 80% 43200 override-expire ignore-reload ignore-no-cache refresh_pattern brazzers\? 43200 80% 43200 override-expire ignore-reload ignore-no-cache refresh_pattern \.adtology\? 43200 80% 43200 override-expire ignore-reload ignore-no-cache refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 43200 20% 43200 override-expire ignore-reload ignore-auth negative-ttl=40320 max-stale=10 refresh_pattern ^.*safebrowsing.*google 43200 80% 43200 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 43200 80% 43200 override-expire ignore-reload ignore-private refresh_pattern ytimg\.com.*\.jpg 43200 80% 43200 override-expire ignore-reload refresh_pattern images\.friendster\.com.*\.(png|gif) 43200 80% 43200 override-expire ignore-reload refresh_pattern garena\.com 43200 80% 43200 override-expire reload-into-ims refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 43200 80% 43200 override-expire ignore-reload refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 43200 80% 43200 ignore-no-cache override-expire override-lastmod refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 43200 80% 43200 reload-into-ims override-expire ignore-private refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 43200 80% 43200 reload-into-ims ignore-no-cache ignore-reload override-expire refresh_pattern ^http:\/\/www.onemanga.com.*\/ 43200 80% 43200 reload-into-ims ignore-no-cache ignore-reload override-expire refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 43200 80% 43200 override-expire override-lastmod #images facebook refresh_pattern -i \.facebook.com.*\.(jpg|png|gif) 43200 80% 43200 ignore-reload override-expire ignore-no-cache refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3) 43200 80% 43200 ignore-reload override-expire ignore-no-cache refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png) 43200 80% 43200 ignore-reload override-expire ignore-no-cache refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 43200 80% 43200 ignore-reload override-expire ignore-no-cache #All File refresh_pattern -i \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims refresh_pattern -i \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern ^ftp: 10080 95% 43200 override-lastmod reload-into-ims refresh_pattern . 1440 95% 43200 override-lastmod reload-into-ims
Save& Exit.
Now create storeurl.pl which will be used to pull youtube video from cache.
touch /etc/squid/storeurl.pl
chmod +x /etc/squid/storeurl.pl
Now edit this file and paste the following contents.
nano /etc/squid/storeurl.pl
#!/usr/bin/perl
# $Rev$
# by chudy_fernandez@yahoo.com
# Updates at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
$|=1;
while (<>) {
@X = split;
$X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];
# compatibility for old cached get_video?video_id
if (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";
# youtube HD itag=22
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=22).*?\&(id=[a-zA-Z0-9]*)/) {
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $2 . "&" . $3 . "\n";
# youtube Normal screen always HD itag 35, Normal screen never HD itag 34, itag=18 <--normal?
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=[0-9]*).*?\&(id=[a-zA-Z0-9]*)/) {
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $3 . "\n";
} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";
#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;×tamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";
#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";
# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";
#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";
} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";
#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $3 . $5 . "\n";
#...spicific servers end here.
#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
print $x . "http://photos.ak.fbcdn.net/" . $1 . "\n";
#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2 . "." . $3 . "\n";
#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/" . $3 . "\n";
#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/" . $y[1] . "\n";
#generic http://variable.domain.com/path/filename."ext" or "exte" with or withour "?"
} elsif (m/^http:\/\/(.*)([^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{3,4})(\?.*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-A-]+[0-9-]+)|(.*cdn.*)|(.*cache.*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
# generic http://variable.domain.com/...
} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 . "\n";
# spicific extention that ends with ?
} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)\?(.*)/) {
print $x . "http://" . $1 . "/" . $2 . "." . $3 . "\n";
# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";
} else {
print $x . $_ . "\n";
}
}
Save & Exit.
Now create cache dir and assign proper permission to proxy user
mkdir /cache1
chown proxy:proxy /cache1
chmod -R 777 /cache1
Now initialize squid cache directories by
squid -z
You should see Following message
Creating Swap Directories
After this, start SQUID service by
service squid start
Now Test your youtube video, and test it again and you will see the cache hit in squid logs, you can view them by
tail -f /var/log/squid/access.log | grep HIT
More information can be found at
http://aacable.wordpress.com/tag/aacable-howto-cache-youtube/
Advertisement
Dear bro how to solve this….
root@khan-desktop:~# squid -z
2012/01/20 01:17:52| Creating Swap Directories
FATAL: Failed to make swap directory /cache1: (13) Permission denied
Squid Cache (Version 2.7.STABLE7): Terminated abnormally.
CPU Usage: 0.000 seconds = 0.000 user + 0.000 sys
Maximum Resident Size: 3536 KB
Page faults with physical i/o: 0
wbr,
NASIR
Comment by NASIR — January 20, 2012 @ 12:19 AM
It means you didn’t read the instructions provided with the config file.
You have to assign proper permission to /cache1 dir so that it can be writable by squid proxy user.
If you are running Ubuntu, issue this command
mkdir /cache1
chown proxy:proxy /cache1
chmod 777 /cache1
Then run
squid -z
Then it will create cache dir successfully without any error. (provided you don’t have any config mistakes)
Comment by Syed Jahanzaib / Pinochio~:) — January 20, 2012 @ 10:45 AM
Syed i had the same problem like NASIR ,
i solved it by typing :
chmod -R 777 /cache1/
because it was trying to create a folder under /cache1/dir1/dir2 like this …
now it’s OK
thx for this gr8 tutorial .
I like ur blog and i’m visiting everyday for new tutorials …
Comment by Nori — January 20, 2012 @ 6:33 PM
Yeah, it was mentioned in the article too, but some times people are in too hurry to implement things that they skip few steps or do it blindly without modifying things according to distribution OS or there network.
Comment by Syed Jahanzaib / Pinochio~:) — January 21, 2012 @ 9:11 PM
Dear Syed,
I followed your article and already implemented in our network. Within two days we are getting very good hit. Now I want to made a replica of the same configuration (without the cache) and setup another proxy server to distribute HTTP load of various packages into specific proxy server. As earlier done for proxy1, I added a mangle rule in Mikrotik 3.30 router, and then marked DSCP = 12 for zph_local 0×30. It works brilliantly for users who are redirected to (proxy1) and getting cache from (proxy1). Mikrotik marks those packets and send cache_hit data in LAN speed. But for users who are in proxy2, getting cache data from proxy1, I can see in the access.log that I got hit from SIBLING_HIT, and therefore I set zph_sibling 0×30 to mark those packets in mikrotik, but I don’t get LAN speed.
I believe something is wrong is the squid …
Do you have any suggestions?
Comment by Saiful Alam — January 22, 2012 @ 3:13 AM
ya its working fine….. thanks Syed Jahanzaib. you are really great…
keep it up..
wish you all the best..
wbr,
Muhammad Nasir Javed
Comment by Muhammad Nasir Javed — January 26, 2012 @ 11:35 AM
Assalam-o-Alaikum
Shah gee My name is Muhammad Imran Khan
I read you blog and successfully I had configured my squid 2.7 also test it
working fine
my question is how can i enable downloading cache in squid
Kindly help me My e-mail is
imran_niazi_2004@yahoo.com
Regards:)
Comment by Muhammad Imran Khan — February 1, 2012 @ 5:32 PM
If you are using good refresh pattern, squid by will cache all contents those are cacheable.
However if you download via IDM or any other download accelerator, it will not be cached by default. once it is downloaded by browser , other user can download it from the cache eitehr using browse or IDM download
Comment by Syed Jahanzaib / Pinochio~:) — February 2, 2012 @ 10:40 AM
What do you thing would be a good hardware to use as squid if we have a 200users and almost all the month trafic that the users uses with just 1.7TB DOWNLOAD and 400GB UPload within a month .
But on the server excluding the .flv files and .mp4 files and files that are bigger than 30MB .
In this case what hardware do u prefer to me to use it.
( b.th.w i would like to use any dell or hp computer , can u tell me which one can i use but without needing to get to much energy like servers with 2800W because sometimes i need to work with invertors without having energy on our base. I mean which model of computer do u prefer …)
Comment by Nori — February 2, 2012 @ 10:03 PM
Any powerful server usually requires higher power consumption.
For about 200 Users, Any Dual core with 3 Ghz or Quad core will do more then enough.
The main focus should be on RAM and HDD, CPU is less concerned issue.
Adding 4 or 16 GB is an good idea to get good performance from squid. Add 2 HDD, one for OS and second dedicated for the CACHE.
Comment by Syed Jahanzaib / Pinochio~:) — February 2, 2012 @ 10:59 PM
Hello i have a problem, and I don’t know what’s wrong, can you help me, difference from tour config is only IPaddress, and different dir for cache
here is whats happening::::
2012/02/06 00:48:26| Starting Squid Cache version 2.7.STABLE9 for x86_64-pc-linux-gnu…
2012/02/06 00:48:26| Process ID 8837
2012/02/06 00:48:26| With 1024 file descriptors available
2012/02/06 00:48:26| Using epoll for the IO loop
2012/02/06 00:48:26| Performing DNS Tests…
2012/02/06 00:48:26| Successful DNS name lookup tests…
2012/02/06 00:48:26| DNS Socket created at 0.0.0.0, port 54307, FD 6
2012/02/06 00:48:26| Adding domain enet.rs from /etc/resolv.conf
2012/02/06 00:48:26| Adding nameserver 192.168.2.129 from /etc/resolv.conf
2012/02/06 00:48:26| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| User-Agent logging is disabled.
2012/02/06 00:48:26| Referer logging is disabled.
2012/02/06 00:48:26| logfileOpen: opening log /var/log/squid/access.log
2012/02/06 00:48:26| logfileOpen: opening log /var/log/squid/access.log
2012/02/06 00:48:26| Swap maxSize 174080000 + 1048576 KB, estimated 13471428 objects
2012/02/06 00:48:26| Target number of buckets: 673571
2012/02/06 00:48:26| Using 1048576 Store buckets
2012/02/06 00:48:26| Max Mem size: 1048576 KB
2012/02/06 00:48:26| Max Swap size: 174080000 KB
2012/02/06 00:48:26| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2012/02/06 00:48:26| logfileOpen: opening log /var/log/squid/store.log
2012/02/06 00:48:26| Rebuilding storage in /var/spool/squid (DIRTY)
2012/02/06 00:48:26| Using Least Load store dir selection
2012/02/06 00:48:26| Current Directory is /var/spool
2012/02/06 00:48:26| Loaded Icons.
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| Accepting transparently proxied HTTP connections at 192.168.1.202, port 3128, FD 21.
2012/02/06 00:48:26| HTCP Disabled.
2012/02/06 00:48:26| WCCP Disabled.
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
2012/02/06 00:48:26| Ready to serve requests.
2012/02/06 00:48:26| WARNING: store_rewriter #1 (FD 7) exited
2012/02/06 00:48:26| WARNING: store_rewriter #2 (FD 8) exited
2012/02/06 00:48:26| WARNING: store_rewriter #3 (FD 9) exited
2012/02/06 00:48:26| WARNING: store_rewriter #4 (FD 10) exited
2012/02/06 00:48:26| Too few store_rewriter processes are running
2012/02/06 00:48:26| ALERT: setgid: (1) Operation not permitted
FATAL: The store_rewriter helpers are crashing too rapidly, need help!
i really don’t know what to do anymore… Thanks
Comment by nenad — February 6, 2012 @ 4:52 AM
There might be some problem with the storeurl.pl content, either its not copy pasted correctly.
First try without storeurl.pl
IF it works ok , then Try to create storeurl.pl from following URL.
http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/
Comment by Syed Jahanzaib / Pinochio~:) — February 6, 2012 @ 8:53 AM
Thanks, it was problem with storeurl.pl
Now squid starts, and another thing i have to comment out log parameters, because squid don’t understand it…
Now Ill test it…
Thanks again
Comment by nenad — February 6, 2012 @ 11:43 PM
Another very strange thing…
When using LUSCA, while system runs everything is ok, youtube is caching… but after I reboot ubuntu, i have same error like i mention above, then i create again storeurl.pl and lusca is starting normal… Do you maybe know why this is happening??? And ofc because of this problem, lusca cannot start on boot… Do you have any idea
Thanks for helping me!
Comment by nenad — February 7, 2012 @ 5:14 AM
I have updated the squid.conf, It was wordpress who changed the code with special characters, , It really annoys me sometimes
any how check it again,
Comment by Syed Jahanzaib / Pinochio~:) — February 7, 2012 @ 9:23 AM
and another thing, with this log format how to make sarg to create report….
Thanks
Comment by nenad — February 7, 2012 @ 6:47 AM
I have used this log format with SARG and it works fine.
Sometimes WORDPRESS really messes with the codes pasted, sorry for inconvenience , I have re pasted the config, check it again.
Comment by Syed Jahanzaib / Pinochio~:) — February 7, 2012 @ 9:16 AM
SARG CONFIG
http://aacable.wordpress.com/2011/10/21/howto-install-sarg-squid-analysis-report-generator-in-ubuntu/
Comment by Syed Jahanzaib / Pinochio~:) — February 7, 2012 @ 9:21 AM
same thing, when rebooting ubuntu lusca wont start, manual starting result errors, and then just re create storeurl.pl and it starts normaly…
I didnt manage to get sarg create log
and yes another thing, i get reciving this message : clientNatLookup: NF getsockopt(SO_ORIGINAL_DST) failed: (92) Protocol not available
I dont use NAT for proxy, ubuntu knows all routes
Thanks a lot man…
Best regards
Comment by nenad — February 7, 2012 @ 4:35 PM
This message means that Squid received a request but the kernel has no NAT
tracking information about it’s IP address.
Comment by Syed Jahanzaib / Pinochio~:) — February 8, 2012 @ 12:10 AM
my networks are 172.16.x.x / 255.255.0.0 and from that address Squid gets requests, and i bootup and in active configuration i have ip route on eth1 address 172.16.0.0 mask 255.255.0.0 gateway x.x.x.x I think that is enough but maybe not…
Comment by nenad — February 8, 2012 @ 12:15 AM
hi any news about youtube error?
Comment by tom — February 10, 2012 @ 9:47 AM
chuddy said:create a redirect that will remove the “&range=xxx-xxx”
Comment by tom — February 10, 2012 @ 9:48 AM
hmm not able to understand, an example would be useful.
Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2012 @ 10:23 AM
i am using ubuntu-11.10 i386
i implemented this config ( i had to remove all the “ ” in your code, because when i copy it it shows up a lot of ’s such as: refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 43200 80% 43200 ignore-no-cache override-expire override-lastmod reload-into-ims
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 )
Anyway, squid is now running and seems to be working, BUT it does not seem to be caching the youtube videos.
With one video i got a 500 internal server error and the others just does not cache, even though i see the following:
root@eesa-server:~# tail -f /var/log/squid/access.log | grep HIT
192.168.1.45 – - [19/Feb/2012:16:47:11 +0200] “GET http://o-o.preferred.mweb-jnb1.v23.lscache4.c.youtube.com/generate_204? HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [19/Feb/2012:16:47:37 +0200] “GET http://o-o.preferred.mweb-jnb1.v23.lscache4.c.youtube.com/generate_204? HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [19/Feb/2012:16:47:39 +0200] “GET http://clients1.google.com/generate_204 HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [19/Feb/2012:16:47:53 +0200] “GET http://www.youtube.com/watch? HTTP/1.1″ 500 3143 TCP_NEGATIVE_HIT:NONE
What could be the problem?
My server ip: 192.168.1.28 and my ip is 192.168.1.45 ( i am testing it on a lan fist)
Comment by Eesa — February 19, 2012 @ 7:52 PM
Hmm I have tested this config at various networks and it works fine.
To copy script , you will see icon at script like this “” , click on it, and new window will appear and you will see RAW code.
HIT shows your video are caching fine.
Have you setup any Queue for speed limitation ?
At ubuntu box, open terminal and issue following command
ps aux | grep squid
Check if you are able to see 5-7 storeurl.pl entries
Comment by Syed Jahanzaib / Pinochio~:) — February 20, 2012 @ 9:34 AM
I used the source button and copied and repasted the code into both the squid.conf and the storeurl, i restarted squid but still no caching of youtube is taking place
Here is the output of ps aux | grep squid
root@eesa-server:/home/eesa# ps aux | grep squid
proxy 2105 0.3 1.3 13884 6780 ? Ssl 20:01 0:00 /usr/sbin/squid -N -D
proxy 2106 0.0 0.2 3948 1444 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2107 0.0 0.2 3948 1440 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2108 0.0 0.2 3948 1444 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2109 0.0 0.2 3948 1440 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2110 0.0 0.2 3948 1440 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2111 0.0 0.2 3948 1440 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 2112 0.0 0.2 3948 1444 ? Ss 20:01 0:00 /usr/bin/perl /etc/squid/storeurl.pl
root@eesa-server:/home/eesa#
I use gedit instead of nano to edit the files, could that cause an issue?
Here is the exact output of what i do
( i get a few Gtk-WARNINGs when using gedit but i dont think they
are of any importance as i always see them, they look like this:
(gedit:1795): Gtk-WARNING **: Attempting to store changes into `/root/.local/share/recently-used.xbel’, but failed: Failed to create file ‘/root/.local/share/recently-used.xbel.BS4CAW’: No such file or directory)
eesa@eesa-server:~$ sudo su
[sudo] password for eesa:
root@eesa-server:/home/eesa# gedit /etc/squid/squid.conf
then i paste config
root@eesa-server:/home/eesa# gedit /etc/squid/storeurl.pl
then i paste config
–
but still i can see that its not serving the video from the cache, because i see my internet is being used and the video is loading at internet speed, not LAN speed.
here is the output of: tail -f /var/log/squid/access.log | grep HIT when i open the same video over and over again
root@eesa-server:/home/eesa# tail -f /var/log/squid/access.log | grep HIT
192.168.1.45 – - [20/Feb/2012:20:11:09 +0200] “GET http://clients1.google.com/generate_204 HTTP/1.1″ 204 273 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [20/Feb/2012:20:11:36 +0200] “GET http://clients1.google.com/generate_204 HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [20/Feb/2012:20:11:36 +0200] “GET http://o-o.preferred.mweb-jnb1.v18.lscache3.c.youtube.com/generate_204? HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [20/Feb/2012:20:11:37 +0200] “GET http://clients1.google.com/generate_204 HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [20/Feb/2012:20:11:56 +0200] “GET http://o-o.preferred.mweb-jnb1.v18.lscache3.c.youtube.com/generate_204? HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
192.168.1.45 – - [20/Feb/2012:20:11:59 +0200] “GET http://clients1.google.com/generate_204 HTTP/1.1″ 204 274 TCP_NEGATIVE_HIT:NONE
i really cant understand it, should i downgrade the ubuntu version?
Comment by Eesa — February 20, 2012 @ 11:14 PM
You don’t need to downgrade your ubuntu.
I have tested this configuration at various places with always success ratio.
Any how you may try this and check the results.
http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/
Comment by Syed Jahanzaib / Pinochio~:) — February 21, 2012 @ 9:38 AM
Oh i forgot to mention, i dont have any mikrotik queues set, this is in a LAN environment for testing purpose
Comment by Eesa — February 20, 2012 @ 11:20 PM
as salaam alaikum All i follow all steps but its not caching any think place help me
Comment by ali — February 23, 2012 @ 3:22 AM
Have you tried this ?
http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/
Comment by Syed Jahanzaib / Pinochio~:) — February 23, 2012 @ 9:09 AM
When you say” Make sure you have setup proper internet connection in Ubuntu BOX.” do you mean anything specific?
or is it enough that i can access the internet from the ubuntu box?
Comment by Eesa — February 23, 2012 @ 1:11 PM
by proper internet connection, means just simple internet access , nothing specific.
Comment by Syed Jahanzaib / Pinochio~:) — February 23, 2012 @ 1:38 PM
sac int one caching http and other formats files but youtube video not caching, and not working on transparent mode ?
Comment by ali — February 24, 2012 @ 3:27 AM