Syed Jahanzaib Personal Blog to Share Knowledge !

April 21, 2014

Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue [April, 2014 , zaib]


LAST UPDATED: 22nd April, 2014 / 0800 hours pkst

Youtube caching is working as of 20th June,  2014

[1st version > 11th January, 2011]

.

Following web sites are tested and working good 🙂

.

If this method helps you, please post your comment.

 

.

 What is LUSCA / SQUID ?

LUSCA is an advance version or Fork of  SQUID 2. The Lusca project aims to fix the shortcomings in the Squid-2. It also supports a variety of clustering protocols. By Using it, you can cache some dynamic contents that you previously can’t do with the squid.

For example [jz]
#  Video Cachingi.e Youtube / tube etc . . .
#  Windows / Linux Updates / Anti-virus , Anti-Malware i.e. Avira/ Avast / MBAM etc . . .
#  Well known sites i.e. facebook / google / yahoo etch. etch.
#  Download caching mp3’s/mpeg/avi etc . . .

Advantages of Youtube Caching   !!!

In most part of the world, bandwidth is very expensive, therefore it is (in some scenarios) very useful to Cache Youtube videos or any other flash videos, so if one of user downloads video / flash file , why again the same user or other user can’t download the same file from the CACHE, why he sucking the internet pipe for same content again n again?
Peoples on same LAN ,sometimes watch similar videos. If I put some youtube video link on on FACEBOOK, TWITTER or likewise , and all my friend will  watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my lan users / friends / zaib.

This is the reason why I wrote this article. I have implemented Ubuntu with LUSCA/ Squid on it and its working great, but to achieve some results you need to have some TB of storage drives in your proxy machine.

Disadvantages of Youtube Caching   !!!

The chances, that another user will watch the same video, is really slim. if I search for something specific on youtube, i get more then hundreds of search results for same video. What is the chance that another user will search for the same thing, and will click on the same link / result? Youtube hosts more than 10 million videos. Which is too much to cache anyway. You need lot of space to cache videos. Also accordingly you will be needing ultra modern fast hardware with tons of RAM to handle such kind of cache giant. anyhow Try it

We will divide this article in following Sections

1#  Installing SQUID / LUSCA in UBUNTU
2#  Setting up SQUID / LUSCA Configuration files
3#  Performing some Tests, testing your Cache HIT
4# Using ZPH TOS to deliver cached contents to clients vai mikrotik at full LAN speed, Bypassing the User Queue for cached contents.

.

.

1#  Installing SQUID / LUSCA in UBUNTU

I assume your ubuntu box have 2 interfaces configured, one for LAN and second for WAN. You have internet sharing already configured. Now moving on to LUSCA / SQUID installation.

Here we go ….

Issue following command. Remember that its a one go command (which have multiple commands inside it so it may take a bit long to update, install and compile all required items)


apt-get update &&
apt-get install gcc -y &&
apt-get install build-essential -y &&
apt-get install libstdc++6 -y &&
apt-get install unzip -y &&
apt-get install bzip2 -y &&
apt-get install sharutils -y &&
apt-get install ccze -y &&
apt-get install libzip-dev -y &&
apt-get install automake1.9 -y &&
apt-get install acpid -y &&
apt-get install libfile-readbackwards-perl -y &&
apt-get install dnsmasq -y &&
cd /tmp &&
wget -c http://wifismartzone.com/files/linux_related/lusca/LUSCA_HEAD-r14942.tar.gz &&
tar -xvzf LUSCA_HEAD-r14942.tar.gz &&
cd /tmp/LUSCA_HEAD-r14942 &&
./configure \
--prefix=/usr \
--exec_prefix=/usr \
--bindir=/usr/sbin \
--sbindir=/usr/sbin \
--libexecdir=/usr/lib/squid \
--sysconfdir=/etc/squid \
--localstatedir=/var/spool/squid \
--datadir=/usr/share/squid \
--enable-async-io=24 \
--with-aufs-threads=24 \
--with-pthreads \
--enable-storeio=aufs \
--enable-linux-netfilter \
--enable-arp-acl \
--enable-epoll \
--enable-removal-policies=heap \
--with-aio \
--with-dl \
--enable-snmp \
--enable-delay-pools \
--enable-htcp \
--enable-cache-digests \
--disable-unlinkd \
--enable-large-cache-files \
--with-large-files \
--enable-err-languages=English \
--enable-default-err-language=English \
--enable-referer-log \
--with-maxfd=65536 &&
make &&
make install

EDIT SQUID.CONF FILE

Now edit SQUID.CONF file by using following command

nano /etc/squid/squid.conf

and Delete all previously lines , and paste the following lines.

Remember following squid.conf is not very neat and clean , you will find many un necessary junk entries in it, but as I didn’t had time to clean them all, so you may clean them as per your targets and goals.

Now Paste the following data … (in squid.conf)


#######################################################
## Squid_LUSCA configuration Starts from Here ...     #
## Thanks to some INDO friendsfor sharing Configs     #
## Syed.Jahanzaib / 22nd April, 2014                  #
## https://aacable.wordpress.com / aacable@hotmail.com #
#######################################################

# HTTP Port for SQUID Service
http_port 8080 transparent
server_http11 on

# Cache Pee, for parent proxy if you ahve any, or ignore it.
#cache_peer x.x.x.x parent 8080 0

# Various Logs/files location
pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid/
error_directory /usr/share/squid/errors/English
icon_directory /usr/share/squid/icons
mime_table /etc/squid/mime.conf
access_log daemon:/var/log/squid/access.log squid
cache_log none
#debug_options ALL,1 22,3 11,2 #84,9
referer_log /var/log/squid/referer.log
cache_store_log none
store_dir_select_algorithm  round-robin
logfile_daemon /usr/lib/squid/logfile-daemon
logfile_rotate 1

# Cache Policy
cache_mem 6 MB
maximum_object_size_in_memory 0 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA

minimum_object_size 0 KB
maximum_object_size 10 GB
cache_swap_low 98
cache_swap_high 99

# Cache Folder Path, using 5GB for test
cache_dir aufs /cache-1 5000 16 256

# ACL Section
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8            # RFC1918 possible internal network
acl localnet src 172.16.0.0/12        # RFC1918 possible internal network
acl localnet src 192.168.0.0/16        # RFC1918 possible internal network
acl localnet src 125.165.92.1        # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80                # http
acl Safe_ports port 21                # ftp
acl Safe_ports port 443                # https
acl Safe_ports port 70                # gopher
acl Safe_ports port 210                # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488                # gss-http
acl Safe_ports port 591                # filemaker
acl Safe_ports port 777                # multiling http
acl CONNECT method CONNECT
acl purge method PURGE
acl snmppublic snmp_community public

acl range dstdomain .windowsupdate.com
range_offset_limit -1 KB range

#===========================================================================
#    Loading Patch
acl DENYCACHE urlpath_regex \.(ini|ui|lst|inf|pak|ver|patch|md5|cfg|lst|list|rsc|log|conf|dbd|db)$
acl DENYCACHE urlpath_regex (notice.html|afs.dat|dat.asp|patchinfo.xml|version.list|iepngfix.htc|updates.txt|patchlist.txt)
acl DENYCACHE urlpath_regex (pointblank.css|login_form.css|form.css|noupdate.ui|ahn.ui|3n.mh)$
acl DENYCACHE urlpath_regex (Loader|gamenotice|sources|captcha|notice|reset)
no_cache deny DENYCACHE

range_offset_limit 1 MB !DENYCACHE
uri_whitespace strip

#===========================================================================
#    Rules to block few Advertising sites
acl ads url_regex -i .youtube\.com\/ad_frame?
acl ads url_regex -i .(s|s[0-90-9])\.youtube\.com
acl ads url_regex -i .googlesyndication\.com
acl ads url_regex -i .doubleclick\.net
acl ads url_regex -i ^http:\/\/googleads\.*
acl ads url_regex -i ^http:\/\/(ad|ads|ads[0-90-9]|ads\d|kad|a[b|d]|ad\d|adserver|adsbox)\.[a-z0-9]*\.[a-z][a-z]*
acl ads url_regex -i ^http:\/\/openx\.[a-z0-9]*\.[a-z][a-z]*
acl ads url_regex -i ^http:\/\/[a-z0-9]*\.openx\.net\/
acl ads url_regex -i ^http:\/\/[a-z0-9]*\.u-ad\.info\/
acl ads url_regex -i ^http:\/\/adserver\.bs\/
acl ads url_regex -i !^http:\/\/adf\.ly
http_access deny ads
http_reply_access deny ads
#deny_info http://yoursite/yourad,htm ads
#==== End Rules: Advertising ====

strip_query_terms off

acl yutub url_regex -i .*youtube\.com\/.*$
acl yutub url_regex -i .*youtu\.be\/.*$
logformat squid1 %{Referer}>h %ru
access_log /var/log/squid/yt.log squid1 yutub

# ==== Custom Option REWRITE ====
acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback.*id)

acl store_rewrite_list urlpath_regex \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)\?
acl store_rewrite_list urlpath_regex \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)\?
acl store_rewrite_list urlpath_regex \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)\?
acl store_rewrite_list urlpath_regex \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)\?
acl store_rewrite_list urlpath_regex \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)\?
acl store_rewrite_list urlpath_regex \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)\?
acl store_rewrite_list urlpath_regex \.(htm|html|mhtml|css|js)\?

acl store_rewrite_list_web url_regex ^http:\/\/([A-Za-z-]+[0-9]+)*\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_web_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.com doubleclick\.net

acl store_rewrite_list_path urlpath_regex \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)$
acl store_rewrite_list_path urlpath_regex \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)$
acl store_rewrite_list_path urlpath_regex \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)$
acl store_rewrite_list_path urlpath_regex \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)$
acl store_rewrite_list_path urlpath_regex \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)$
acl store_rewrite_list_path urlpath_regex \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)$
acl store_rewrite_list_path urlpath_regex \.(htm|html|mhtml|css|js)$

acl getmethod method GET

storeurl_access deny !getmethod
#this is not related to youtube video its only for CDN pictures
storeurl_access allow store_rewrite_list_web_CDN
storeurl_access allow store_rewrite_list_web store_rewrite_list_path
storeurl_access allow store_rewrite_list
storeurl_access deny all
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 10
storeurl_rewrite_concurrency 40
# ==== End Custom Option REWRITE ====

#===========================================================================
#    Custom Option REFRESH PATTERN
#===========================================================================
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv\?|\.fid\?) 43200 99% 43200 override-expire ignore-reload ignore-must-revalidate ignore-private
refresh_pattern -i (get_video\?|videoplayback\?|videodownload\?) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
# -- refresh pattern for specific sites -- #
refresh_pattern ^http://*.jobstreet.com.*/.* 720 100% 10080 override-expire override-lastmod ignore-no-cache
refresh_pattern ^http://*.indowebster.com.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-auth
refresh_pattern ^http://*.21cineplex.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-auth
refresh_pattern ^http://*.atmajaya.*/.* 720 100% 10080 override-expire ignore-no-cache ignore-auth
refresh_pattern ^http://*.kompas.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.theinquirer.*/.* 720 100% 10080 override-expire ignore-no-cache ignore-auth
refresh_pattern ^http://*.blogspot.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.wordpress.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache
refresh_pattern ^http://*.photobucket.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.tinypic.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.imageshack.us/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.kaskus.*/.* 720 100% 28800 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://www.kaskus.com/.* 720 100% 28800 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.detik.*/.* 720 50% 2880 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.detiknews.*/*.* 720 50% 2880 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://video.liputan6.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://static.liputan6.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.friendster.com/.* 720 100% 10080 override-expire override-lastmod ignore-no-cache ignore-auth
refresh_pattern ^http://*.facebook.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://apps.facebook.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.fbcdn.net/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://profile.ak.fbcdn.net/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://static.playspoon.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://cooking.game.playspoon.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern -i http://[^a-z\.]*onemanga\.com/? 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://media?.onemanga.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.yahoo.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.google.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.forummikrotik.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.linux.or.id/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
# -- refresh pattern for extension -- #
refresh_pattern -i \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(htm|html|mhtml|css|js)(\?.*|$) 1440 90% 86400 override-expire ignore-reload reload-into-ims
#===========================================================================
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern ^ftp: 10080 95% 10080 override-lastmod reload-into-ims
refresh_pattern . 0 20% 10080 override-lastmod reload-into-ims

http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

http_access allow localnet
http_access allow all
http_access deny all

icp_access allow localnet
icp_access deny all
icp_port 0

buffered_logs on

acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast

acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

forwarded_for off
header_access From deny all
header_access Server deny all
header_access Link deny all
header_access Via deny all
header_access X-Forwarded-For deny all
httpd_suppress_version_string on

shutdown_lifetime 10 seconds

snmp_port 3401
snmp_access allow snmppublic all
dns_timeout 1 minutes

dns_nameservers 8.8.8.8 8.8.4.4

fqdncache_size 5000    #16384
ipcache_size 5000    #16384
ipcache_low 98
ipcache_high 99
log_fqdn off
log_icp_queries off
memory_pools off

maximum_single_addr_tries 2
retry_on_error on

icp_hit_stale on

strip_query_terms off

query_icmp on
reload_into_ims on
emulate_httpd_log off
negative_ttl 0 seconds
pipeline_prefetch on
vary_ignore_expire on
half_closed_clients off
high_page_fault_warning 2
nonhierarchical_direct on
prefer_direct off
cache_mgr aacable@hotmail.com
cache_effective_user proxy
cache_effective_group proxy
visible_hostname proxy.zaib
unique_hostname syed_jahanzaib
cachemgr_passwd none all
client_db on
max_filedescriptors 8192

# ZPH config Marking Cache Hit, so cached contents can be delivered at full lan speed via MT
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

.

.

SOTEURL.PL

Now We have to create an important file name storeurl.pl , which is very important and actually it does the
main job to redirect and pull video from cache.

touch /etc/squid/storeurl.pl
chmod +x /etc/squid/storeurl.pl
nano /etc/squid/storeurl.pl

Now paste the following lines, then Save and exit.

#!/usr/bin/perl
#######################################################
## Squid_LUSCA storeurl.pl starts from Here ...     #
## Thanks to Mr. Safatah [INDO] for sharing Configs  #
## Syed.Jahanzaib / 22nd April, 2014 #
## https://aacable.wordpress.com / aacable@hotmail.com #
#######################################################
$|=1;
while (<>) {
@X = split;
$x = $X[0] . " ";
##=================
## Encoding YOUTUBE
##=================
if ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
@itag = m/[&?](itag=[0-9]*)/;
@CPN = m/[&?]cpn\=([a-zA-Z0-9\-\_]*)/;
@IDS = m/[&?]id\=([a-zA-Z0-9\-\_]*)/;
$id = &GetID($CPN[0], $IDS[0]);
@range = m/[&?](range=[^\&\s]*)/;
print $x . "http://fathayu/" . $id . "&@itag@range\n";
} elsif ($X[1] =~ m/(youtube|google).*videoplayback\?/ ){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
print $x . "http://fathayu/";
# ==========================================================================
#    VIMEO
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/av\.vimeo\.com\/\d+\/\d+\/(.*)\?/) {
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/pdl\.vimeocdn\.com\/\d+\/\d+\/(.*)\?/) {
print $x . "http://fathayu/" . $1 . "\n";
# ==========================================================================
#    DAILYMOTION
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/proxy-[0-9]{1}\.dailymotion\.com\/(.*)\/(.*)\/video\/\d{3}\/\d{3}\/(.*.flv)/) {
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/vid[0-9]\.ak\.dmcdn\.net\/(.*)\/(.*)\/video\/\d{3}\/\d{3}\/(.*.flv)/) {
print $x . "http://fathayu/" . $1 . "\n";
# ==========================================================================
#   YIMG
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://fathayu/" . $3 . "\n";
# ==========================================================================
#   YIMG DOUBLE
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://fathayu/" . $3 . "\n";
# ==========================================================================
#   YIMG WITH &sig=
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://fathayu/" . $y[0] . ".yimg.com/" . $y[1] . "\n";
# ==========================================================================
#    YTIMG
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/i[1-4]\.ytimg\.com(.*)/) {
print $x . "http://fathayu/" . $1  . "\n";
# ==========================================================================
#   PORN Movies
# ==========================================================================
} elsif (($X[1] =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";
#   Domain/path/.*/path/filename
} elsif (($X[1] =~ /fucktube/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?[^\/]*\/[^\/]*)\/(.*)\/([^\/]*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5,$6);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "/" . $y[3] . "." . $y[4] . "\n";
#   Like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($X[1] =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
} elsif (($u =~ /tube8|redtube|hardcore-teen|pornhub|tubegalore|xvideos|hostedtube|pornotube|redtubefiles/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";
#   acl store_rewrite_list url_regex -i \.xvideos\.com\/.*(3gp|mpg|flv|mp4)
#   refresh_pattern -i \.xvideos\.com\/.*(3gp|mpg|flv|mp4) 1440 99% 14400 override-expire override-lastmod ignore-no-cache ignore-private reload-into-ims ignore-must-revalidate ignore-reload store-stale
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/.*\.xvideos\.com\/.*\/([\w\d\-\.\%]*\.(3gp|mpg|flv|mp4))\?.*/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\d]+\.[\d]+\.[\d]+\.[\d]+\/.*\/xh.*\/([\w\d\-\.\%]*\.flv)/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\d]+\.[\d]+\.[\d]+\.[\d]+.*\/([\w\d\-\.\%]*\.flv)\?start=0/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/.*\.youjizz\.com.*\/([\w\d\-\.\%]*\.(mp4|flv|3gp))\?.*/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\.\%]*\.keezmovies[\w\d\-\.\%]*\.com.*\/([\w\d\-\.\%]*\.(mp4|flv|3gp|mpg|wmv))\?.*/){
print $x . "http://fathayu/" . $1 . $2 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\.\%]*\.tube8[\w\d\-\.\%]*\.com.*\/([\w\d\-\.\%]*\.(mp4|flv|3gp|mpg|wmv))\?.*/) {
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\.\%]*\.youporn[\w\d\-\.\%]*\.com.*\/([\w\d\-\.\%]*\.(mp4|flv|3gp|mpg|wmv))\?.*/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\.\%]*\.spankwire[\w\d\-\.\%]*\.com.*\/([\w\d\-\.\%]*\.(mp4|flv|3gp|mpg|wmv))\?.*/) {
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\.\%]*\.pornhub[\w\d\-\.\%]*\.com.*\/([[\w\d\-\.\%]*\.(mp4|flv|3gp|mpg|wmv))\?.*/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif ($X[1] =~ m/^http:\/\/[\w\d\-\_\.\%\/]*.*\/([\w\d\-\_\.]+\.(flv|mp3|mp4|3gp|wmv))\?.*cdn\_hash.*/){
print $x . "http://fathayu/" . $1 . "\n";
} elsif (($X[1] =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://fathayu/" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";
} elsif (($X[1] =~ /fucktube/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?[^\/]*\/[^\/]*)\/(.*)\/([^\/]*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5,$6);
$y[0] =~ s/(([a-zA-Z]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . "http://fathayu/" . $y[0] . $y[1] . "/" . $y[2] . "/" . $y[3] . "." . $y[4] . "\n";
} elsif (($X[1] =~ /media[0-9]{1,5}\.youjizz/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?\.[^\/]*)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-Z]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . "http://fathayu/" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
# ==========================================================================
#   Filehippo
# ==========================================================================
} elsif (($X[1] =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://fathayu/" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
} elsif (($X[1] =~ /filehippo/) && (m/^http:\/\/(.*?)(\.[^\/]*?)\/(.*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-Z]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
print $x . "http://fathayu/" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
} elsif ($X[1] =~ m/^http:\/\/.*filehippo\.com\/.*\/([\d\w\%\.\_\-]+\.(exe|zip|cab|msi|mru|mri|bz2|gzip|tgz|rar|pdf))/){
$y=$1;
for ($y) {
s/%20//g;
}
print $x . "http://fathayu//" . $y . "\n";
} elsif (($X[1] =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://fathayu/" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
# ==========================================================================
#   4shared preview
# ==========================================================================
} elsif ($X[1] =~ m/^http:\/\/[a-z]{2}\d{3}\.4shared\.com\/img\/\d+\/\w+\/dlink__2Fdownload_2F.*_3Ftsid_(\w+)-\d+-\w+_26lgfp_3D1000_26sbsr_\w+\/preview.mp3/) {
print $x . "http://fathayu/" . $3 . "\n";

} else {
print $x . $X[1] . "\n";
}
}

sub GetID
{
$id = "";
use File::ReadBackwards;
my $lim = 200 ;
my $ref_log = File::ReadBackwards->new('/var/log/squid/referer.log');
while (defined($line = $ref_log->readline))
{
if ($line =~ m/.*youtube.*\/watch\?.*v=([a-zA-Z0-9\-\_]*).*\s.*id=$IDS[0].*/){
$id = $1;
last;
}
if ($line =~ m/.*youtube.*\/.*cpn=$CPN[0].*[&](video_id|docid|v)=([a-zA-Z0-9\-\_]*).*/){
$id = $2;
last;
}
if ($line =~ m/.*youtube.*\/.*[&?](video_id|docid|v)=([a-zA-Z0-9\-\_]*).*cpn=$CPN[0].*/){
$id = $2;
last;
}
last if --$lim <= 0;
}
if ($id eq ""){
$id = $IDS[0];
}
$ref_log->close();
return $id;
}
### STOREURL.PL ENDS HERE ###

INITIALIZE CACHE and LOG FOLDER …

Now create CACHE folder (here I used test in local drive)

# Log Folder and assign permissions
mkdir /var/log/squid
chown proxy:proxy /var/log/squid/

# Cache Folder
mkdir /cache-1
chown proxy:proxy /cache-1
#Now initialize cache dir by
squid -z

START SQUID SERVICE

Now start SQUID service by following command

squid 

and look for any error or termination it may get. If all ok, just press enter few times and you will be back to command prompt.
to verify if squid is running ok issue following command and look for squid instance, there should be 10+ instance for the squid process

ps aux |grep squid

 Something like below …

123

TIP:

To start SQUID Server in Debug mode, to check any erros, use following

squid -d1N

TEST time ….

It’s time to hit the ROAD and do some tests….

 

YOUTUBE TEST

Open Youtube and watch any Video. After complete download, Check the same video from another client. You will notice that it download very quickly (youtueb video is saved in chunks of 1.7mb each, so after completing first chunk, it will stop, if a user continue to watch the same video, it will serve second chunk and so on , you can watch the bar moving fast without using internet data.
As Shown in the example Below . . .

lusca_test.

YT cache HIT.

 

.

FILEHIPPO TEST [ZAIB]

FILHIPPO

As Shown in the example Below . . .

FILHIPPO

MUSIC DOWNLOAD TEST

Now test any music download. For example Go to
http://www.apniisp.com/songs/indian-movie-songs/ladies-vs-ricky-bahl/690/1.html
As Shown in the example Below . . .

and download any song , after its downloaded, goto 2nd client pc, and download the same song, and monitor the Squid access LOG. You will see cache hit TPC_HIT for this song.

As Shown in the example Below . . .

EXE / PROGRAM  DOWNLOAD TEST

Now test any .exe file download.
Goto http://www.rarlabs.com and download any package. After Download completes, goto 2nd client pc , and download the same file again. and monitor the Squid access LOG. You will see cache hit TPC_HIT for this file.

As Shown in the example Below . . .

SQUID LOGS

Other methods are as follow (I will update following (squid 2.7) article soon)

https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/
https://aacable.wordpress.com/2012/08/13/youtube-caching-with-squid-nginx/

.

.

.

MIKROTIK with SQUID/ZPH: how to bypass Squid Cache HIT object with Queues Tree in RouterOS 5.x and 6.x

.

zph

.

Using Mikrotik, we can redirect HTTP traffic to SQUID proxy Server, We can also control user bandwidth, but its a good idea to deliver the already cached content to user at full lan speed, that’s why we setup cache server for, to save bandwidth and have fast browsing experience , right :p , So how can we do it in mikrotik that cache content should be delivered to users at unlimited speed, no queue on cache content. Here we go.

By using ZPH directives , we will mark cache content, so that it can later pick by Mikrotik.

Basic requirement is that Squid  must be running in transparent mode, can be done via iptables and squid.conf directives.
I am using UBUNTU squid 2.7 , (in ubuntu , apt-get install squid will install squid 2.7 by default which is gr8 for our work)
Add these lines in SQUID.CONF

#===============================================================================
#ZPH for SQUID 2.7 (Default in ubuntu 10.4) / Syed Jahanzaib aacable@hotmail.com
#===============================================================================
tcp_outgoing_tos 0x30 lanuser [lanuser is ACL for local network, change it to match your's]
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

Use following if you have squid 3.1.19


#======================================================
#ZPH for SQUID 3.1.19 (Default in ubuntu 12.4) / Syed Jahanzaib aacable@hotmail.com
#======================================================

# ZPH for Squid 3.1.19
qos_flows local-hit=0x30

That’s it for SQUID, Now moving on to Mikrotik box ,
Add following rules,

# Marking packets with DSCP (for MT 5.x) for cache hit content coming from SQUID Proxy

/ip firewall mangle add action=mark-packet chain=prerouting disabled=no dscp=12 new-packet-mark=proxy-hit passthrough=no comment="Mark Cache Hit Packets / aacable@hotmail.com"
/queue tree add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=pmark packet-mark=proxy-hit parent=global-out priority=8 queue=default

# Marking packets with DSCP (for MT 6.x) for cache hit content coming from SQUID Proxy

/ip firewall mangle add action=mark-packet chain=prerouting comment="MARK_CACHE_HIT_FROM_PROXY_ZAIB" disabled=no dscp=12 new-packet-mark=proxy passthrough=no
/queue simple
add max-limit=100M/100M name="ZPH-Proxy Cache Hit Simple Queue / Syed Jahanzaib >aacable@hotmail.com" packet-marks=zph-hit priority=1/1 target="" total-priority=1

MAKE SURE YOU MOVE THE SIMPLE QUEUE ABOVE ALL OTHER QUEUES 😀
.

Now every packet which is marked by SQUID CACHE_HIT, will be delivered to user at Full lan speed, rest of traffic will be restricted by user Queue.

TROUBLESHOOTING:

the above config is fully tested with UBUNTU SQUID 2.7 and FEDORA 10 with LUSCA

Make sure your squid is marking TOS for cache hit packets. You can check it via TCPDUMP

tcpdump -vni eth0 | grep ‘tos 0×30′

(eht0 = LAN connected interface)

Can you see something like ???

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:25:07.961722 IP (tos 0×30, ttl 64, id 45167, offset 0, flags [DF], proto TCP (6), length 409)
20:25:07.962059 IP (tos 0×30, ttl 64, id 45168, offset 0, flags [DF], proto TCP (6), length 1480)
192 packets captured
195 packets received by filter
0 packets dropped by kernel
_________________________________

Regard’s
SYED JAHANZAIB

 

 

Regard’s
SYED JAHANZAIB
https://aacable.wordpress.com

428 Comments »

  1. ur amazing gr8 , thx for everything …

    Comment by Nori — January 11, 2012 @ 1:39 PM

    • Hi,suddenly lusca stops responding but still running.i have to restart in-order to work again.
      please help

      Comment by Tarek El Ali — March 16, 2012 @ 6:42 PM

  2. awesome!! great sharing

    Comment by faizan — January 11, 2012 @ 3:49 PM

  3. Nice project zaib bhai….

    Comment by adeel — January 11, 2012 @ 3:52 PM

  4. spectacular! thanks man

    Comment by benk — January 12, 2012 @ 3:05 PM

  5. assalamualaikum
    hi brother I’ll try your tutorial, thank’s 😉

    Comment by bambang — January 13, 2012 @ 9:33 AM

  6. plz yar aap mary number per kar karain aap bohat sari baatian kar ni hai

    Comment by ashraf — January 14, 2012 @ 12:23 AM

  7. number hai 3217701039

    Comment by ashraf — January 14, 2012 @ 12:24 AM

  8. nice post mr. zaib… i have squid 3.1 on slitaz (small linux distro with 30MB iso and serve repository for squid 3.1). is that possible if i used your lusca conf to my quid? or i must compile lusca source on slitaz by my self? thx…

    Comment by mbahsimak — January 18, 2012 @ 12:13 PM

    • I am sure it will work with squid 3.1
      Just use the config and the storeurl.pl and I think it will work even with SLITAZ.

      Comment by Syed Jahanzaib / Pinochio~:) — January 18, 2012 @ 12:38 PM

      • thx again mr. zaib 🙂
        i’ll try it

        Comment by mbahsimak — January 18, 2012 @ 1:10 PM

      • Hello Mr.SYED JAHANZAIB
        I check your perl script and squid configure and all work fine.
        But I have some question i have approximately 300-350 throughput what about hardware suggest , now i use HP server with 192 GB RAM , 8*300 GB Hard disk , 2960 intel cpu with 6-Core , I use debian 6.0.7 with your LUSCA squid . That is fine Hardware and software or you suggest any develop ?

        Our main problem that squid use one core of CPU and that make that core load reach 90%-95% in some time ,can you help me for use other 5-core of my CPU by squid.

        I have one question for your squid configure . can i cancel the LOG write such as
        access_log daemon:/var/log/squid/access.log squid
        referer_log /var/log/squid/referer.log
        acl yutub url_regex -i .*youtube\.com\/.*$
        acl yutub url_regex -i .*youtu\.be\/.*$
        logformat squid1 %{Referer}>h %ru
        access_log /var/log/squid/yt.log squid1 yutub

        Can I remove the above lines from squid configure?

        Regards,
        Basharmshafik

        Comment by basharmshafik — April 30, 2014 @ 12:12 PM

      • lusca/squid
        Here are the FACTS that you should know about lusca (squid fork). Lusca is based on squid, and it is inherently a non-smp and single thread only,
        the only multi-thread on squid are the I/O disk function,so using any multi-core cpu is totally useless with lusca/squid

        Comment by Syed Jahanzaib / Pinochio~:) — April 30, 2014 @ 1:33 PM

      • Dear Mr.SYED JAHANZAIB

        thank you very much for your reply.
        >Here are the FACTS that you should know about lusca (squid fork). Lusca is based on squid, and it is inherently a non-smp and single thread only,
        >the only multi-thread on squid are the I/O disk function,so using any multi-core cpu is totally useless with lusca/squid

        But you have any suggest about my problem
        >squid use one core of CPU and that make that core load reach 90%-95% in some time ,can you help me for use other 5-core of my CPU by squid.

        because developed of Intel CPU will increase number of cores not increase CPU frequency .
        I heard there is a way to install squid in every core that meaning install six squid every one on core or install six operating system every one on core , you have any information about that?

        waiting for reply, and thank you again

        Regards,
        Basharmshafik

        Comment by basharmshafik — May 3, 2014 @ 12:16 PM

    • squid 3.1 dont support storeurl_rewrite yet

      Comment by Daniel Echizen — April 14, 2012 @ 9:46 PM

    • Salaam Syed i hope you are doing well, i followed your example on how to cache squid youtube at this site https://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/#postcomment

      it is working very good but i am facing some problems on login page of mikrotik

      i have installed ubuntu and installed squid as your example shows and i set up wan lan server and i have connected mikrotik on lan if i enable the proxy from mikrotik nat the proxy is bypassing the login page of mikrotik they dont get the login page. please help

      Comment by Faisal — July 10, 2012 @ 5:15 PM

  9. Everything works fine, however, I found 1 thing … May be its a squid bug…I am not sure…But files downloaded with browser download manager is cached, and files downloaded with Internet Download Manager is not cached, even though I am using transparent proxy for both. Please check this in your server, and see if the same happens for u as well….

    Comment by Saiful — January 18, 2012 @ 10:44 PM

    • I have observed this behavior.
      But currently I don’t have any solution for this.

      Comment by Syed Jahanzaib / Pinochio~:) — January 18, 2012 @ 11:14 PM

      • No browser addon issue.
        Its not CDN in this case, I am trying to download a simple static mp3 file. Its actually because IDM uses multiple parts downloads … and therefore its not caching. But is there any solution other than that?

        Comment by Saiful — January 19, 2012 @ 2:10 AM

      • I have observed this behavior.
        But currently I don’t have any solution for this.

        Comment by Syed Jahanzaib / Pinochio~:) — January 19, 2012 @ 11:17 AM

  10. Nice job! only thing which I dont se is file in /etc/init.d …
    So how to autostart lusca ?

    Comment by Damir — January 25, 2012 @ 12:00 AM

    • Various Solutions:

      # You can have some caching functionality even on simple squid 2.7 (even with default installation in ubuntu by apt-get install squid), You don’t need hard coded LUSCA to achieve youtube or dynamic contents, Use the following guide.
      https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/

      # For simply running LUSCA after the computer started but before a user logs in, you can simply edit the script /etc/rc.local which is meant to solve exactly this task.
      For example: use this to start squid on start-up
      echo “/usr/local/squid/sbin/squid -D” >> /etc/rc.local

      # Add it in your CRON cron daemon, one of the predefined cron time hooks is @reboot, which naturally runs when the system starts/reboot. Run crontab -e to edit your crontab file, and add a line:
      @reboot /your/command/here

      # Add LUSCA as a service , then you can control it as normal squid service. There are lot of guides available for this purpose. hit the road, i mean google it 🙂

      Comment by Syed Jahanzaib / Pinochio~:) — January 25, 2012 @ 11:08 AM

      • Thanks ! I make cron job.
        anyway I was notice one error for example when waching direct video like this everythin is fine :

        but when try to wach same video with apendix like this:

        I have error …
        This is probably related to storeurl.pl …. do you have some update ?

        Comment by Damir — January 29, 2012 @ 1:18 AM

      • I also faced this issue.
        I googled and found out that This is some what related to TTL. Google it , and if you found some working solution, do let me know 🙂

        Comment by Syed Jahanzaib / Pinochio~:) — January 29, 2012 @ 9:46 AM

  11. This is really a great explanation
    But I have a question
    Why take squid speed Internet, despite all of the client take only half the speed of the Internet

    thanks for help me

    Comment by malak — January 25, 2012 @ 2:18 AM

  12. sorry Syed Jahanzaib

    iam not good in english

    i have mikrotik loadbalance —— squid —— mikrotik hotspot
    and i have limted download in 512K bytes and iam only in my network
    when i download and take 512K speed squid have all triffice

    see this PIC

    this mean the squid server get all triffice 😦

    can you help me

    Comment by malak — January 25, 2012 @ 10:02 PM

    • The Mikrotik is showing that wlan1 pppoe client is taking 4 mbps bandwidth. but the IDM is showing its taking only 512 kbps.

      Check your queuing section.

      Comment by Syed Jahanzaib / Pinochio~:) — January 26, 2012 @ 11:01 AM

  13. Hey thanks for the tutorial, I have not really make the squid to connect to the mikrotik could guide me to make the connection you make would be greatly appreciated

    Comment by esteban — January 26, 2012 @ 9:11 AM

  14. Dear Syed Jahanzaib,
    Do configuraton files need to be changed if iam using squid through one nic ?

    Comment by Mohamad Hassan — January 26, 2012 @ 9:12 AM

  15. Dear Syed Jahanzaib,

    lam very very very sorry

    Is it normal squid take all internet speed ?????? when user download any file and user have limted speed ????

    i see when one only in my network and do download he take 512K only but in the same time squid take 4M

    i want squid take same speed of client

    explanation

    i want to do this

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan , wan have speed = 512 K
    in Load balace Local = 512
    ————————————————————–

    but when i setup squid

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan = 512K , wan have speed = 4M ???
    in Load balace Local = 4M ???

    squid download the file in all speed internet it`s make Browsing soooooo slowly

    see this Pic

    this pic Immediately when i download squid take allllllllll speed but i have only 512K???

    thanks for help

    Comment by malak — January 26, 2012 @ 3:33 PM

  16. Please mikrotik setting…..thanks

    Comment by slams — January 26, 2012 @ 6:41 PM

  17. Dear Syed Jahanzaib,

    lam very very very sorry

    Is it normal squid take all internet speed ?????? when user download any file and user have limted speed ????

    i see when one only in my network and do download he take 512K only but in the same time squid take 4M

    i want squid take same speed of client

    explanation

    i want to do this

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan , wan have speed = 512 K
    in Load balace Local = 512
    ————————————————————–

    but when i setup squid

    when user make download
    in mikrotik lan , wan have speed = 512K
    in squid lan = 512K , wan have speed = 4M ???
    in Load balace Local = 4M ???

    squid download the file in all speed internet it`s make Browsing soooooo slowly

    see this Pic

    this pic Immediately when i download squid take allllllllll speed but i have only 512K???

    thanks for help

    plzzzzz help mee

    Comment by malak — January 28, 2012 @ 2:48 AM

  18. Dear Sayed,
    I’m using squid 2.7 with storeurl.pl
    its not caching youtube and the dynamic pages its giving this error :

    root@proxy:~# squid -z
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).*’: ignore-auth  
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^http:\/\/images|pics|thumbs[0-9]\.                    ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘^http:\/\/www.onemanga.com.*\/                         ‘: ignore-no-cache 
    2012/02/01 06:12:08| parseConfigFile: squid.conf:209 unrecognized: ‘refresh_pattern ‘
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)      ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar) ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)        ‘: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob)’: ignore-no-cache 
    2012/02/01 06:12:08| parse_refreshpattern: Unknown option ‘\.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv)’: ignore-no-cache 

    please help me, i appreciate it
    Best regards . Mohamad

    Comment by Mohamad Hassan — February 1, 2012 @ 9:14 AM

  19. Search and remove following directive
    ignore-no-cache
    ignore-auth

    squid 2.7 dont support few refresh pattern.
    LUSCA is better in some cases.
    any how try it

    Comment by Syed Jahanzaib / Pinochio~:) — February 1, 2012 @ 12:55 PM

  20. Dear Syed Jahanzaib,

    I just tried caching using SQUID / Lusca, as described above, but the videos are not even playing for the first time!
    Could you help me fix this?

    Best regards

    Comment by laziz — February 7, 2012 @ 6:58 PM

    • If not playing even for the first time, there could be problem with the storerul.pl
      try to start lusca/squid without storeurl.pl and check if it works.

      Comment by Syed Jahanzaib / Pinochio~:) — February 7, 2012 @ 11:17 PM

      • I was also notice this behavior … after few refeshes … somethimes is open … maybe is smarter to contact this original author of storeurl to ask him for some update ?

        Comment by damir — February 8, 2012 @ 1:35 AM

      • There is no absolute fix for this problem at a moment, At least I am not aware of it 🙂
        If you find one, do let me know.

        and yes contacting author of storeurl.pl is the best method to get the authentic solution or reply.

        Comment by Syed Jahanzaib / Pinochio~:) — February 8, 2012 @ 9:02 AM

  21. Well I was sent email to autor … without sucess … they was working now with videocashe and sugested me to swich to videocache which is to much expensive for my requierments cca 300 $ 😦

    Comment by damir — February 9, 2012 @ 1:17 AM

    • Yes that’s the problem that videocache is paid solution and costly one too, also after 6 months, you have to pay half of the amount for the renewal or updated version :s

      Comment by Syed Jahanzaib / Pinochio~:) — February 9, 2012 @ 8:56 AM

  22. sir video cache ka lya pay karni para gi kya

    Comment by sheeraz — February 9, 2012 @ 9:55 AM

  23. Jahanzaib bahi

    Lusca install kar nay kay liyeh kiya phelay say muj ko squid install cahiyeh hoga ya nahii

    Comment by Asif Lai — February 15, 2012 @ 9:17 PM

    • You don’t need SQUID to install LUSCA
      LUSCA works like SQUID, infact it is squid’s more advance version you can say.

      Comment by Syed Jahanzaib / Pinochio~:) — February 15, 2012 @ 10:51 PM

      • Hi do you have a version for windows xp???? or can you make a version of squid/lusca for windows xp thx. and more power….

        Comment by tapitz — February 16, 2012 @ 12:47 PM

      • There are win32 base squid version available on internet.
        But You can’t get real performance from windows base SQUID.
        So its better if you do it in Linux.
        Ubuntu is good for beginners and starters.

        Comment by Syed Jahanzaib / Pinochio~:) — February 17, 2012 @ 8:55 AM

  24. jahanzeb bhai

    main nay app kay way try kiya magar jab main youtube ka koi bhi clip play karta hoon to kuch tume baad woh error show kar deta hain
    aisa kiyon…

    Comment by Asif Lai — February 16, 2012 @ 4:43 PM

  25. jahanzeb bahi

    eik aur baat jab mian cache server ko restart karta hoon to sari config remove hojatii is ka koi hall batain

    Comment by Asif Lai — February 16, 2012 @ 4:55 PM

  26. jahanzaib bhai i have dual core CPU with 2.50GHz processor. 4 GB RAM and 1 TB harddisk. to kya main easily use kr skta hun jo ap nay caching btaei hai ?

    Comment by SHAFQAT FARHAN — February 26, 2012 @ 9:46 PM

  27. thank your for this useful tutorial syed, im just wondering how you could install cache manager and/or webmin

    Comment by cervelo — February 29, 2012 @ 3:28 PM

  28. Hi Sir can you make a delaypools options… thx and more power….

    Comment by tapitz07 — March 5, 2012 @ 3:04 PM

  29. hi dear
    hope that you are doing fine

    thanks for this good job , but there is a problem with youtube.com videos

    please see this picture and let me know how to fix it

    waiting for the replay ASAP

    Comment by hamid — March 6, 2012 @ 2:43 PM

  30. Hi,

    i’m very thankful for this deep explanation.

    I managed to install it and it is working fine, but wanted to enable SNMP, so I can draw some graphs.
    First, uncommented the lines in the squid.conf that were # before, but it give error when trying to start it, then I realized that it is compiled with disable-snmp, so i made ./configure –enable-snmp but when tried to compile again I got errors

    /temp/LUSCA_HEAD-r14809/src/snmp_core.c:323: undefined reference to `theOutSnmpConnection’
    collect2: ld returned 1 exit status
    make[3]: *** [squid] Error 1
    make[3]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make[2]: *** [all-recursive] Error 1
    make[2]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make[1]: *** [all] Error 2
    make[1]: Leaving directory `/temp/LUSCA_HEAD-r14809/src’
    make: *** [all-recursive] Error 1

    Have you tried to compile it with enable-snmp?

    Comment by x-man — March 10, 2012 @ 1:54 PM

  31. range_offset_limit 512 KB
    can you define it please and how can it be good for improving the performance??

    Comment by mohamed koubar — March 11, 2012 @ 7:05 AM

  32. Can you help me… Im configure Lucas Squid , almost all is ok , my problem is with the use ACL , i need use filter arp , filter by hours , filter by contents , in this case i create a files with all mac address of the lan , and do the same for the other files , but i dont have nothing answer , with mac in the file or within a can navagate, in other version of squid all that is ok, im compile de Lucas package with arp support.What is my mistake

    Comment by Elfoman — March 13, 2012 @ 8:32 AM

  33. Hello Sir,

    I have try your tutorial but .. i have some faced problem below
    10.201.21.10 – – [14/Mar/2012:10:27:44 +0700] “GET http://cdn.api.twitter.com/1/urls/count.json? HTTP/1.1″ 200 513 TCP_MISS:DIRECT
    10.201.21.10 – – [14/Mar/2012:10:27:46 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:27:47 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:27:51 +0700] “POST http://ubuntu-indonesia.com/forums/ubbthreads.php HTTP/ 1.1″ 200 655 TCP_MISS:DIRECT
    10.201.21.10 – – [14/Mar/2012:10:27:51 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:27:53 +0700] “GET http://0-38.channel.facebook.com/pull? HTTP/1.1″ 200 159 1 TCP_MISS:DIRECT
    10.201.21.10 – – [14/Mar/2012:10:27:56 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:27:56 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:27:58 +0700] “GET http://0-38.channel.facebook.com/pull? HTTP/1.1″ 200 159 2 TCP_MISS:DIRECT
    10.201.21.10 – – [14/Mar/2012:10:27:59 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:28:00 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE
    10.201.21.10 – – [14/Mar/2012:10:28:02 +0700] “HEAD error:invalid-request HTTP/0.0” 400 236 TCP_DENIED:NONE

    But proxy is running and can cache would you like to help me please

    regards

    Kamb33ng

    Comment by kamb33ng — March 14, 2012 @ 8:37 AM

  34. Thanks for such information…. Greet

    Comment by mp4 videos — March 19, 2012 @ 2:37 AM

  35. thank you very much

    Comment by sara — March 22, 2012 @ 6:12 PM

  36. Too few store_rewriter processes are running
    FATAL: The store_rewriter helpers are crashing too rapidly, need help!

    whenever i start
    /usr/local/squid/sbin/squid -d1

    Comment by aws — March 27, 2012 @ 5:37 PM

    • This usually happens when there is a typo / paste mistake in storeurl.pl

      Try to use newer version and make sure you copy the right source code.

      Comment by Syed Jahanzaib / Pinochio~:) — March 27, 2012 @ 10:28 PM

      • i have same problem too. wriiten like this :
        WARNING: store_rewriter #1 (FD 7) exited
        WARNING: store_rewriter #2 (FD 8) exited
        WARNING: store_rewriter #3 (FD 9) exited
        WARNING: store_rewriter #10 (FD 10) exited
        Too few store_rewriter processes are running
        FATAL: The store_rewriter helpers are crashing too rapidly, need help!

        I use Ubuntu server 11.04. And i copy paster the squid.conf and storeurl.pl files using view source and then paste it to a file in notepad in windows. Then i copy them to the directory as written above.

        Is that causing this problem ? Please help

        Comment by djynny — June 30, 2012 @ 10:40 AM

    • I think i found the problem, but still don’t know how to solve it. If you open the /var/log/squid/cache.log you will find the problem.
      I got this error :
      helperOpenServer: Starting 7 ‘storeurl.pl’ processes
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      ipcCreate: /etc/squid/storeurl.pl : (2) No such file or directory
      I think it related with file permission. Because i am a linux newbie, I hope someone could help.

      Syed, I hope when you are not busy you can help us..

      Best Regards,

      Djemmy

      Comment by djynny — July 13, 2012 @ 12:50 PM

      • The problem is solved. After try to reinstalling the ubuntu server 5 times i found that the problem because of i copy paste the squid.conf and storeurl.pl via notepad in windows. Although I save as it as all files and put the file extension with .conf and .pl it seem that ubuntu didn’t recognize the .pl file that created by notepad. That was my own mistake…..
        Syed thanks for all the share knowledge here.

        Cheers,
        Djemmy

        Comment by djynny — July 16, 2012 @ 10:15 AM

      • You are welcome !

        Comment by Syed Jahanzaib / Pinochio~:) — July 16, 2012 @ 11:29 AM

  37. Thanks for the post…….

    I’m using Fedora for forwarding my mikrotik port 80 request of browsing…. Wil I use all these stuff in Fedora or I have to use debain for this especially….. I thought zph methos not works in fedora

    Comment by kashif khan — March 27, 2012 @ 10:02 PM

    • Most of my posts related to Linux are Ubuntu base, because I found Ubuntu most easy to use Linux flavor. You or other may found any other flavor more suitable for various reasons, As I have my own 🙂
      By default FEDORA installs SQUID 3 which deoesn’t support ZPH by default, you have to compile Squid 2.7 on Fedora in order to use ZPH or Use LUSCA on FED , as LUSCA supports ZPH well.

      On the other hand UBUNTU installs squid 2.7 which supports ZPH so no extra work is required.

      It’s all about personnel preference, in past, FEDORA was my first choice, but from past 2 year,s UBUNTU have become First 🙂 but don’t worry, You can achieve your goal by using any flavor of Linux, doesn’t matter its Ubuntu , Fedora, Centos , FreeBSD , or any OS with Linux as its base OS /. Kernel , it will work 🙂

      Comment by Syed Jahanzaib / Pinochio~:) — March 27, 2012 @ 10:19 PM

      • Thanks Syed Jhanzaib.. I install Ubuntu 10.4 and Everything works fine…. I ‘m using Mikrotik OS and zph working fine tooo………. Thanks a lot

        Comment by kashif — April 21, 2012 @ 11:49 PM

      • I am glad it helped you.

        Comment by Syed Jahanzaib / Pinochio~:) — April 22, 2012 @ 12:01 AM

  38. i’m using your source code !!!!

    and what’s new version you are talking about ????

    thanks for your reply

    Comment by aws — March 28, 2012 @ 3:52 AM

  39. Buenas noches, lo instale en mi equipo portátil para realizar varias pruebas pero soy novato, como hago para ver si esta funcionando y mi equipo tiene dirección ip asignada por dhcp, y también como detengo squid?

    Comment by tonyvzla — April 1, 2012 @ 5:17 AM

  40. hi im ask what is different lusca fmi and lusca head? so it lusca works on ubuntu server 10.10 64bit?

    Comment by Rinocomp Ajust — April 2, 2012 @ 4:33 PM

  41. i am new to the forum is very interesting your project and followed it step by step but when redirected to Lusca / squid will not let me navigate PORSIACASO mikrotik use which may be the error I would greatly appreciate it

    Comment by nickeliott — April 6, 2012 @ 9:41 AM

  42. what is PORSIACASO?

    Comment by Syed Jahanzaib / Pinochio~:) — April 9, 2012 @ 4:37 PM

  43. i have not seen yet somone adapt Content-Type: video/webm youtube using it now wide on most of the new video

    i hoop you cancame up with solution

    Comment by joe — April 11, 2012 @ 4:46 AM

  44. cache webm seen a little dificult.. can you help us?? by the way.. thanks for the solution for youtube.. works great!

    Comment by Daniel Echizen — April 13, 2012 @ 6:30 AM

  45. some news of webm .. need to test more but… for anyone to want.. try this.. set in squid.conf range_offset_limit -1, also change your refresh pattern to ensure that squid will cache webm file format

    Comment by Daniel Echizen — April 13, 2012 @ 6:52 AM

    • Have you tried it ?

      Comment by Syed Jahanzaib / Pinochio~:) — April 13, 2012 @ 9:20 AM

      • yeap.. working like a charm .. seens like video/webm keep a NaN range

        Comment by Daniel Echizen — April 13, 2012 @ 8:03 PM

      • just a tip.. to work correctly try to avoid caching redirect text/html content

        Comment by Daniel Echizen — April 13, 2012 @ 8:06 PM

      • Hmmm what impact you have seen for not to cache text/html contents ?

        Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 8:00 PM

      • redirect content from webm has no Cache-Control:no-cache so instead of webm video you’ll cache the redirect file which is empty. Which will also loop back to redirect content.

        Comment by Daniel Echizen — April 14, 2012 @ 10:07 PM

      • news.. the fix in here is exactly what im says.. http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube tested direct on source and working great! dont need to worry about text/html.. just patch with the fix in client_side.c

        Comment by Daniel Echizen — April 14, 2012 @ 10:15 PM

  46. jahanzaib bhai ubuntu main lan and wan kesay configure krtay hain ??

    Comment by Shafqat Farhan — April 13, 2012 @ 3:13 PM

  47. im having problems with sourceforge.net … when i try to redownload a file in another browser, the counter restarts and the download do not start properly, anyone hav this issue too?

    Comment by Daniel Echizen — April 13, 2012 @ 9:34 PM

    • Nops. However going through same WAN IP every time, it can cause access deny sometime.

      Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 7:59 PM

      • will see about that! thanks for the tip.. by the way.. did you get working 4shared preview files? i only get working download files.. but preview.mp3 dont..

        Comment by Daniel Echizen — April 14, 2012 @ 10:08 PM

  48. jahanzaib bhai mujh say ubuntu mainn lan and wan configure nhi ho raha . will u plz tell me .

    Comment by Shafqat Farhan — April 15, 2012 @ 12:34 PM

  49. Everything is working fine. but video cach nhi ho rahi . Kya masla ho skta hai?? youtube, metacafe etc etc. koi b cach nhi ho rahi . baqi softwares, mp3 files etc cach ho rahi hain

    Comment by Shafqat Farhan — April 16, 2012 @ 6:43 PM

  50. one big problem in youtube.. if you get a cached video and try to change dpi the video doesnt play anymore.. only the original dpi.. anyone hav a fix for this?

    Comment by Daniel Echizen — April 16, 2012 @ 9:52 PM

    • ps.: the problem is with the webm content.. range_offset to -1 cause this issue

      Comment by Daniel Echizen — April 16, 2012 @ 10:24 PM

      • These are the bugs with the storeurl.pl
        or you can say ” It comes with the Package ” 😉

        Comment by Syed Jahanzaib / Pinochio~:) — April 17, 2012 @ 8:45 AM

      • fixed forcing cache with quick_abort_min -1 in squid.conf.. for anyone that hav the same issue!

        Comment by Daniel Echizen — April 17, 2012 @ 10:00 AM

      • forcing cache from beginning* with quick_abort_min -1

        Comment by Daniel Echizen — April 17, 2012 @ 10:02 AM

  51. jahanzaib bhai u did’nt replye me

    Comment by Shafqat Farhan — April 17, 2012 @ 6:52 PM

  52. My cache is working now. But yay btaein k yay small videos ki caching ho jati hai but bari videos ki nhi hoti . Means jo videos 30 ya phr 35 seconds ki hon on ki to cache ho jati hai but jo 1 minute say above hun on ki caching nhi hoti . same with softwares . jo softwares approx 15 MB say baray hon on ki caching nhi hoti . Help Plz

    Comment by Shafqat Farhan — April 17, 2012 @ 9:11 PM

    • Check File Size limitation in squid.

      Comment by Syed Jahanzaib / Pinochio~:) — April 18, 2012 @ 12:42 PM

      • From where i can check the file size limitaion ? I mean in which variable in squid code ?

        Comment by Shafqat Farhan — April 18, 2012 @ 2:58 PM

      • minimum_object_size and maximum_object_size in squid.conf

        Comment by Daniel Echizen — April 19, 2012 @ 2:32 AM

  53. i have already tried this already minimum_object_size 0 bytes and maximum_object_size is 100 MB . but still i get the problem.

    Comment by Shafqat Farhan — April 19, 2012 @ 9:33 AM

  54. jahanzaib bhai waiting for your reply

    Comment by Shafqat Farhan — April 19, 2012 @ 6:14 PM

    • Hi, If you copy the content to you clipboard from above tutorial, you need to make sure that you remove all the %nbsp”. The code will not read correctly. I had the same problem at first.

      Comment by A.J. Hart — May 1, 2012 @ 7:31 PM

  55. HI, i have follow this tutorial but i have this error “root@debian:/home/debian# /usr/local/squid/sbin/squid -d1
    2012/04/19 01:46:16| ACL name ‘videocache_deny_dom’ not defined!
    FATAL: Bungled squid.conf line 160: storeurl_access deny videocache_deny_dom
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.”

    Comment by bmwfrs — April 22, 2012 @ 6:28 AM

    • remvoe all lines which have videocache_deny_dom

      Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 4:27 PM

    • Hi, If you copy the content to you clipboard from above tutorial, you need to make sure that you remove all the %nbsp”. The code will not read correctly. I had the same problem at first. Sorry about the previous comment, it was ment for bmwfrs

      Comment by A.J. Hart — May 1, 2012 @ 7:34 PM

  56. Is there any way to graph the squid cache perfomance???

    Comment by kashif khan — April 25, 2012 @ 3:57 PM

    • Yes I worked on MRTG graphs in depth, I made graphs for squid cache hit also, but unfortunately, I did this long time ago and I don’t have its cfg files. but its simple. Make sure your squid is snmp enabled, then by querying squid snmp oid, you can create graph for every instance of it. I will try to search for it on my local repositories.

      Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 4:37 PM

      • jahanzaib bhai ap nay btaya nhi k squid large file kesay cache krta hai ?

        Comment by Shafqat Farhan — April 26, 2012 @ 9:27 AM

  57. i have done the settings but IDM page k ilaawa kuch kuch pages ki caching ho rahi hay aur wo bhi all objects ko cache nahi karta kuch DIRECT khultay hien aur kuch HIT hotay hien…. Youtube se bhi yehee problem hay, kuch aik videos k ilaawa kisi video ki caching nahi ho rahi…. please jahanzaib bhai iska solution bataaein kia problem hay… sab commands without error install hui hay kisi mein koi error nahi aaya tha…

    Looking for you reply Sir….

    Comment by Ijaz Ahmed — April 27, 2012 @ 2:47 PM

    • Most IDM downloads will not fit into cache due to multipart chunks downloads.
      I haven’t tested squid youtube caching since few weeks. I will test it again and will let you know the results.

      Comment by Syed Jahanzaib / Pinochio~:) — April 28, 2012 @ 4:22 PM

      • sir i need to make cache of youtube and facebook. please tell me the method… and this script is not caching all the site I mean it saves some objects of a site and dont save many objects…

        Comment by Ijaz Ahmed — April 28, 2012 @ 5:27 PM

  58. from about 2 month ago youtube chache always tcp_miss, anyone have the solution?

    Comment by sevensvr — May 6, 2012 @ 7:07 AM

  59. Hi Syed,
    congrats for the excellent work!
    One question if I may…
    If I want to use an apache as a proxy for utube..and squid as cache..
    How can I tell squid to always try to find cached content first before trying to proxy?
    These rules are enough for apache I guess(I am only interested for embedded videos):
    RewriteRule ^/get_video_info(.*)$ http://www.youtube.com%{REQUEST_URI} [P,L]
    RewriteRule ^(.*)$ http://www.youtube.com/embed%{REQUEST_URI} [P,L]

    -Thanks

    Comment by Thanassis Zakopoulos — May 7, 2012 @ 3:16 AM

    • I have no idea for apache as youtube caching server.

      Youtube caching can be done using etiher storeurl.pl file, or python base scripts (which uses apache for youtube cache distribution) , Also this method is used in VIDEOCACHE plugin which cache youtube using pytin scripts with apache which is a paid addon.

      Comment by Syed Jahanzaib / Pinochio~:) — May 8, 2012 @ 3:18 PM

      • Hello Syed,

        Do you have any fix about the “51% youtube caching”, when you play a certain video in youtube then cached it, the next time it will be played is delayed or stucks at 51% then buffers again, but when i checked that certain video, its been already cached, the problem here is the delay.

        Thank you 🙂

        Comment by Incognito79 — May 12, 2012 @ 9:49 AM

      • No solution for it yet.

        Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2012 @ 8:57 AM

      • this is not a problem at all.. look at access.log.. youtube work with range files .. so you’ll cache alot of parts of the same video, you’ll buffer another part when playing part is about 90%, thats why u hav the delay on preload..but if you got hit on access.log its all fine.

        Comment by Daniel Echizen — May 14, 2012 @ 11:33 AM

  60. i success make cache youtube .. 😀

    Comment by fahri — May 12, 2012 @ 1:18 PM

    • Syed,

      Hi again, do you have the new solution for the youtube partial caching problem?…

      thank you

      Comment by Incognito79 — May 15, 2012 @ 10:49 AM

  61. no hard work cache youtube 😀

    Comment by fahri — May 12, 2012 @ 1:21 PM

  62. I did all sir. per squid server ka status check karta hu to squid stop/waiting py hi hota hy…… jub ky usy kafi dafa start kiya hy

    Comment by Muhammad Abdullah Butt — May 12, 2012 @ 8:07 PM

  63. Problem Solved! main ny /var/log/squid/store.log-access.log-cache.log ko chmod 777 ki to squid server start ho gai. eb baqi testing start karny laga hu.

    Comment by Muhammad Abdullah Butt — May 13, 2012 @ 1:01 AM

  64. Dear Sir,

    @karthik : can u please help me, if we check the log it is blank ” tail -f /var/log/squid/access.log | grep o-o”

    root@karthik:/var/log/squid# du -s -h *
    0 access.log
    12K cache.log
    0 store.log

    in microtik dns ip given same has squid ip.

    Comment by karthik — May 17, 2012 @ 7:29 PM

  65. ~ # tail -f /var/log/squid/access.log | grep o-o
    127.0.0.1 – – [22/May/2012:13:14:06 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/generate_204? HTTP/1.1″ 204 217 TCP_MISS:DIRECT
    127.0.0.1 – – [22/May/2012:13:14:06 +0700] “GET http://o-o.preferred.excelcom-cgk1.v17.lscache3.c.youtube.com/videoplayback? HTTP/1.1″ 200 873541 TCP_MISS:DIRECT
    127.0.0.1 – – [22/May/2012:13:14:08 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782153 TCP_HIT:NONE
    127.0.0.1 – – [22/May/2012:13:14:09 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782166 TCP_HIT:NONE
    127.0.0.1 – – [22/May/2012:13:14:27 +0700] “GET http://o-o.preferred.excelcom-cgk1.v20.lscache5.c.youtube.com/videoplayback? HTTP/1.1″ 200 1782166 TCP_HIT:NONE

    whats wrong TCP_MISS:DIRECT ?
    im install lusca on linux mint 12

    Comment by paydoel — May 22, 2012 @ 11:19 AM

  66. One billion thanks

    Comment by mahgobsoft — May 28, 2012 @ 4:27 PM

  67. After downloading the Ubuntu 10.4 LTS desktop and the work of keeping track of all the wonderful Arif unfortunately not out of Ubuntu to another computer science does not work with Squid on the same Ubuntu
    Please utilize a how the output from the Ubuntu desktop to another computer with Cache
    thanks very .

    Comment by mahgobsoft — May 28, 2012 @ 4:39 PM

  68. thanks for your amazing explication
    but its not working correctly with me
    i mean when i stop the line in squid.conf “storeurl_rewrite_program /etc/squid/storeurl.pl ” the proxy work but not cached youtube…
    when i run this line the proxy stop working
    can you help me plz

    Comment by Ismail — June 9, 2012 @ 1:01 AM

  69. its saying squid is already running but when i run youtube nothing is cached yet
    Do i have to configure squid.conf and put my ips??

    Comment by mungaz — June 13, 2012 @ 2:33 AM

  70. Dear Jhanzeb Bahi !!!!!!!

    I have 1 TB of HDD and it almost full with the cache drive for approx. 89%, what to do now.??? could you suggest me

    Comment by kashif khan — June 14, 2012 @ 10:25 AM

    • Well, either add another drive for caching support, or clear the current cache.

      Comment by Syed Jahanzaib / Pinochio~:) — June 14, 2012 @ 10:57 AM

      • After i plug a new device how to add or format this drive… well I dont even fromat after installation.. thats y im asking

        Comment by kashif khan — June 14, 2012 @ 3:31 PM

      • Jhanzaib bahi …. I added a new hard disk are reinstall the Ubuntu 10.4 again and after 4 days I saw this error and after every 2 hours my squid stops working…… Could you help me or suggest me….

        IN=eth1 OUT= MAC=00:19:db:55:52:7d:94:0c:6d:c8:4e:f3:08:00 SRC=66.220.149.94 DST=192.168.250.9 LEN=40 TOS=0x00 PREC=0x00 TTL=232 ID=19055 DF PROTO=TCP SPT=80 DPT=43107 WINDOW=0 RES=0x00 ACK RST URGP=0

        Comment by kashif khan — July 2, 2012 @ 8:11 PM

  71. help me please about youtube

    Comment by vickyajah — June 14, 2012 @ 2:25 PM

  72. sir , how to make autostart lusca head service on ubuntu 12.04 ?

    Comment by beruang — June 22, 2012 @ 12:46 PM

  73. AS-SALAM-WALEYKUM
    THIS IS REHMAT ALI GULWATING, HAVE TO INFORM THAT I INSTALL LUSCA/SQUID WITH MY MIKROTIK OS ALL WORK FINE ONLY 1 ISSUE IS THAT:
    WHEN ANY PC HIT YOUTUBE VIDEO IT PLAYS WELL AND COMPLETED NO ERROR BUT WHEN I REFRESH PAGE TO CHECK CACHE ONLY 1 SEGMENT OF VIDEO IS UTILIZED FROM LUSCA CACHE. DO U GET ANY UPDATE STOREURL.PL TO CACHE COMPLETE SEGMENTS OF YOUTUBE VIDEOS BECAUSE I USED UR LAST STOREURL.PL SCRIPT LINK: https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/ IF YES THEN PLEASE PLZ PLZ MAIL ME A LINK ON MY MAIL E-MAIL=BIG.BANG.NOW@GMAIL.COM.
    GOOD JOB KEEP IT UP ALLAH BLESS YOU. U R TRUELY DOING A PERFECT JOB.
    THANKING YOU,

    Comment by rehmat ali — June 26, 2012 @ 5:30 PM

  74. thanks alot sir i configure it is working great sir, but i have some problem as i try many times that i configure it but i can not get any results sir,
    i want to make transparent proxy,
    i can do browsing and everything by defining the proxy but i can not download anything by internet download manager.

    Comment by aliakber — July 16, 2012 @ 5:07 PM

  75. to put your (0x30 tcp_outgoing_tos all). By doing this the ZPH has no effect, ie mark everything as 0x30 and not just hit squid. And if you remove all the tcp_outgoing_tos 0x30 does not work. Any ideas?

    Comment by sadamssh — July 16, 2012 @ 10:53 PM

  76. I forgot to say that use DEBIAN

    Comment by sadamssh — July 16, 2012 @ 11:02 PM

  77. hi thank for your configuration
    squid will support form 3000 users?
    maximun how much it will support?

    Comment by karthik — August 1, 2012 @ 10:55 PM

    • Yes SQUID can support this number of users, but it really depend on the hardware you use. For 3000 users, you would be needing some latest hardware machine with quadx2 core cpus and lot of RAM.
      But putting big load on single machine, is not a wise idea, and consider to be a dad network design.
      Its better to divide proxy load on at least two machines.

      Comment by Syed Jahanzaib / Pinochio~:) — August 3, 2012 @ 7:30 AM

  78. Syed Jahanzaib Sir please help me mera ubuntu squid 5 ,6 hour bad cache khtm kar deta hai mery squid wly pc ki ye requirment hain Dell GX 280 Tower
    Processor 2.8 ghz
    Ram 1 GB
    Hard 120 GB Sata

    Comment by ahmad subhani — August 9, 2012 @ 12:14 PM

  79. assalamualaikum
    sir maine ye puchna tha k ye squid 10mb se zyada ki exe file cache nahi kar raha plz help me ……

    Comment by shani — August 12, 2012 @ 7:38 AM

  80. hello sir ,
    it work with squid3 ??

    Comment by Hussien — August 13, 2012 @ 2:52 AM

  81. assalamualaikum…
    please help me solved this problem
    2012/08/17 12:35:39| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-redhat-linux-gnu…
    2012/08/17 12:35:39| Process ID 1334
    2012/08/17 12:35:39| NOTICE: Could not increase the number of filedescriptors
    2012/08/17 12:35:39| With 1024 file descriptors available
    2012/08/17 12:35:39| Using epoll for the IO loop
    2012/08/17 12:35:39| Performing DNS Tests…
    2012/08/17 12:35:39| Successful DNS name lookup tests…
    2012/08/17 12:35:39| Adding nameserver 127.0.0.1 from squid.conf
    2012/08/17 12:35:39| Adding nameserver 221.132.112.8 from squid.conf
    2012/08/17 12:35:39| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2012/08/17 12:35:39| logfileOpen: opening log /var/log/squid/access.log
    2012/08/17 12:35:39| Swap maxSize 512000 + 8192 KB, estimated 40014 objects
    2012/08/17 12:35:39| Target number of buckets: 2000
    2012/08/17 12:35:39| Using 8192 Store buckets
    2012/08/17 12:35:39| Max Mem size: 8192 KB
    2012/08/17 12:35:39| Max Swap size: 512000 KB
    2012/08/17 12:35:39| logfileOpen: opening log /var/log/squid/store.log
    2012/08/17 12:35:39| AUFS: /cache1: log ‘/cache1/swap.state’ opened on FD 18
    2012/08/17 12:35:39| AUFS: /cache1: tmp log /cache1/swap.state.new opened on FD 18
    2012/08/17 12:35:39| Rebuilding storage in /cache1 (DIRTY)
    2012/08/17 12:35:39| Using Least Load store dir selection
    2012/08/17 12:35:39| Current Directory is /root
    2012/08/17 12:35:39| Loaded Icons.
    2012/08/17 12:35:39| commBind: Cannot bind socket FD 20 family 2 to 192.168.3.2 port 3128: (98) Address already in use
    FATAL: Cannot open HTTP Port

    Comment by widi — August 17, 2012 @ 10:43 AM

    • It shows that port 3128 is already occupied by another application, possibly another instance of squid or lusca is already running
      check it with

      ps aux |grep squid

      or

      netstat -tulpn | grep :8080

      first kill previous instance, then start new one,
      you can kill it by

      killall -9 squid (or app name)

      Comment by Syed Jahanzaib / Pinochio~:) — August 18, 2012 @ 10:17 PM

      • Thank you sir
        the problem has been found, for the problematic proxy port (3128) I changed the port 8080 results instantly

        Comment by widi — August 21, 2012 @ 9:04 AM

      • gr8

        Comment by Syed Jahanzaib / Pinochio~:) — August 22, 2012 @ 9:30 AM

  82. i am try squid configuration in centos 5.5, (sudo apt-get install gcc build-essential sharutils ccze libzip-dev automake1.9) i am install gcc and build-essential but other packages not found . even tho it was compiled every steps are work, but i am stuck up in this command (chown proxy:proxy /cache1 ) error chown: `proxy:proxy’: invalid user any solution.

    i have give root instead of proxy
    /usr/local/squid/sbin/squid -z
    FATAL: getpwnam failed to find userid for effective user ‘proxy’
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.
    CPU Usage: 0.006 seconds = 0.003 user + 0.003 sys
    Maximum Resident Size: 6816 KB
    Page faults with physical i/o: 0
    Aborted

    please help me, in my server Ubuntu is not installing(after language section it was thronging error CD ROM is not stable)system x3850 m2 server, so that installed centos but in Ubuntu every thin is working fine IBM xeon

    Comment by karthik — August 21, 2012 @ 1:59 PM

  83. thank a lot sir…it’s work even i join with coovachilli in 1pc

    Comment by jadiakbar — August 26, 2012 @ 5:45 AM

  84. Salam alkm

    i just installed on my ubuntu box, and the youtube is cached 100%….. thank you

    i want to know make the the youtube open as default same ( 240/ video quality ) to all users, how i can make that ????

    Comment by SAT LINK — August 29, 2012 @ 3:44 PM

    • Currently there is no function to force 240p streamed to user.
      somewhere at videocache forum, I read that they were talking about implementing this feature that by default, 240p video should be served to user as default. I don’t know if they have implemented it on there software, check there forum or ask there support.

      Comment by Syed Jahanzaib / Pinochio~:) — August 30, 2012 @ 9:09 AM

  85. hello mr syed..
    i have 4 harddisks 1Tb and 3 500gb
    i want to use 1 of 500 gb for system
    and 1tb for youtube cahce
    and the 2 other for files

    can you tell me how can do this..thanks you

    Comment by ismail — August 30, 2012 @ 4:00 PM

  86. Is youtube caching working with this guide as of right now or still have to refer to nginx method.

    Comment by Badr — August 30, 2012 @ 5:52 PM

    • Some people claims that its still working at there end.
      I know that youtube didnt changed the this range behavior for the whole
      globe because as for now few people emailed me that storeurl.pl still working at there end. So in the same country you can still get two different url patterns.

      Comment by Syed Jahanzaib / Pinochio~:) — August 31, 2012 @ 8:31 AM

  87. Hello….
    I have successful installation of squid video caching but I am facing some issues like serving of old/ stale pages…. Happens mostly with flckr.com, news websites and few blogs…. Could there be any solution to the same….? Thanks….

    Comment by Dinesh Bhardwaj — September 3, 2012 @ 8:33 PM

  88. sayed can you tell me how can i do that

    i have RB 1100AH2 and i have PPPOE setup on 10 interface each interface have
    defrant ip addresses and i have squid cash set up and evrything working find with all PPPOE interface
    the issue is i want to
    enable the cash rule for just one PPPOE interface

    and im using this nat rule
    add action=dst-nat chain=dstnat disabled=yes dst-port=80 protocol=tcp \
    to-addresses=10.3.0.100 to-ports=800

    10.3.0.100 my squid ip
    i had try to put the in interface = the PPPOE interface i need it work with cash but wont work
    and also i tried to put the interface ip on src address and did not work

    Comment by thegoogl — September 14, 2012 @ 12:58 AM

    • in src-address, add the ip range which is assigned to specific interface. this way only clients with the specific pool will be redirected to squid, rest of clients will go directly, as per default route.

      Comment by Syed Jahanzaib / Pinochio~:) — September 16, 2012 @ 12:56 PM

  89. If you are using centos tiens to change the file
    line “proxy” for squid and ready

    Comment by zeus — September 17, 2012 @ 10:33 AM

  90. sir mena ap ka blog sey bahoot kuch hasel kiyah hai ma ser bahoot bahoot khuch ho but kuch din chelna ka bad mera computer ki cache memmry pyll ho gai hai sir plz is ko delled karna ki bhi command bata dey

    Comment by hafizfidaali — October 5, 2012 @ 12:04 AM

  91. sir mari total disk space 72 gb hai jes ma sey 60 gb cache ka liyah rahki hai plz help me sir

    Comment by hafizfidaali — October 6, 2012 @ 2:04 AM

  92. sir mena ap ka blog sey bahoot kuch hasel kiyah hai ma sir bahoot bahoot khuch ho but kuch din chelna ka bad mera ubuntu server ki cache memmry full ho gai hai sir plz is ko delled karna ki bhi command bata dina sir mari total disk space 72 gb hai jes ma sey 60 gb cache ka liyah rahki hai plz help me sir

    Comment by hafizfidaali — October 7, 2012 @ 12:08 AM

  93. Assalamualaikum..
    Dear Jahanzaib..
    Please help me solved this problem
    root@ubuntu:~# 2012/10/10 00:33:37| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-pc-linux-gnu…
    -bash: 2012/10/10: No such file or directory
    2012/10/10 00:33:37| NOTICE: Could not increase the number of filedescriptors
    2012/10/10 00:33:37| With 1024 file descriptors available
    2012/10/10 00:33:37| Using epoll for the IO loop
    2012/10/10 00:33:37| Performing DNS Tests…
    2012/10/10 00:33:37| Successful DNS name lookup tests…
    2012/10/10 00:33:37| Adding nameserver 192.168.3.254 from squid.conf
    2012/10/10 00:33:37| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2012/10/10 00:33:38| User-Agent logging is disabled.
    2012/10/10 00:33:38| Referer logging is disabled.
    2012/10/10 00:33:38| logfileOpen: opening log /var/log/lusca/access.log
    2012/10/10 00:33:38| Unlinkd pipe opened on FD 18
    2012/10/10 00:33:38| Swap maxSize 61891584 + 524288 KB, estimated 4801220 objects
    2012/10/10 00:33:38| Target number of buckets: 240061
    2012/10/10 00:33:38| Using 262144 Store buckets
    2012/10/10 00:33:38| Max Mem size: 524288 KB
    2012/10/10 00:33:38| Max Swap size: 61891584 KB
    2012/10/10 00:33:38| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
    2012/10/10 00:33:38| logfileOpen: opening log /var/log/lusca/store.log
    2012/10/10 00:33:38| AUFS: /cache: log ‘/cache/swap.state’ opened on FD 20
    2012/10/10 00:33:38| AUFS: /cache: tmp log /cache/swap.state.new opened on FD 20
    2012/10/10 00:33:38| Rebuilding storage in /cache (DIRTY)
    2012/10/10 00:33:38| Using Least Load store dir selection
    2012/10/10 00:33:38| Current Directory is /root
    2012/10/10 00:33:38| Loaded Icons.
    2012/10/10 00:33:38| Accepting transparently proxied HTTP connections at 0.0.0.0, port 3128, FD 22.
    2012/10/10 00:33:38| WCCP Disabled.
    2012/10/10 00:33:38| Ready to serve requests.
    2012/10/10 00:33:38| WARNING: store_rewriter #1 (FD 7) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #2 (FD 8) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #3 (FD 9) exited
    2012/10/10 00:33:38| WARNING: store_rewriter #4 (FD 10) exited
    2012/10/10 00:33:38| Too few store_rewriter processes are running
    FATAL: The store_rewriter helpers are crashing too rapidly, need help!

    Comment by Habib Yunus — October 9, 2012 @ 10:40 PM

    • The error occurs when there is any syntax mistake in the storeurl.pl , its the culprit
      Try copying it to notepad then see for any invalid character, or try copy pasting it from another source.

      what is your aim ? are you trying to caching youtube? if yes , then use NGINX method, its much better then this one. Search my blog I have written an article on it too.

      Comment by Syed Jahanzaib / Pinochio~:) — October 10, 2012 @ 9:16 AM

  94. Sorry for my noob and bad english. I want to ask wether the facebook app (like zynga, ninja saga, etc) is dynamic content? how we can cache it?

    Comment by Al Kurnia — October 16, 2012 @ 6:15 PM

    • Try not to cache FB contents, they some times becomes headache. avoid caching dynamic content specially content that updates very frequently 🙂

      Comment by Syed Jahanzaib / Pinochio~:) — October 17, 2012 @ 8:43 AM

  95. Thank u. Now, my proxy be good

    Comment by haloteknisi — October 26, 2012 @ 4:36 PM

  96. […] buat install lusca na di mari […]

    Pingback by install lusca « Tikus Server — October 27, 2012 @ 10:09 AM

  97. Hello there! I simply wish to give you a huge thumbs up for the excellent information you have got here on this post. I am returning to your web site for more soon.

    Comment by filme xxx — October 31, 2012 @ 3:07 AM

  98. i want script redirect port 80 to port 8080

    Comment by malak — November 7, 2012 @ 6:05 AM

  99. Syed , what’s the way to add more than one hdd to save the files …
    i’ve 3hdd * 2TB =6TB

    Comment by darkng — November 21, 2012 @ 3:27 PM

  100. Thanks syed , just a last question about adding a hdd .
    for example if i have 4HDD’s
    1xSATA II – 90GB
    3x2TB Hardisk’s =6TB in total

    if i mount every disk and add just the cashe_dir of all the disk .
    would the disk be balanced so if the first is full to add it to the second or if the first is filled with 1Mb then the second becomes so the third … and so on .

    thanks for every help from you

    Comment by darkng — November 24, 2012 @ 3:05 AM

  101. Hiya, i’m having problem when i finish isntalling iàve got that error trying to run squid:

    FATAL: Bungled (null) line 170: http_access deny all
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.

    Could you help me?

    cheers
    Rafa

    Comment by Rafa TI — November 24, 2012 @ 5:52 PM

  102. hi sir,

    i m getting this msg after installing

    ” squidaio_queue_request: WARNING – Queue congestion”

    any suggestion!

    millions of thanks

    Comment by fajja — November 26, 2012 @ 2:10 PM

  103. hi, sir.
    how can disable caching the video from youtube only ..

    Comment by khaled — November 26, 2012 @ 11:33 PM

  104. Thank u. Now, my proxy be verygood

    Comment by tikkee — December 12, 2012 @ 8:36 PM

  105. Thank before, my problem why not stored in chrome update my proxy. consequently chrome update every time a connection is slow.
    Help me…

    Comment by Habib Yunus — December 13, 2012 @ 10:26 AM


  106. how to do like this?

    Comment by tom — December 21, 2012 @ 1:03 PM

  107. hi sir,
    i m getting this msg after installing
    ” squidaio_queue_request: WARNING – Queue congestion”
    any suggestion!
    millions of thanks

    Comment by fajja — December 21, 2012 @ 11:47 PM

  108. Nice project zaib bhai….

    Comment by Syed Azhar Hussain — December 27, 2012 @ 3:43 AM

  109. syed bhai after installing squid it works fine but after sometime i cant access cachelog and hits it gives this error.

    ./cachelog
    tail: cannot open `/var/log/squid/cache.log’ for reading: No such file or director
    y
    tail: no files remaining

    tail -f /var/log/squid/access.log
    tail: cannot open `/var/log/squid/access.log’ for reading: No such file or directo
    ry
    tail: no files remaining

    after rebooting squid those commands start working but after sometime like 12 hours or more it start giving same error.

    can u please help me y this happens?

    Comment by fajja — December 27, 2012 @ 5:26 PM

  110. 10080 minute is actually 1 week , not 1 month. Anyway, good project. Thanks

    Comment by Indra Gunawan — December 27, 2012 @ 5:34 PM

    • 🙂
      Configuration changes rapidly as per requirement, I forgot to update the heading 🙂

      Comment by Syed Jahanzaib / Pinochio~:) — December 28, 2012 @ 8:08 AM

      • Hi Bro!
        I will be so Grateful if you could please guide me for the line to add some lines: in order to cache from this site ” filehippo.com”

        Alex

        Comment by speedwaves — January 3, 2013 @ 7:23 AM

  111. I am having this problem with my sitemap. i got a video sharing website and its sitemap have some error in it. can anyone give me a solution to that.

    Comment by Sana zahra — January 9, 2013 @ 4:33 PM

  112. Hi
    I was wondering,since i decided to install proxy server in my network,could you advice me what kind of hardware should i buy for proxy machine.I have about 150 clients and average consume about 30mbits.
    I would appreciate your answer.
    10x in advance

    Comment by Fibonacci — January 19, 2013 @ 3:56 AM

    • Fro 150 Users, Any Intel Core2Duo base PC with 4-8 GB RAM, 320/500 GB HDD would be enough.
      XEON base processors are however good and stable as compare to desktop PC. Its all depend on your budget.

      It’s a good idea to have 64 bit architecture hardware/OS support. It works very good and reliable.

      Comment by Syed Jahanzaib / Pinochio~:) — January 19, 2013 @ 11:25 AM

      • Many thanks for quick reply

        Comment by Fibonacci — January 19, 2013 @ 2:19 PM

  113. can not start up automatically after a reboot
    is there a way?

    Comment by rama — January 23, 2013 @ 2:03 PM

  114. Hello ,
    did anyone find a fix for the youtube segmentation issue in order to be downloaded as a single file in the zph tos
    i am using LUSCA_HEAD-r14733
    thank you

    Comment by cr — January 26, 2013 @ 11:29 PM

  115. Good Work !!! Zaib

    Comment by Chetan Muneshwar — January 30, 2013 @ 10:11 PM

  116. Hello
    thank you for all tut.
    I have problem after make install and setup cache work fine with images , exe to by pass queue from mikrotik but youtube and video dnt by pass please bro if can help me with my request and I want you to advise me which better between nginx the one I installed and LUSCA ,
    thank you very much

    Comment by alfanet1 — February 11, 2013 @ 1:52 PM

  117. hi
    how can i use laptop with one LAN interface as a cache server

    Comment by dh — February 13, 2013 @ 9:19 PM

    • on squid, set its default gateway and dns pointing to mikrotik interface, and in mikrotik create a masquerade rule that masquerade all traffic coming from squid pc.
      move this rule on TOP. (if using HOTSPOT you can bypass squid box mac in BYPASSED macs)

      Comment by Syed Jahanzaib / Pinochio~:) — February 14, 2013 @ 4:12 PM

      • it will work with dstnat the one interface will work as input to squid and output.? dstnat in mikrotik with out change squid roule

        Comment by dh — February 15, 2013 @ 2:21 AM

  118. if can the script for one LAN squid with mikrotik nat

    Comment by dh — February 17, 2013 @ 4:51 PM

    • yes you can use one lan card with squid also.
      in squid, configure default gateway and dns pointing to mikrotik interface which is connected with the squid box.
      also create a NAT rule for squid ip.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:42 PM

  119. Hello

    i make 2 squid as peer in 1 box. the peer is working perfect between ( Lusca as video cache ) and ( squid2.7.stabel7 as data files cache )

    the problem is, we can mangle in mikrotik the DSCP TOS of squid only, but not the peer data that coming from Lusca.

    any solution ???

    Comment by Bilal Mahdi — February 18, 2013 @ 3:39 AM

    • I have not worked in such scenario, but TOS wont work in default config i am sure.
      Search google there might be some workaround for it.
      I recommend you to post your query at squid mailing list.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:41 PM

  120. Hello All
    i have install ( lusca – video cache ) and ( squid 2.7 stable 7 ) in same box as peer.
    i Tos between the mikrotik and squid is working fine but the peer data is not include in the Tos for some reason .

    any suggesting !!!

    Comment by Bilal Mahdi — February 18, 2013 @ 3:44 AM

    • I have not worked in such scenario, but TOS wont work in default config i am sure.
      Search google there might be some workaround for it.
      I recommend you to post your query at squid mailing list.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2013 @ 12:40 PM

  121. pcc loadbalacsinig main squid settinig kese karo plz help me

    you help me

    Comment by noman — February 20, 2013 @ 11:35 PM

  122. hello, i wanna ask
    i’ve just use ur storeurl and conf. im forget to backup my old config.
    but now i want to disable youtube cache, which one i should disable?
    i tried # in youtube fix and compability old cached, but cache still work
    thanks

    Comment by merry — February 24, 2013 @ 1:14 AM

  123. Thank you very much for this tutorial. It’s great. But I have two problems. 1. My cache is always full then the squid will stop. how can I automatically remove the old cache object? 2. Internet call is always disconnected?

    Comment by Rolando Casinillo — March 2, 2013 @ 6:14 PM

  124. Assalamu’alaikum im from indonesia, i have finishe doing the instalation, then i try it, its work fine, but while im playing the other video from youtube, in the screen stiil apear and play the video that that i played before, what is th eproblem ,thanks before
    Wassalam

    Comment by Dedy Availatz — March 22, 2013 @ 1:53 PM

  125. dear Brother

    we have install the ( Cache Youtube with SQUID / LUSCA )

    and its work perfect, and after 671 GB cached used, the squid start make a big traffic on our upload, it tack all the upload bandwidth.

    any suggesting ???

    Comment by Bilal Mahdi — March 23, 2013 @ 3:38 PM

    • Hmmm its to understandable that how cache is increasing upload traffic. However first try to clear cache , and see if it helps, then we can do further digging what could be possible wrong.

      Comment by Syed Jahanzaib / Pinochio~:) — March 25, 2013 @ 9:43 AM

  126. Is there a way to test storurl.pl if it works??
    I test the storeurl.pl as the above posted, and fed the youtube url:
    http://r6—sn-5njj-u2xe.c.youtube.com/videoplayback?algorithm=throttle-factor&burst=40&cp=U0hVSVRSVF9HTkNONV9MTFhDOmZYOUhGeEFicHoy&cpn=ifHLBv3rILC7UrfL&expire=1364476224&factor=1.25&fexp=916807%2C914083%2C916623%2C901478%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C904830%2C919373%2C930803%2C906836%2C920201%2C929602%2C930101%2C930603%2C926403%2C900824%2C910223&id=o-AJDf3TDHyp2H0FMpLSoCKM0akW1_kumyjOAtem_dY-_V&ip=203.71.84.209&ipbits=8&itag=34&keepalive=yes&key=yt1&ms=au&mt=1364452453&mv=m&newshard=yes&range=13-1783807&ratebypass=yes&signature=D2474D7D411E60FC505DBF4CBDC6A2BDF1AE32B9.45C16D3AA02E5BBC38F373993A3688480EDA139C&source=youtube&sparams=algorithm%2Cburst%2Ccp%2Cfactor%2Cid%2Cip%2Cipbits%2Citag%2Csource%2Cupn%2Cexpire&sver=3&upn=ZRNE9iDw7gM

    the result is :
    http://r6—sn-5njj-u2xe.c.youtube.com/videoplayback?algorithm=throttle-factor&burst=40&cp=U0hVSVRSVF9HTkNONV9MTFhDOmZYOUhGeEFicHoy&cpn=ifHLBv3rILC7UrfL&expire=1364476224&factor=1.25&fexp=916807%2C914083%2C916623%2C901478%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C904830%2C919373%2C930803%2C906836%2C920201%2C929602%2C930101%2C930603%2C926403%2C900824%2C910223&id=o-AJDf3TDHyp2H0FMpLSoCKM0akW1_kumyjOAtem_dY-_V&ip=203.71.84.209&ipbits=8&itag=34&keepalive=yes&key=yt1&ms=au&mt=1364452453&mv=m&newshard=yes&range=13-1783807&ratebypass=yes&signature=D2474D7D411E60FC505DBF4CBDC6A2BDF1AE32B9.45C16D3AA02E5BBC38F373993A3688480EDA139C&source=youtube&sparams=algorithm%2Cburst%2Ccp%2Cfactor%2Cid%2Cip%2Cipbits%2Citag%2Csource%2Cupn%2Cexpire&sver=3&upn=ZRNE9iDw7gM sucks

    I think the storeurl.pl is not work correctly.

    If I cancel “storeurl_rewrite_program /etc/squid3/storeurl.pl”, and replay the youtube movie again, I can find “TCP_HIT” in the access.log with the same movie.

    Comment by YCC — March 28, 2013 @ 12:29 PM

  127. bahi ager aap ko 400$ pay kr do tu kia aap bna do gy

    Comment by Shahid — March 31, 2013 @ 2:41 PM

  128. few months ago i follow your tutorial, squid can save video youtube HIT good .. but yesterday i play youtube videos and when I play another video why video is my first turn, rotated back, I’ve been trying to clear squid cache but return again like what I described above
    I even re-install the server sdh still as above
    may be updated squid.conf & store.url you

    Comment by hary sasmito — April 1, 2013 @ 10:03 PM

    • I have the same problem. I think the storeurl.pl is useless which should be rewrited.

      Comment by YCC — April 3, 2013 @ 1:52 PM

  129. biar dikata nyontek,, tapi hasil oprek Syed Jhanzaib. hasilnya lebih maknyuzzzzzzz… 🙂

    Comment by ming — April 9, 2013 @ 12:42 PM

  130. few months ago i follow your tutorial, squid can save video youtube HIT good .. but yesterday i play youtube videos and when I play another video why video is my first turn, rotated back, I’ve been trying to clear squid cache but return again like what I described above
    I even re-install the server sdh still as above
    may be updated squid.conf & store.url you

    Comment by Muhammad Ardiansyah Putra — April 20, 2013 @ 10:57 AM

    • This method no longer cache the youtube cache AFAIK.
      Try with NGINX method, possibly it may b working. youtube is blocked in our country from past one year therefore I cant do any testing on it,

      Comment by Syed Jahanzaib / Pinochio~:) — April 24, 2013 @ 10:23 AM

  131. assalamualaikum…..
    I am from indonesia, i’ve tried the tutorial above. but the results I could not browsing.

    sorry, I am not so fluent in English 😀

    Comment by Aji Diyantoro — April 20, 2013 @ 6:23 PM

    • You need to provide more detail. Without knowing you goals and what you have done so far, its hard to assist you. Please provide details how you have setup squid and how it is attached to mikrotik or what? describe your network scenario and goals you want to achieve.

      Comment by Syed Jahanzaib / Pinochio~:) — April 24, 2013 @ 10:22 AM

  132. hello syed ,
    i have install the nginx with YouTube and cache work fine except the download the YouTube video from real player or internet download manger its not any help for that?

    Comment by husam — May 4, 2013 @ 12:27 AM

    • Can’t help you in this regard anymore, youtube got banned by the authorities in our country last year, so i cant test anything on it.

      Comment by Syed Jahanzaib / Pinochio~:) — May 4, 2013 @ 10:20 AM

  133. High Performance Cache HIT Proxy Lusca on Ubuntu Server + Configurasi Router MikroTik

    Sumber: MR-EKOAPRIADI – http://mr-ekoapriadi.blogspot.com/2013/03/high-performance-lusca-squid-proxy.html

    Comment by Adam — May 5, 2013 @ 10:54 PM

  134. hello sayed
    in this case that parameter should modify according to server memory (ram ) and capacity ? and all videos and files are not cashed ?

    Comment by Ali — May 27, 2013 @ 7:51 PM

  135. new youtube any help to cache this
    http://www.youtube.com/ptracking?
    plid=AAThOYd6J3xLA9WC
    oid=SlODfwbK82f2lVHy5P82EA.UnsFrT3f3Ne9E0pAoQfCsQ.VyEtjbeN302yXkP3Qulrog.9b8naKhAjYHfDP3yKaWKtg.GoednyIs730UDQtrvI_P-Q.FvXsj2Mdo_hj4nnNnOxKzg.0YSVm7yx3KFfY2JfQ2_1CA
    pltype=contentugc
    ptk=youtube_multi
    video_id=N-kuHq5rJYU
    cpn=pJmS-oUxyadHDGM-

    GET /ptracking?plid=AAThOYd6J3xLA9WC&oid=SlODfwbK82f2lVHy5P82EA.UnsFrT3f3Ne9E0pAoQfCsQ.VyEtjbeN302yXkP3Qulrog.9b8naKhAjYHfDP3yKaWKtg.GoednyIs730UDQtrvI_P-Q.FvXsj2Mdo_hj4nnNnOxKzg.0YSVm7yx3KFfY2JfQ2_1CA&pltype=contentugc&ptk=youtube_multi&video_id=N-kuHq5rJYU&cpn=pJmS-oUxyadHDGM- HTTP/1.1
    Host: http://www.youtube.com
    User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Referer: http://s.ytimg.com/yts/swfbin/watch_as3-vfldOoVEA.swf
    Cookie: VISITOR_INFO1_LIVE=yv8v0rxZnvI; PREF=fv=11.7.700; YSC=x2u8uvMXD8Y; ACTIVITY=1373537164597
    Connection: keep-alive

    HTTP/1.0 204 No Content
    Date: Thu, 11 Jul 2013 10:05:14 GMT
    Server: gwiseguy/2.0
    Content-Type: video/x-flv
    Expires: Tue, 27 Apr 1971 19:44:06 EST
    X-YouTube-Other-Cookies: VISITOR_INFO1_LIVE=yv8v0rxZnvI
    X-Content-Type-Options: nosniff
    X-XSS-Protection: 1; mode=block
    X-Frame-Options: SAMEORIGIN
    Content-Length: 0
    X-Cache: MISS from cache
    X-Cache-Lookup: MISS from cache:8080

    Comment by joe — July 11, 2013 @ 3:11 PM

  136. FATAL: cache_dir /usr/local/squid/var/cache: (2) No such file or directory
    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.
    CPU Usage: 0.004 seconds = 0.004 user + 0.000 sys
    Maximum Resident Size: 6192 KB
    Page faults with physical i/o: 0
    Aborted (core dumped)

    what the meaning of error

    Comment by AHMAD FARUQ — July 15, 2013 @ 12:25 PM

    • It says that the squid cannot found the cache2 folder in your system.

      Adjust the cache_dir folder in the squid.conf , point to to valid folder, or create that folder in the respective location.

      Comment by Syed Jahanzaib / Pinochio~:) — July 16, 2013 @ 8:21 AM

      • apakah untuk folder cache harus dibuatkan partisi sendiri… atau tidak

        Comment by AHMAD FARUQ — July 16, 2013 @ 8:57 AM

      • whether the cache folder should be created for its own partition … or not

        Comment by AHMAD FARUQ — July 16, 2013 @ 8:59 AM

      • you can use same drive for caching folder, or you can also use separate drive for the cache , actually it depends on the number of users, load and some other factors.

        Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:06 AM

    • change the folder of log

      Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:06 AM

  137. 2013/07/16 07:30:32| aclParseIpData: WARNING: Netmask masks away part of the specified IP in ‘192.168.3.1/8’
    2013/07/16 07:30:32| WARNING: ‘0.0.0.0/0.0.0.0’ is a subnetwork of ‘0.0.0.0/0.0.0.0’
    2013/07/16 07:30:32| WARNING: because of this ‘0.0.0.0/0.0.0.0’ is ignored to keep splay tree searching predictable
    2013/07/16 07:30:32| WARNING: You should probably remove ‘0.0.0.0/0.0.0.0’ from the ACL named ‘all’
    2013/07/16 07:30:32| Starting Squid Cache version LUSCA_HEAD-r14809 for i686-redhat-linux-gnu…
    2013/07/16 07:30:32| Process ID 12774
    2013/07/16 07:30:32| NOTICE: Could not increase the number of filedescriptors
    2013/07/16 07:30:32| With 1024 file descriptors available
    2013/07/16 07:30:32| Using epoll for the IO loop
    2013/07/16 07:30:32| Performing DNS Tests…
    2013/07/16 07:30:32| Successful DNS name lookup tests…
    2013/07/16 07:30:32| Adding nameserver 192.168.3.1 from squid.conf
    2013/07/16 07:30:32| Adding nameserver 8.8.8.8 from squid.conf
    2013/07/16 07:30:32| helperOpenServers: Starting 7 ‘storeurl.pl’ processes
    2013/07/16 07:30:32| logfileOpen: opening log /var/log/squid/access.log
    2013/07/16 07:30:32| logfileOpen: opening log /usr/local/squid/var/logs/access.log

    FATAL: Cannot open ‘/usr/local/squid/var/logs/access.log’ for writing.
    The parent directory must be writeable by the
    user ‘proxy’, which is the cache_effective_user
    set in squid.conf.
    i found the error again
    please syed jahanzaib help me…..

    Comment by AHMAD FARUQ — July 16, 2013 @ 4:34 PM

    • Do you have the correct ownership of files/directories? It says that it cannot open ‘/usr/local/squid/var/logs/access.log’ for writing and that the parent directory must be writeable by the user ‘proxy’ OR ‘squid’. You could try chown -R squid /usr/local/squid

      Comment by Syed Jahanzaib / Pinochio~:) — July 18, 2013 @ 10:05 AM

      • salam…. syed
        I am very curious about the tutorial that you share

        I have several times tried to follow all the tutorial and I have failed but to this day I continue to try to understand because I still do not know much about network …………

        whether you willing help me … provide installation instructions ubuntu 10:04 server especially on the partition corresponding to the appropriate setting squid with this blog

        This specification yng computer I use for external proxies:
        = 1 Gb ram
        = 80 Gb HDD
        192.168.3.2 ip proxy.

        email = bouojog@gmail.com
        or through your blog is because I am waiting for your answer on a daily basis.

        I am from Indonesia and could not speak English well
        apologize if there is a greeting that is less true

        Comment by AHMAD FARUQ — July 18, 2013 @ 12:45 PM

  138. […] There might be some problem with the storeurl.pl content, either its not copy pasted correctly. First try without storeurl.pl IF it works ok , then Try to create storeurl.pl from following URL. https://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-video&#8230; […]

    Pingback by Youtube caching with SQUID 2.7 [using storeurl.pl] | Syed Jahanzaib Personnel Blog to Share Knowledge ! — September 17, 2013 @ 2:46 PM

  139. Dear All,

    I am getting following error when typing “lusca -d1”

    /cache-1//swap.state: (13) Permission denied

    FATAL: storeAufsDirOpenSwapLog: Failed to open swap log.

    Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.

    CPU Usage: 0.012 seconds = 0.004 user + 0.008 sys

    Maximum Resident Size: 16160 KB

    Page faults with physical i/o: 0

    (squid)[0x80b4a84]

    (squid)[0x80b543f]

    (squid)[0x80c25e7]

    (squid)[0x80bf6e3]

    (squid)[0x80af1fe]

    (squid)[0x80aba34]

    (squid)[0x804bdaa]

    (squid)[0x804cb13]

    /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75584d3]

    (squid)[0x804d1b9]

    Comment by Siraj — September 28, 2013 @ 9:42 PM

    • If using Ubuntu

      chown proxy:proxy /cache-1

      Erase the swap.state file
      squid -z
      service squid start

      OR
      squid -d1N

      Comment by Syed Jahanzaib / Pinochio~:) — September 29, 2013 @ 10:17 AM

      • whe i am typing “service squid3 restart:
        i am getting following error:
        stop: Unknown instance:
        start: Rejected send message, 1 matched rules; type=”method_call”, sender=”:1.103″ (uid=1000 pid=18302 comm=”start squid3 “) interface=”com.ubuntu.Upstart0_6.Job” member=”Start” error name=”(unset)” requested_reply=”0″ destination=”com.ubuntu.Upstart” (uid=0 pid=1 comm=”/sbin/init”)

        Comment by Siraj — September 29, 2013 @ 5:59 PM

      • Dear All,
        Cant we just use one interface. I mean same interface is connecting cache server to internet and also receiving request from LAN/user? if it possible then how can we change perimeters in your mentioned configs to achieve this goal.

        Comment by Siraj — September 29, 2013 @ 6:16 PM

  140. Salam Bhai,
    Mikrotik proxy can able to chche youtube video ????

    Comment by rabby — October 7, 2013 @ 9:33 AM

  141. i’ve tested yesterday but seems it doesn’t work anymore, can anyone re-confirm this tut still works or not ?

    Comment by Albert — November 23, 2013 @ 9:34 AM

  142. Mantaf gan atriclenya sangat membantu, ini sy experiment sendiri pake ini lumayan mantaf gan.
    Sumber: http://mediacomptech.blogspot.com/2013/11/cara-baypass-hit-proxy.html

    Comment by Tuan Lang — November 28, 2013 @ 6:35 AM

  143. Mantaf gan atriclenya sangat membantu, ini sy experiment sendiri pake ini lumayan mantaf gan.
    Sumber: http://mediacomptech.blogspot.com/2013/11/high-performance-cache-hit-proxy-lusca.html

    Comment by Tuan Lang — November 28, 2013 @ 6:39 AM

  144. thanks for the tutorial,
    i’m newbie who learn microtik
    🙂

    Comment by bersihwajah — December 19, 2013 @ 6:19 PM

  145. […] by Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue « Syed Jah… — March 28, 2012 @ 10:38 […]

    Pingback by Youtube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.pl | Syed Jahanzaib Personnel Blog to Share Knowledge ! — January 20, 2014 @ 1:48 PM

  146. hati – hati … perhatikan baik2…. banyak configurasi yg aneh …. jangan di serap 100% … dia ga mungkin share ilmu 100% logikanya gitu , tp kasian 1000’an orang awan jd korban ….

    Comment by Bajak Laut — February 1, 2014 @ 7:04 PM

  147. aoa. kya ap torrent ke ports forward kar sakty hy

    Comment by Naeem Ahmed — April 10, 2014 @ 2:50 AM

  148. Sir …………This Line means ( change the quantity of my local networks user )
    tcp_outgoing_tos 0x30 lanuser [lanuser is ACL for local network, change it to match your’s]
    we can’t understood ……..plz explain………..Allah AAp Ko Jaza e Khair dey.

    Comment by Salman — April 21, 2014 @ 8:32 PM

  149. I cant install this is saying ”bash: syntax error near unexpected token `newline”

    Comment by Tnet — April 22, 2014 @ 12:15 AM

  150. Post Upgrade 🙂

    Comment by Waseem Ahmed — April 23, 2014 @ 10:11 AM

  151. I have successfully installed squid but when I used the command service squid start, it couldn.t
    ]# service squid start
    squid: unrecognized service must write squid to run and don’t run on startup and don’t cache youtube only files and witch command from this install squid file?

    apt-get install gcc -y
    apt-get install build-essential -y
    apt-get install libstdc++6 -y
    apt-get install unzip -y
    apt-get install bzip2 -y
    apt-get install sharutils -y
    apt-get install ccze -y
    apt-get install libzip-dev -y
    apt-get install automake1.9 -y
    apt-get install acpid -y
    apt-get install libfile-readbackwards-perl -y
    apt-get install dnsmasq -y
    apt-get install havp -y

    thnx for your work wait ur answer

    Comment by Ma7m0d — April 23, 2014 @ 11:38 AM

    • service is not installed by default when you COMPILE squid from source.
      To start squid, simply run

      squid -d1N

      Comment by Syed Jahanzaib / Pinochio~:) — April 24, 2014 @ 8:22 AM

      • i wanna start it auto on start up i must write squid to start every reboot and don’t cache youtube but cache files good whats proplem?

        Comment by Mahmoud Salem — April 25, 2014 @ 9:46 AM

      • you can add squid command in /etc/rc.local
        then squid will start on every reboot auto.

        I tested youtube caching at a remote server, and it worked fine (I dont say 100%, but most of videos do get cached perfectly, I tested it several times).
        Do you have public IP configured at SQUID box? then I can check it.

        Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2014 @ 11:02 AM

  152. sir…………..i have seen someting like this..
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
    20 packets captured
    20 packets received by filter
    0 packets dropped by kernel
    Just two line isn’t see
    20:25:07.961722 IP (tos 0×30, ttl 64, id 45167, offset 0, flags [DF], proto TCP (6), length 409)
    20:25:07.962059 IP (tos 0×30, ttl 64, id 45168, offset 0, flags [DF], proto TCP (6), length 1480)

    Comment by Salman — April 24, 2014 @ 1:25 AM

  153. Bhai if i am already running squid 2.7 stable 7, then same process will be followed? and what is the difference in new and old configuration? i have customized my squid config too much. little hesitant to change…. kindly guide what to do, upgrade or not? Thanks

    Comment by Adeel — April 25, 2014 @ 8:27 PM

  154. I have successfully installed squid but why it not cache https sites like youtube and facebook.
    there is a way cach https in squid

    Comment by Mohammed — April 26, 2014 @ 12:49 AM

    • https will not get cached by default.

      Comment by Syed Jahanzaib / Pinochio~:) — April 26, 2014 @ 10:13 AM

      • can you tell me how https cached by squid??thanks

        Comment by Mohammed — April 26, 2014 @ 8:12 PM

      • Maybe if you put manually squid cache ports in your browser to work with https I think this can work.

        Comment by Kiko — October 14, 2014 @ 1:03 AM

  155. Youtube not Cached again , My Squid Cache is not caching again . Please Someone Confirm

    Comment by Tne — April 27, 2014 @ 6:11 AM

    • you are right

      Comment by hussien — May 3, 2014 @ 6:02 AM

  156. I have successfully installed squid but why it not cache dailymotion ….?

    Comment by Qasim — April 28, 2014 @ 4:28 PM

  157. hi mr syed youtube is change to https how can i cached them what i want in config to start cache or how i can redirect https to http

    Comment by Mostafa Mohamed — April 28, 2014 @ 8:32 PM

  158. this didnt cache youtube , they must be change some thing for sure , no new configs ?

    Comment by hussien — May 3, 2014 @ 6:01 AM

  159. how to add speed test also to be cached?

    Comment by surya — May 3, 2014 @ 3:32 PM

  160. gracias

    Comment by Lorenz Avila — May 6, 2014 @ 11:16 PM

  161. i`ve problem with LUSCA configure … as

    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –prefix=/usr \
    >
    bash: –prefix=/usr: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –exec_prefix=/usr \
    >
    bash: –exec_prefix=/usr: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –bindir=/usr/sbin \
    >
    bash: –bindir=/usr/sbin: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –sbindir=/usr/sbin \
    >
    bash: –sbindir=/usr/sbin: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –libexecdir=/usr/lib/squid \
    >
    bash: –libexecdir=/usr/lib/squid: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –sysconfdir=/etc/squid \
    >
    bash: –sysconfdir=/etc/squid: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –localstatedir=/var/spool/squid \
    >
    bash: –localstatedir=/var/spool/squid: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –datadir=/usr/share/squid \
    >
    bash: –datadir=/usr/share/squid: No such file or directory
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-async-io=24 \
    >
    –enable-async-io=24: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-aufs-threads=24 \
    >
    –with-aufs-threads=24: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-pthreads \
    >
    –with-pthreads: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-storeio=aufs \ss
    –enable-storeio=aufs: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942#
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-linux-netfilter \
    >
    –enable-linux-netfilter: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-arp-acl \
    >
    –enable-arp-acl: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-epoll \>
    –enable-epoll: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-removal-policies=heap \
    >
    –enable-removal-policies=heap: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-aio \
    >
    –with-aio: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-dl \
    >
    –with-dl: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-snmp \
    >
    –enable-snmp: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-delay-pools \
    >
    –enable-delay-pools: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-htcp \
    >
    –enable-htcp: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-cache-digests \
    >
    –enable-cache-digests: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –disable-unlinkd \
    >
    –disable-unlinkd: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-large-cache-files \
    >
    –enable-large-cache-files: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-large-files \
    >
    –with-large-files: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-err-languages=English \
    >
    –enable-err-languages=English: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-default-err-language=English \
    >
    –enable-default-err-language=English: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –enable-referer-log \
    >
    –enable-referer-log: command not found
    root@zishan-VirtualBox:/tmp/LUSCA_HEAD-r14942# –with-maxfd=65536
    –with-maxfd=65536: command not found

    Comment by smn4all — May 7, 2014 @ 6:40 PM

  162. Syed,

    The storeurl uses the same amount of hard drive space using in lusca directive cache_dir.?

    thanx

    Comment by patito — May 14, 2014 @ 1:51 AM

  163. sorry i give a comment/question in a wrong post…i should write it down in here….
    here my question……
    i use your squid.conf an store.pl but got this error message and the squid stopped…not everytime but about once or twice in a day
    Can’t call method “readline” on an undefined value at /etc/squid/storeurl.pl line 151, line 21
    would you help me…
    thanks in advance……..

    Comment by drain — May 16, 2014 @ 1:36 PM

    • please mr…i need your help….btw…i didn’t use your lusca,i already install lusca when i use your configuration…is that that become my problem?
      sorry for my bad english……..

      Comment by drain — May 20, 2014 @ 8:09 AM

  164. Zaib estimated:

    do cache to http://www.speedtest.net to understand velocity measurements that are JPG images, but do HIT because they are in another format eg “/ speedtest/random750x750.jpg x = 1400687173339 & y = 2?” and not makes cache.

    Greetings and Thanks

    Comment by Javier — May 21, 2014 @ 9:05 PM

  165. brother Syed,

    i get this error while installing Lusca / Squid on Ubuntu server 12.04

    E: Could not open lock file /var/lib/dpkg/lock – open (13: Permission denied)
    E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

    Comment by Moaz Dabsheh — May 22, 2014 @ 12:56 AM

  166. how to cache update google chrome?

    Comment by muib — May 26, 2014 @ 4:52 PM

  167. hello Syed,
    i use your squid.conf and storeurl.pl. it works fine excpet for that, when i pause and then resume IDM, it cannot resume downloading. Please any advice. thanks in advance.

    Comment by zd — May 28, 2014 @ 8:03 AM

  168. Syed,

    Do you take donations.? How..?

    Greetings.

    Comment by patito — May 29, 2014 @ 8:32 AM

  169. FATAL: The store_rewriter helpers are crashing too rapidly, need help!
    dear sir ya meg aarha ha plz help

    Comment by tahirmirza786 — May 31, 2014 @ 6:32 PM

  170. Sir…………plz this article convert into automtated script for easy to installation
    ALLAH aap ko JAZA E KHAIR dey………!

    Comment by Salman — June 4, 2014 @ 2:38 PM

  171. man did this still working for caching youtube ?
    please confirm ..
    thanks

    Comment by hussien — June 5, 2014 @ 11:58 PM

  172. is this still working ?

    Comment by mak — June 14, 2014 @ 3:03 PM

  173. [root@system ~]# Can’t call method “readline” on an undefined value at /etc/squid/storeurl.pl line 150, line 1.
    2014/06/17 15:18:54| WARNING: store_rewriter #1 (FD 8) exited
    2014/06/17 15:18:54| WARNING: store_rewriter #1 (FD 8) exited
    Can’t call method “readline” on an undefined value at /etc/squid/storeurl.pl line 150, line 1.
    2014/06/17 15:18:55| WARNING: store_rewriter #2 (FD 42) exited
    2014/06/17 15:18:55| WARNING: store_rewriter #2 (FD 42) exited

    How this problem…?

    Comment by Novasandi Cpu — June 17, 2014 @ 1:25 PM

  174. Hi Syed Jahanzaib
    My squid poxy have problem
    It can’t cache files with upper than 1 MB size
    That’s not cache even I change maximum_cache_ object size to 1 GB
    Please help me
    thanks a lot.

    Comment by hamid hz — June 19, 2014 @ 6:03 PM

  175. Mr Syed, how to cache https such as facebook game in lusca, i have no idea to solve it

    many thanks,

    Prast

    Comment by prast — June 21, 2014 @ 7:24 PM

  176. hi sir….
    i follow above setp install on ubuntu 12.04 x86_64 system…..but cache youtube always show TCP_MISS in /var/log/squid/access.log ,but it success in above examples when download from winrar and music website……. anyidea??tks…

    Comment by squawell — June 23, 2014 @ 2:18 PM

  177. hy bosss
    im from indo
    i wana ask something

    i suksesed install this proxy
    but when i open youtube via google chrome/firefox
    the video cannod load bandwidth fully…why boss?

    Comment by jojo — July 2, 2014 @ 9:04 AM

  178. […] Following is an automated script to install LUSCA r14942 for UBUNTU with aggressive content caching support including some video web sites like YOUTUBE and few others as described in my other article @ https://aacable.wordpress.com/2014/04/21/howto-cache-youtube-with-squid-lusca-and-bypass-cached-video&#8230; […]

    Pingback by LUSCA Automated Install Script | Syed Jahanzaib Personnel Blog to Share Knowledge ! — July 2, 2014 @ 12:22 PM

  179. Sir…………plz this article “Make” into automtated script for easy to installation
    ALLAH aap ko JAZA E KHAIR dey………!

    Comment by Salman — July 3, 2014 @ 8:49 PM

  180. Hai syed,

    what should i delete from squid.conf and storeurl.pl if i don’t want to cache youtube? only for youtube cache?
    Thanks,

    Comment by one kuyak — July 6, 2014 @ 6:27 PM

  181. assalamualikum , tanks syed, thats very help,.. but on /var/log/squid/access.log only ip gateway the proxy server that appears when a user accesses a web destination , no much has changed from your script I just adjust the cache directory … i hope you can reply my question or you can send in my e-mail, sorry i’am not good in english , jazakallah

    Comment by aldykustyandi — July 9, 2014 @ 9:25 AM

  182. HI. thank for this tip. i have a 2nd hard drive is sdb. Do i need to modify my cache as /dev/sdb /data ? also do i need to modified nano /etc/rc.local?? iptables -t nat -A PREROUTING -i eth1 -p tcp –dport 80 -j DNAT –to 192.168.50.50:8080

    route add -net 192.168.100.0 netmask 255.255.255.0 gw 192.168.50.254 dev eth1

    thank you

    Comment by alex — July 10, 2014 @ 7:34 AM

  183. HI when i run ps aux |grep squid
    i only get root 24219 0.0 0.0 4428 836 pts/0 S+ 22:46 0:00 grep –color=auto squid . Its that okay? Thank you

    Comment by alex — July 11, 2014 @ 8:49 AM

  184. hi. i have finally got install.but i dont get HIT. I only ge MISS. do i need to connect cross over cable? from Mikrotik to Squid server? Thank you

    Comment by alex — July 11, 2014 @ 9:24 AM

  185. Finally. I got to work! its alive. the only thing i do tcpdump -vni eth1 | grep ‘tos 0×30′ and nothing comes out but when i do tcpdump -vni eth1… it coems up tox0.. And youtube dont seem to work. but its okay. after 5 days of cracking my head. THANK YOU SIR. i recomend you guys to use LINUX internet SHARING SCRIPT its a must for this to work and always do everything in Root mode. THANK YOU THANK YOU! i looked at thsi post last year and i didnt do it. now finally i got it !! THANK YOU AGAIN

    Comment by Alex — July 15, 2014 @ 4:10 AM

  186. Excellent script, worked perfect for me on Ubuntu 12.04

    Only issue is I’m not able to check logs. No logs are showing up in /var/log/squid/access.log

    Comment by arnoldrodrigues — July 19, 2014 @ 4:44 PM

  187. Required help with iptables configuration on Mikrotik and Squid Server

    Mikrotik Router Scenario

    Ether1 – WAN (/30 IP Pool)
    Ether2 – Squid Server (/30 Pool)
    Ether3 – LAN (/28 Public Pool) Connected to static users

    Required Lan Public Pool traffic to be redirected to squid server

    Tried Firewall Rule which does not work
    https://aacable.wordpress.com/tag/dscp-12/

    Thanks In Advance.

    Comment by Arnold Rodrigues — July 21, 2014 @ 5:45 PM

  188. hey >>>>>>>>>problem SQUID NOT WORKING AFTER REBOOT….plz help me for solve this problem

    Comment by mohamed — July 25, 2014 @ 4:26 PM

  189. youtube is now https no longer works any solution

    Comment by matheusluis — August 4, 2014 @ 6:27 PM

    • Yes is true

      Comment by Tnet — August 8, 2014 @ 4:56 AM

      • computers increasingly open youtube with https. as we solve that?

        Comment by Jorge — August 21, 2014 @ 10:08 PM

      • there is no free reliable solution for it yet.

        Comment by Syed Jahanzaib / Pinochio~:) — August 26, 2014 @ 8:56 AM

  190. hello syed i have 2 problems:first the squid not start automaticaly whe ubuntu start..i will type squid -d1N in every restart..
    second its not caching youtube videos but caching other website videos..
    thanks you very much

    Comment by ismail — August 8, 2014 @ 2:39 PM

  191. hi gys
    for centos who facing problem with perl backward file follow these steps:-

    1. -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
    2. yum upgrade perl
    3. yum install perl-IO-Compress perl-Net-DNS perl-Email-Valid perl-File-ReadBackwards perl-Mail-SPF-Query perl-Mail-SRS perl-LDAP perl-MIME-tools perl-File-Scan-ClamAV

    thats it after follow these steps start ur squid
    squid -dl

    Comment by rehmat ali — August 11, 2014 @ 12:45 AM

  192. salam to all,
    Dear Janzaib,
    only 1 problem i m facing with this solution is that when i restart my centos server squid could not starts and i find some errors in cache.log
    ipcCreate: /etc/squid/storeurl.pl: (13) Permission denied
    ipcCreate: /etc/squid/storeurl.pl: (13) Permission denied
    ipcCreate: /etc/squid/storeurl.pl: (13) Permission denied
    squid terminate abnormally but when i start rc.local manually in terminal after login to to ROOT then it starts normally.
    plz guide me what to do and i m googling it if i get some then i’ll post but if u know then post before me it will help us alot.
    thxxzzzzz for ur king work for peoples.

    Comment by rehmat ali — August 17, 2014 @ 2:08 PM

  193. is it true youtube turn over https now ??????? no more caching lol

    Comment by joe — August 24, 2014 @ 10:21 PM

  194. sir i configured as per your instructions. first 5 days its working after that im observing access.log different ips which is not belogs to my network..

    Comment by satish — September 1, 2014 @ 5:02 PM

  195. acl range dstdomain .windowsupdate.com
    range_offset_limit -1 KB range

    what these lines work is these lines only worked for acl range or for any traffic passing from squid?

    Comment by rehmat ali — September 2, 2014 @ 4:06 AM

  196. Put this line at squid.conf to cache youtube

    acl reverbnation url_regex -i reverbnation.*(audio_player|ec_stream_song).*$
    acl reverbnation url_regex -i \.c\.(reverbnation|c2lo)\.com\/(get_audio|audioplayback|audioplay).*$
    acl youtube url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204).*$
    acl youtube url_regex -i (youtube|google).*\/videoplayback\?.*
    acl dontrewrite url_regex redbot\.org

    Comment by Muhammad — September 9, 2014 @ 7:10 PM

  197. I found this error:
    “Can’t locate File/ReadBackwards.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /etc/squid/storeurl.pl line 147.
    BEGIN failed–compilation aborted at /etc/squid/storeurl.pl line 147;”

    where is ReadBackwards.pm located?

    Comment by Us Jauhari — September 16, 2014 @ 9:26 AM

  198. Execution of /etc/squid/storeurl.pl aborted due to compilation errors.
    Missing right curly or square bracket at /etc/squid/storeurl.pl line 167, at end of line

    help lol 😀

    Comment by amgad — September 17, 2014 @ 4:14 AM

  199. How can i Start/Stop/Restart squid?….. is there any init script….. ?

    Comment by Md.Murshed — September 21, 2014 @ 1:40 PM

  200. How can i Start/Stop/Restart squid?….. is there any init script..?

    Comment by Md.Murshed — September 21, 2014 @ 1:42 PM

  201. I’ll try this, thanks for share, Allah may bless us

    Comment by Rudi — September 22, 2014 @ 10:36 PM

  202. how can i clear cache for squid , what’s command to clear cache , or reset configuration it ??????

    Comment by mohamed — September 23, 2014 @ 11:32 PM

    • T clear cache, first stop squid service,
      then remove the folder inside your cache directory,
      then re initialize squid folders again by command
      squid -z
      now start the service, cache will be cleared

      Comment by Syed Jahanzaib / Pinochio~:) — September 24, 2014 @ 8:49 AM

      • how i stop the squid ,
        how i remove the folder inside my cache directory *i set the same configuration in this Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue [April, 2014 , zaib] i not change any thing

        so please tell me the commands in details

        Comment by mahmoud — September 25, 2014 @ 2:49 PM

      • PLZ …what command to stop , remove the folder in cache , cause my hard disk is full now , i need command to clear cache after 5 days , and i need to take backup from squid for any problem i will restore it……Thanks for reply .

        Comment by mohamed — September 26, 2014 @ 11:54 AM

  203. hi sir
    how can i check the youtube video serving from cache
    because the bandwidth not incresed when the squid connected with mikrotik

    Comment by mohammad_alsharqi — September 26, 2014 @ 10:13 PM

  204. Hi, thank you for your great effort.

    I implemented this, and having an issue that the same (first loaded) video now loads for any requested video.

    Firstly some background:

    I am trying to make my daughters YT videos cache as she watches them over and over, and if my network is busy, they stutter.
    She uses them from a child locked app (android phone), that plays approved YT videos from within the app.
    As far as I can see this is just a shell around a html browser.

    I am using lusca as a standalone – so box is ubuntu server, and is not the router as well.

    On the first video load I get this from the log: (this is on the second load f this video, so you can see it got cached fine, as it is a TCP HIT

    http://gdata.youtube.com/feeds/api/videos/wTEqV0MTVvU
    TCP_MISS/200 6832 GET http://gdata.youtube.com/feeds/api/videos/wTEqV0MTVvU – DIRECT/58.162.60.148 application/atom+xml
    http://www.youtube.com/get_video_info?&video_id=wTEqV0MTVvU
    TCP_MISS/200 23714 GET http://www.youtube.com/get_video_info?&video_id=wTEqV0MTVvU – DIRECT/58.162.60.148 application/x-www-form-urlencoded
    TCP_HIT/200 5121136 GET http://r3—sn-uxanug5-2xge.googlevideo.com/videoplayback?ipbits=0&mm=31&signature=AA901EA6DD59B7EFD1021942671F400B1ADF753C.F8A3CB18A21A53FD384AF4988D7E0540FE493DC1&upn=PBmnTUALRpc&sver=3&expire=1411798569&fexp=908550%2C914005%2C917000%2C924622%2C927622%2C930666%2C931983%2C932404%2C934030%2C943415%2C946013%2C947209%2C951904%2C952302%2C953801&sparams=id%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmm%2Cms%2Cmv%2Cratebypass%2Csource%2Cupn%2Cexpire&ms=au&mt=1411776894&mv=m&ratebypass=yes&source=youtube&key=yt5&ip=120.146.232.86&initcwndbps=613750&id=o-AEEdGv8suyp-dLwa1PrjL6urqAr3DcwWx-onhCydGNos&itag=18&signature=null – NONE/- video/mp4

    and on a second, different video load

    http://gdata.youtube.com/feeds/api/videos/cE3LzPQgQmk
    TCP MISS/200 3802 GET http://gdata.youtube.com/feeds/api/videos/cE3LzPQgQmk – DIRECT/58.162.60.153 application/atom+xml
    http://www.youtube.com/get_video_info?&video_id=cE3LzPQgQmk
    TCP_MISS/200 27883 GET http://www.youtube.com/get_video_info?&video_id=cE3LzPQgQmk – DIRECT/58.162.60.177 application/x-www-form-urlencoded
    TCP_HIT/200 5121136
    GET http://r7—sn-uxanug5-2xge.googlevideo.com/videoplayback
    ?sparams=id%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Cmm%2Cms%2Cmv%2Cratebypass%2Csource%2Cupn%2Cexpire&ipbits=0&fexp=916637%2C927622%2C930663%2C931983%2C932404%2C934030%2C935662%2C944907%2C946012%2C947209%2C952302%2C953801%2C955400&mm=31&id=o-ACEgeYP7ZME__vHj7VvKyjKotgsVVzqSXiEs6sLFDJqs&source=youtube&mv=m&ms=au&ip=120.146.232.86&initcwndbps=652500&mt=1411777015&signature=F4EA75E3FDB7E3E5C51B045ED357CD90C319D0A7.061D74258239E08523B7D1288BD7D5E65237AF1E&upn=7l6fc_oHuSU&ratebypass=yes&key=yt5&sver=3&expire=1411798730&itag=18&signature=null – NONE/- video/mp4

    But this (and any other videos) end up being the first video watched. – So the system is always serving that fist vide for any YT requests.

    Any suggestions?

    Comment by Lucas — September 27, 2014 @ 2:51 PM

  205. Please Help ……
    On Debian 6 …..
    checking for a BSD-compatible install… /usr/bin/install -c
    checking whether build environment is sane… configure: error: newly created file is older than distributed files!
    Check your system clock

    Comment by ian — October 3, 2014 @ 7:04 AM

    • Ian just update/correct your system clock. Like using ntpdate -u pool.ntp.org or using date command, maybe is this your issue.

      Comment by Kiko — October 14, 2014 @ 12:44 AM

  206. kalau buat cache game facebook bagaimana ya? ini game 8 pool tidak tercache, terima kasih sebelumnya.

    Comment by Muhammad Rifqi — October 9, 2014 @ 10:13 AM

  207. Hello this Squid / Lusca are ipv6 compatible ? Can enable ipv6 ?

    Comment by Kiko — October 14, 2014 @ 12:43 AM

  208. WOW sir … it works like a charm, i tried other tutorials but they failed with the youtube advertisements but this one is awesome and i dont get any issue of ads … you are hero bro .. Allah Ap ko lambi umer day our aisay hi app logo ki help kartay rahain .. Ameen

    Comment by Malik — October 19, 2014 @ 2:00 PM

  209. Sallam sir, everything works fine but youtube cache doesnt its always Miss_Direct …. for transparnet proxy i have to forward traffic from Ubuntu otherwise it doesnt work but if i add proxy settings manually in my client browser it works ..
    echo 1 > /proc/sys/net/ipv4/ip_forward

    what should i do to get youtube cache working, everything else is fine 🙂 at least with the manual proxy we get rid of ads from youtube hehe ..

    Comment by Malik — October 19, 2014 @ 6:08 PM

  210. dear janzaib,
    after this script 2 times in my squid server i found superblocks bad blocks, i change my harddrive 2 times in duration of 4 months
    what to do plzzz guide me my squid is connected with ups and power failer on squid server during 4 months is maximum 30 times i guess,
    plz tell me to choose right hard drive,
    i have 2 tb sata on /usr/local/cache mount
    and 76gb SCSI drive on boot and root

    Comment by rehmat ali — October 26, 2014 @ 11:49 AM

  211. I have IBM X3400 M3 server, QUARD CORE XEON, 4GB RAM. SAS HDD 250 GB, 80 GB sata, want the proxy server settings, we use at home / cafe in rural areas, have a suggestion for this? whereas currently has wear LUSCA_HEAD-r14942, which has been the optimization of these links, good results, and earlier thanks, I wait for suggestions or no squid were more suitable for PC I’ve had this

    Comment by satya nirmala — October 29, 2014 @ 2:23 PM

  212. Hi Dear Jahanzaib,
    I have followed the tutorial as i have done it previously with other methods of squid, and its done well. But facing one problem which should be addressed. When I browse youtube or other sites and it cache them very well on the same PC (in ubuntu), but when I connect it to Mikrotik with DST Nat & Route methodology (ZPH doesnt work at my side doesnt know why), then I see these errors in Squid Debug mode:

    ClientEatRequestBodyHandler: FD 83: No more data left in socket; but request header says there should be; aborting for now

    above error continuously coming with different FD numbers. I have searched on internet and it looks lusca bug, but to run my cache smoothly i have to address this problem. Can you please give suggestion on it; how to solve it?

    Comment by Kabir Aftab — November 1, 2014 @ 2:27 PM

    • well YT will not go in cache probably, because now youtube have shifted its CDN to HTTPS , and ssl is not cacheable (officially).

      Comment by Syed Jahanzaib / Pinochio~:) — November 6, 2014 @ 8:25 AM

  213. Dear Syed Bhai,

    Is there any script to delete old cache video as there is not enough space in cache folder or is there any way to auto delete old cache video???

    Comment by Md.Murshed — November 2, 2014 @ 10:53 AM

    • actually you cannot distinguish VIDEO files from the squid cache, so in short, no official method to do this. you have to clear the whole cache.

      Comment by Syed Jahanzaib / Pinochio~:) — November 6, 2014 @ 8:24 AM

  214. Dear sir,

    I followed as you mentioned but when i try to open this

    ———————————————————————————————————————————
    http://iso.esd.microsoft.com/W9TPI/FD54DF81A4CCF4511BA1445C606DDBA2/WindowsTechnicalPreview-x64-EN-US.iso
    ———————————————————————————————————————————-

    it is showing an error like this

    ———————————————————————————————————————————–

    While trying to process the request:

    GET /W9TPI/FD54DF81A4CCF4511BA1445C606DDBA2/WindowsTechnicalPreview-x64-EN-US.iso HTTP/1.1
    Host: iso.esd.microsoft.com
    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:33.0) Gecko/20100101 Firefox/33.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Referer: http://windows.microsoft.com/en-us/windows/preview-iso
    Cookie: MC1=GUID=bcb2aa67b78d5f4fa8684cdd44da709d&HASH=67aa&LV=201411&V=4&LU=1415179406865; MS0=3d2ae6313a554302b666dd2a15ac7219; A=I&I=AxUFAAAAAACqCAAAozBb8OdL4Ih3D1ijqfghyA!!&V=4; MSFPC=ID=bcb2aa67b78d5f4fa8684cdd44da709d&CS=3&LV=201411&V=1
    Connection: keep-alive

    ———————————————————————————————————————————-

    please help me…

    Comment by satish — November 5, 2014 @ 2:38 PM

  215. Dear sir,

    Your site is very helpfull to people like us!!!

    I am big fan to your site.

    when i try to acces below site

    http://iso.esd.microsoft.com/W9TPI/FD54DF81A4CCF4511BA1445C606DDBA2/WindowsTechnicalPreview-x64-EN-US.iso

    i configure squid as you mention but when i try to access below it is showing error like

    GET /W9TPI/FD54DF81A4CCF4511BA1445C606DDBA2/WindowsTechnicalPreview-x64-EN-US.iso HTTP/1.1
    Host: iso.esd.microsoft.com
    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:33.0) Gecko/20100101 Firefox/33.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Referer: http://windows.microsoft.com/en-us/windows/preview-iso
    Connection: keep-alive

    please help me…

    Comment by satish — November 5, 2014 @ 3:32 PM

    • Well this doesn’t seems to be an error.
      what problem you are facing at user end? what error ?

      Comment by Syed Jahanzaib / Pinochio~:) — November 6, 2014 @ 8:23 AM

      • Thanks for your fast and valuable reply sir…

        User want to access this site. Without our proxy server some iso file is downloading but with our proxy it is displaying this type of error.

        Is there any way to over come this issue.

        Comment by Satish — November 10, 2014 @ 6:43 PM

  216. youtube firefox cache but chrome explorer youtube not cache why please help me

    Comment by James — November 17, 2014 @ 5:42 AM

  217. 1416183852.146 2234 10.42.0.92 TCP_MISS/200 587973 GET http://r2—sn-nv47en76.googlevideo.com/videoplayback?
    gcr=tr&itag=248&id=o-ADcumbvMvnSlKiBSzzGuEnIZrx3OZAZCXXpDiRyjDkcX&key=yt5&ip=78.180.46.208&initcwndbps=822500&g
    ir=yes&sparams=clen%2Cdur%2Cgcr%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cm
    s%2Cmv%2Csource%2Cupn%2Cexpire&source=youtube&lmt=1411522166893831&expire=1416205357&dur=269.060&mt=1416183692&
    mv=m&clen=78808010&ms=au&fexp=3300115%2C3300115%2C3300132%2C3300132%2C3300137%2C3300137%2C3300164%2C3300164%2C3
    310366%2C3310366%2C3310700%2C3310700%2C3312232%2C3312232%2C3312478%2C3312478%2C905639%2C907259%2C908585%2C91400
    5%2C927622%2C932404%2C938694%2C943909%2C945246%2C947209%2C947215%2C948124%2C952302%2C952600%2C952605%2C952901%2
    C953912%2C957103%2C957105%2C957201&sver=3&ipbits=0&mime=video%2Fwebm&upn=evOzqXUrJug&mm=31&keepalive=yes&cpn=Zx
    oqOu4M6QMWQFVT&alr=yes&ratebypass=yes&signature=6C33D890913886F63DC886F5F85058B38C246DF8.223B0049C2F7A10D53FB19
    FF0DA8FEF8DAD3400D&c=WEB&cver=html5&range=27976754-28564204 – DIRECT/74.125.11.71 video/webm

    chrome log no cache

    Comment by James — November 17, 2014 @ 5:45 AM

  218. Hi! Thanks for ur howto, btw automake1.9 is now automake1.11 (on last debian stable).
    cya!

    Comment by Max — November 17, 2014 @ 9:06 PM

  219. hi sir, now youtube using https , how to update squid to make it working with https

    Comment by Mahal — November 24, 2014 @ 2:38 PM

  220. youtube update nov 2014 ? plis

    Comment by newbangherbangher — December 5, 2014 @ 5:15 PM

  221. few month ago this script still work, if you log out and cleared cookies in google chrome, but now every we open youtube it redirect to https, some time if used firefox youtube still cache in proxy if the globe color is grey like this https://support.cdn.mozilla.net/media/uploads/gallery/images/2014-06-26-21-38-46-78f915.png i am looking away for disable https on all google site but it seem cant be done now

    Comment by dioey — December 11, 2014 @ 3:18 PM

    • @dioley, you cannot redirect https to http or it will break the certificate adn it will become invalid/unavailable.
      caching https is a headache for normal purposes. You need some very valid sort of certificate which is yearly paid solution.

      Comment by Syed Jahanzaib / Pinochio~:) — December 13, 2014 @ 9:51 AM

  222. admin give me email address or skyp please

    Comment by Jaze — December 13, 2014 @ 4:54 PM

  223. admin how to get report in squid log

    Comment by tajudeen — January 3, 2015 @ 5:50 PM

  224. does it work in https ?

    Comment by Sanyog Maharjan — January 4, 2015 @ 3:37 PM

  225. Hello Syed and thank you for this helpful post!
    It has been successfully working in a Raspberry Pi (Moebius Debian flavour) from this summer 2014. But now, as you might have noticed, youtube is forced to use https connection and this makes it impossible to be cached. Is there any solution with this Squid fork?
    Thank you in advance 🙂

    Comment by Francisco Fiesta Segura — January 6, 2015 @ 5:42 PM

  226. do you have any updates ? i see that youtube is using webm now.

    Comment by Piju — January 13, 2015 @ 7:51 PM

  227. hello Syed, thank you for sharing your knowledge. I am wondering if there is a way to make this work with pfsense 2.1.5 squid 2.7 or 3.3? Thank you!

    Comment by Justin — January 23, 2015 @ 2:26 AM

  228. youtube https and not caching

    Comment by James — January 27, 2015 @ 4:54 PM

  229. salam sir ijust want to know in this line 43200 is showing mint or seconds ?
    REFRESH PATTERN 43200 999% 43200

    Comment by rehmat ali — February 5, 2015 @ 11:13 AM

  230. Assalamualaikum warahmatullahi wabarakatuh

    Sir How to fix this error 🙂

    root@Cache-Server:~# tcpdump -vni eth0 | grep ‘tos 0×30′
    grep: 0×30′: No such file or directory
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    16 packets captured
    17 packets received by filter
    0 packets dropped by kernel
    #####################################################
    root@Cache-Server:~# tcpdump -vni eth1 | grep ‘tos 0×30′
    grep: 0×30′: No such file or directory
    tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
    19 packets captured
    19 packets received by filter
    0 packets dropped by kernel

    Comment by Mubashir — February 11, 2015 @ 2:06 PM

  231. when i try to download any iso from internet squid gives this error

    ERROR
    The requested URL could not be retrieved

    While trying to process the request:

    GET /centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-Everything.iso HTTP/1.1
    Host: mirrors.nayatel.com
    User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: en-US,en;q=0.5
    Accept-Encoding: gzip, deflate
    Referer: http://mirrors.nayatel.com/centos/7.0.1406/isos/x86_64/
    Connection: keep-alive

    The following error was encountered:

    Invalid Response

    The HTTP Response message received from the contacted server could not be understood or was otherwise malformed. Please contact the site operator. Your cache administrator may be able to provide you with more details about the exact nature of the problem if needed.

    all works fine except iso downloads
    please guide me

    Comment by rehmat ali — February 26, 2015 @ 9:32 PM

  232. Hello, how could apply these lines lusca?:
    #———————————————————————-
    acl google url_regex -i (googlevideo.com|youtube.com)
    request_header_access User-Agent deny google
    request_header_replace User-Agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    #———————————————————————-
    in SQUID3.x this is used to not load https in youtube….

    Comment by JORGE — March 24, 2015 @ 8:20 AM

    • This way worked for me in this project:
      acl google url_regex -i (googlevideo.com|youtube.com)
      header_access User-Agent deny google
      header_replace User-Agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

      so, all you have to do is to replace new request_header_access by the old header_access and request_header_replace by header_replace.
      I have written those rules after this line:
      header_access X-Forwarded-For deny all

      Comment by Francisco Fiesta Segura — May 17, 2015 @ 10:34 PM

  233. not working why ?

    Comment by Łukasz — April 13, 2015 @ 12:19 AM

  234. olar’m problem, with a few androd mobile does not work with this conf squid someone is having the same problem?

    Comment by Edilson — May 10, 2015 @ 5:14 PM

  235. please update to new squid 3.5.4

    Comment by dijeeponsel — May 22, 2015 @ 1:57 AM

  236. Dear find torrent server (torrent cache p2p).
    please help and reply ASAP

    Comment by jatin — June 8, 2015 @ 9:29 PM

    • Dear find torrent server (torrent cache p2p).
      please help and reply ASAP

      Comment by jatin — June 8, 2015 @ 9:30 PM

  237. Estimated as I can empty the cache? and other questions as you might see it in graphic form for storing so far ???? thanks regards from Argentina

    Comment by Walter — August 13, 2015 @ 8:39 AM

  238. So youtube reopened in pakistan. dear Ziab have to tried new method using ssl-bump ? I so please share with us.

    Comment by Nasir Javed — January 18, 2016 @ 9:29 PM

  239. can someone help me regarding LAN & WAN part in above topic..i cant connect to internet after installing the configuration

    Comment by Syed Muhammad — February 17, 2016 @ 3:26 AM

  240. hi,

    i followed the steps mentioned by you and facing an issue while restarting my squid3. When i am trying to restart the following message will be appeared.

    root@ubuntu:~# sudo service squid3 start
    start: Job failed to start

    What to do now?

    Comment by vbrao@esflabs.com — March 18, 2016 @ 11:19 AM

  241. Too bad FB can’t be cached in this fashion as they are now requiring https :/

    Comment by Terry Sine — March 27, 2016 @ 4:20 PM

  242. http://www.squid-cache.org/Doc/config/https_port/

    What is that , is it possible

    Comment by Raqib Abdur — June 2, 2016 @ 2:21 AM

  243. Please help me…
    I run the first code and got this msg..
    E: Unable to locate package automake1.9
    E: Couldn’t find any package by glob ‘automake1.9’
    E: Couldn’t find any package by regex ‘automake1.9
    How can I solve my problem?
    I am using Ubuntu 16.04

    Comment by Deep Dey — June 15, 2016 @ 2:50 PM


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: