Syed Jahanzaib Personnel Blog to Share Knowledge !

January 30, 2012

Youtube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.pl


YOUTUBE CACHING WITH SQUID -by- Syed Jahanzaib

YOUTUBE CACHING WITH SQUID -by- Syed Jahanzaib

The last updated guide 22nd April, 2014 about youtube caching is as follows …

http://aacable.wordpress.com/2014/04/21/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/


 

=========================================================================================================

Last updated > 20th January, 2014

If you are caching YouTube using storeurl.pl method

EXAMPLE:

http://aacable.wordpress.com/2014/04/21/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/

http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/ 

and you encounter following error while watching any cached video  “An error occured, Please try again later
As showed in the image below . . .

Then try using the following January, 2014 updated storeurl.pl

#!/usr/bin/perl
# This script is NOT written or modified by me, I only copy pasted it from the internet.
# It was First originally Written by chudy_fernandez@yahoo.com
# & Have been modified by various persons over the net to fix/add various functions.
# Like For Example modified by member of comstuff.net to satisfy common and dynamic content.
# th30nly @comstuff.net a.k.a invisible_theater , and possibly other people too.
# For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
# Syed Jahanzaib / aacable@hotmail.com
# http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/

#####################
#### REFERENCES #####  http://www2.fh-lausitz.de/launic/comp/misc/squid/projekt_youtube/
#####################
#####################

## storeurl.pl-130411-0.05
## s. 130411.howto.squid.youtube.html
## CHANGES
## 130411 hl+mf
## - fix youtube loop redir
## - rel. 0.05
## 130409 hl
## - add usleep
## 120726 hl
## - add printtimenow
## 120725 hl
## - remove html-tags
## - add debug
## - disable unused rules
## - rel. 0.04
## 120130 aacable
## - fix match variable.domain.com
##   http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/
## 120111 aacable
##   http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/

#### notes
## - config (s. 130411.howto.squid.youtube.html)
##   squid.conf:storeurl_rewrite_program /etc/squid/storeurl.pl
## - test
##   $ cat squid.diag.youtube.url | awk '{print "0",$7,"10.0.0.1/- - GET - myip=10.0.0.2 myport=3128"}' | ./storeurl.pl
#
#
################################################################################################
## UPDATED on 20 January, 2014 **** thanks to Mr. Safatah Purwonoto [Indonesia] / Syed Jahanzaib
################################################################################################

#### var

use IO::File;
$|=1;
STDOUT->autoflush(1);
$debug=1;        ## recommended:0
$bypassallrules=0;    ## recommended:0
$sucks="";        ## unused
$sucks="sucks" if ($debug>=1);
$timenow="";
$printtimenow=1;      ## print timenow: 0|1
my $logfile = '/tmp/storeurl.log';

open my $logfh, '>>', $logfile
or die "Couldn't open $logfile for appending: $!\n" if $debug;
$logfh->autoflush(1) if $debug;

#### main
while (<>) {
$timenow=time()." " if ($printtimenow);
print $logfh "$timenow"."in : $_" if ($debug>=1);
chop; ## strip eol
@X = split;
$x = $X[0]; ## 0
$u = $X[1]; ## url
$_ = $u; ## url

if ($bypassallrules){
$out="$u";    ## map 1:1

#youtube with range (YOUTUBE has split its videos into segments)
}elsif (m/(youtube|google).*videoplayback\?.*range/ ){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@range = m/[&?](range=[^\&\s]*)/;
@begin = m/[&?](begin=[^\&\s]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
$out="http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range&@redirect";
#sleep(1);    ## delay loop

#youtube without range
}elsif (m/(youtube|google).*videoplayback\?/ ){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
$out="http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@redirect";
#sleep(1);    ## delay loop

#speedtest
}elsif (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
$out="http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . "";

#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
$out="http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . "";

#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
$out="http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . "";

#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
$out="http://www.filesonic.com.SQUIDINTERNAL/" . $1 . "";

#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
$out="http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3";

#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
$out="http://www.4shared.com.SQUIDINTERNAL/$2";

#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
$out="http://photos" . $1 . "/" . $2 . $3  . "";

#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
$out="http://photos.ak.fbcdn.net/" . $1  ."/". $2 . "";

#maps.google.com
} elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {
$out="http://" . $1  . $2 . "";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
$out="http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "";
#sleep(1);    ## delay loop

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
$out="http://www.google-analytics.com/__utm.gif";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
$out="http://" . $1 . $2 . $y . "";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
$out="http://" . $1 . "/" . $2  . "";

# spicific servers starts here....
} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
$out="http://" . $1 . "";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
$out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
$out="http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
$out="http://" . $1 . "/SQUIDINTERNAL/" . $3 . "";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
$out="http://cdn." . $4 . $6 . "";

#...spicific servers end here.

#photos-X.ak.fbcdn.net where X a-z
} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
$out="http://photos.ak.fbcdn.net/" . $1  . "";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
$out="http://cdn.yimg.com//" . $3 . "";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
$out="http://cdn.yimg.com/"  . $3 . "";

#for yimg.com with &sig=
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+[0-9]+/cdn/;
$y[1] =~ s/&sig=.*//;
$out="http://" . $y[0] . ".yimg.com/"  . $y[1] . "";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
$out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
$out="http://squid-cdn-url//" . $2  . "." . $3 . "";

#generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %"
} elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/(([a-zA-Z]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/;
$out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "";

} else {
$out="$u"; ##$X[2]="$sucks";
}
print $logfh "$timenow"."out: $x $out $X[2] $X[3] $X[4] $X[5] $X[6] $X[7]\n" if ($debug>=1);
print "$x $out $X[2] $X[3] $X[4] $X[5] $X[6] $X[7]\n";
}
close $logfh if ($debug);

.

Please post your comments regarding updated storeurl.pl

Regard’s
SYED JAHANZAIB

About these ads

242 Comments »

  1. squid is not running, after i update. why ?

    thanks.

    Comment by neo — January 31, 2012 @ 8:10 AM

    • any solutions on pfsense+lusca, i have the same problem regarding with the youtube error.. i tried reinstalling everthing from scratch but still the same — youtube error

      Comment by Incognito — January 31, 2012 @ 7:30 PM

      • So far I have not found any 100% solution for this issue. However after adding negative_ttl 0 seconds , situation gets much better almot 80-90% problem solved. Try it

        I also heard from some folks that php method works 100% without error. you can try that, google for it.

        Comment by Syed Jahanzaib / Pinochio~:) — January 31, 2012 @ 7:43 PM

      • Thanks for your post Syed. Where do i put or which file do i edit to add the “negative_ttl 0 seconds?.. when i tried opening the file squid.conf, i didn’t see any.. any guides? thanks again.. :)

        Comment by Incognito — February 3, 2012 @ 6:33 AM

      • add negative_ttl 0 seconds any where , preferably before Refresh Patterns.

        Comment by Syed Jahanzaib / Pinochio~:) — February 3, 2012 @ 6:52 PM

      • Thanks Syed for the quick feedback.. You’re very helpful. thank you….

        Comment by Incognito — February 3, 2012 @ 9:22 PM

  2. Hi Syed. I have the same issues and adding that line in squid, situation gets better but sometimes the error returns. Would you like to contact me so we can find he solution together?

    Comment by Mario — February 3, 2012 @ 8:06 PM

  3. This issue occurs when youtube loads video partially. I looked on youtube url video part by part, and saw &range= at the end. So I tried to change regex on storeurl.pl and seems working until now. And all parts of a video are cached and ‘hit’ normally.

    sorry for my bad english

    Comment by lif2k3 — February 4, 2012 @ 4:59 PM

    • That would be wonderful, But try it for 2-3 days on testing and then do let me know,
      Post the updated storeurl.pl , maybe I will be able to fine tune it too :)

      Thx for the contribution

      Comment by Syed Jahanzaib / Pinochio~:) — February 4, 2012 @ 5:58 PM

  4. this picture may explain how youtube loads video partialy :

    and this image shows my access log while logging a youtube video part by part.

    as you can see, several parts of a video are succesfully load from cache (HIT), then rest part of it is still ‘miss’. However, a video could be loaded part by part or normally. So, a video could take double size on cache swap (partially video and normal video).

    Again, sorry for my bad english :(

    Comment by lif2k3 — February 5, 2012 @ 2:03 PM

    • Can you confirm if the problem have been resolved ?
      Can you post the updated storeurl.pl ??

      Comment by Syed Jahanzaib / Pinochio~:) — February 5, 2012 @ 3:45 PM

      • No problem appear since 2 days ago.. and running well. No “an error occured’ again.

        hmm.. do u have IM contact? YM, Gtalk, or skype?

        We can talk about this later, if u mind.

        Comment by lif2k3 — February 5, 2012 @ 5:59 PM

      • The problem is I rarely use MSN.
        My email is aacable [at] hotmail.com
        You can email me for any discussion.

        Comment by Syed Jahanzaib / Pinochio~:) — February 5, 2012 @ 9:07 PM

  5. Hi, is there a new way to fix youtube error?.. after i add negative_ttl 0 seconds to squid.conf, problem still occurs as of now. Can you guide me with the new way to fix the problem? Thanks in advance.

    Comment by Incognito — February 8, 2012 @ 8:51 AM

    • negative_ttl only better the situation, but not 100%
      There is no absolute fix for this problem at a moment, At least I am not aware of it :)
      If you find one, do let me know.

      Comment by Syed Jahanzaib / Pinochio~:) — February 8, 2012 @ 8:57 AM

    • What’s the result of this storeurl.pl
      Is there any improvement in it ?

      Comment by Syed Jahanzaib / Pinochio~:) — February 9, 2012 @ 9:00 AM

      • now, video youtube with file range, so if you use your old storeurl that error acure come to you. sorry if my english not good. take a look at my capture log

        Comment by syaifuddinjw — February 10, 2012 @ 12:00 AM

      • Hmmm,, can you please post updated storeurl.pl , or email me. It would be great help for our open source community. :)
        Ok Can you provide me with your email address for communication.
        My email is aacable at hotmail.com

        Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2012 @ 9:02 AM

  6. Okay, Tnx Syed, i’ll post some info regarding this problem if I could find one.. this one is good at the moment – at least..
    Thanks for the big help. :)

    Comment by Incognito — February 8, 2012 @ 4:05 PM

  7. with the squid.conf file you provided, can you also cache software and large files? any type of file?
    Thank you

    Comment by Try — February 9, 2012 @ 10:41 PM

    • You can define file size in squid.conf

      Everything which is cache-able goes in cache.
      Or you can use some advance scripting like storeurl.pl to force cache for some non cacheable objects, for example you-tube videos, antivirus updates, windows updates etc. Yes its possible.

      Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2012 @ 8:59 AM

  8. hai syed, if you ask about new storeurl, so you can take a look at my pastebin

    http://pastebin.com/e3TUtigH

    Comment by syaifuddinjw — February 10, 2012 @ 1:16 PM

  9. Heh, a little bug with new storeurl.pl script, or maybe squid… If you fastforward song somewhere after reload it starts from that place… i hope that you understand me… ;)

    Comment by nenad — February 11, 2012 @ 1:04 AM

    • Well I have configured the new updated storeurl.pl at two different cable.network setup, and so far , its working great. much mroe better then its previous version, so I can confirm that its working good.

      Comment by Syed Jahanzaib / Pinochio~:) — February 11, 2012 @ 11:43 AM

      • Did you test with Lusca or squid or both ?

        Comment by Damir — February 16, 2012 @ 8:01 PM

      • Yes I have tested it with LUSCA and SQUID 2.7
        Both works fine.

        Comment by Syed Jahanzaib / Pinochio~:) — February 17, 2012 @ 8:52 AM

    • maybe you can try to reconfigure your squid config

      quick_abort_min
      quick_abort_max

      and read the manual

      Comment by syaifuddinjw — February 11, 2012 @ 7:23 PM

  10. very big thanks to syed there is not more error on youtube …. works very fine with me

    Comment by hamid — February 15, 2012 @ 4:06 AM

  11. sou do brasil, por favor me ajudem a instalar o squid no Debian
    obrigado ;)

    Comment by Diogo Pereira — February 15, 2012 @ 6:49 PM

  12. I’m from Brazil, please help me install squid on Debian
    thank you ;)

    Comment by Diogo Pereira — February 16, 2012 @ 2:36 AM

  13. Syed
    i Would like to know your opinion to tell me what hardware can i use for approximatelly
    200 active users every one having a bantwith 384kbps-512kbps
    with a network of 25Mbps and a bantwith of 1.5TB within a month of traffic used .

    I want to build a squid without caching .flv files and files larger than 150Mb .
    What do u thing , what hardware computer do u prefer for me like DELL Servers , IBM and which type of that if u like any of them .

    By the way i would like to buy any without needing much energy (if any exist with at least 500W)

    Comment by nori — February 17, 2012 @ 8:47 PM

    • It doesn’t matter what brand you go for.
      Any Dual base processor machine with enough RAM (min of 4 GB or Plus) can easily accommodate your 200+ users load. For example

      Xeon 2.4G hz OR 3.0 GHz Dual / 4GB RAM / 2 HDD on proxy, one for OS and Logs and one reserved for caching , like 500GB x 2 HDD

      It all depends on your network scenario and Budget you have. Mikrotik doesn’t requires much heavier hardware for this amount of users, Also Mikrotik supports only 2 GB of ram maximum.
      but for proxy, the better hardware you put on it, better performance you will get from it.

      Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2012 @ 1:00 PM

      • Im runing mikrotik on intel atom (1,6Ghz) with 2 GB RAM , with proxy , lot of firewall rules … and cpu is only 5% with 20 users :)

        Comment by Damir — February 18, 2012 @ 1:20 PM

  14. thx men :) i’ve more than 8 mikrotiks all RB433 or Rb435 and i want to build a squid proxy server for my clients
    with just one lan card with 200users :D
    now i’ll buy a xeon 3.0 Dual with 8GB Ram and 1TB HDD :)
    what i want to know if possible is :
    Why are some servers with 2 Power Supply ?
    Do i need to connect both of them in power or can i connect just one and if the Server has 750W
    Does it mean that the server will take that power or if i have jut one HDD and just 8GB ram it will take just less than 750W , because sometimes we have a power cut into my town and it can stay for 3 hours without power :S

    thx anyway for everything :D

    Comment by Nori — February 18, 2012 @ 4:26 PM

    • Some server have 2 power supplies, because they provide fault tolerance, redundancy , means if one power source fails, the second will keep working , like UPS working.
      (Provided power is coming on second when first failure occurs)

      Usually heavy servers requires more power.

      Comment by Syed Jahanzaib / Pinochio~:) — February 19, 2012 @ 7:39 PM

  15. thanks bro, for this perl scripts

    Comment by http://mandailingnatal.com — February 19, 2012 @ 11:20 PM

  16. hahaha…. tonggo ….
    aku baca situse orang luar negeri, tapi ternyata resolvnya dari dalam negeri.
    Bravo, Congratulation My Brother

    Comment by riez — February 20, 2012 @ 7:46 AM

  17. Thx syed :=) wish u all the best :) thx for sharing knowledge

    Comment by Nori — February 22, 2012 @ 2:19 AM

    • Thank you :)
      but from now on, I won’t be able to participate much due to my new job. Now I am highly involved with MICRSOFT / IBM world :)

      Comment by Syed Jahanzaib / Pinochio~:) — February 22, 2012 @ 8:54 AM

  18. Gracias.

    Comment by Spartacus — March 13, 2012 @ 7:48 PM

  19. I modified the lines below, but now storeurl.pl Squid increased hits, but now the storeurl.pl is consuming 100% cpu.

    $url = ;
    while ($url) {

    while () {

    Comment by Moises Souza — March 22, 2012 @ 7:28 PM

  20. youtube error occured again..it started yesterday…T_T

    Comment by kjshoot2ill — March 23, 2012 @ 2:08 PM

    • True dont know why, probably google change something… If someone find new storeurl please post here… Thanks

      Comment by nenad — March 25, 2012 @ 4:42 AM

  21. Updated New STOREURL.PL
    update and comment.

    Comment by Syed Jahanzaib / Pinochio~:) — March 28, 2012 @ 10:53 AM

  22. Testing testing… and tell me please, ignore-must-revalidate, lusca dont give me error on that.

    thanks

    Comment by nenad — March 28, 2012 @ 1:05 PM

  23. Ok I have posted new storeurl.pl today (27th March)

    So far result is Fine. Comment its updates . . .

    Comment by Syed Jahanzaib / Pinochio~:) — March 28, 2012 @ 3:43 PM

  24. Wow… Amazing…. I’d tried & all problem fix…. Thank’s so much. and I’ll try it for more days…!!! Hope it work well… :)

    Comment by Reds Ryan (Rilyandi) (@Golden_Cikampak) — March 28, 2012 @ 6:25 PM

  25. Thanks Bro, it is working now, or… for now :) i have one suggestion, maybe we should support WebM format for youtube…

    Thanks again

    Comment by nenad — March 28, 2012 @ 9:21 PM

  26. any help, when i paste/edit from any source storeurl.pl end restarting squid, my squid wont work, but if paste again with originial storeurl.pl its work again, i’m edit storeurl.pl with notepad++ and save as perl, and try edit with perl editor too, and try from this too just copy paste not edit http://pastebin.com/e3TUtigH, error again, please help, me still newbie

    Comment by tono — March 29, 2012 @ 9:38 AM

    • There must be some copy pasting error , either due to blogs coding, or your notepad saving changing codes.
      use putty to login to your squid box, then from the browser , directly copy paste the content in storeurl.pl in putty

      Start SQUID in debug mode , with following parameters, it will show exactly for what reasons SQUID not restarting.

      squid -d1n

      Comment by Syed Jahanzaib / Pinochio~:) — March 29, 2012 @ 7:18 PM

    • make a new storeurl.pl and execute chmod 777 storeurl.pl.
      It’s all.

      Comment by newbie — March 29, 2012 @ 10:26 PM

  27. thanks for posting because it really helped me.

    Comment by Abenk Ones Skd — March 30, 2012 @ 12:15 PM

  28. Greetings, I apologize for that necessarily have ubuntu with 2 network cards? I ask because I use mikrotik and thundercache used previously only used a network card … thanks in advance

    Comment by Alfredo — March 31, 2012 @ 10:48 AM

    • Most of my posts related to Linux are Ubuntu base, because I found Ubuntu most easy to use Linux flavor. You or other may found any other flavor more suitable for various reasons, As I have my own :)

      For example: By default FEDORA installs SQUID 3 which doesn’t support ZPH by default, you have to compile Squid 2.7 on Fedora in order to use ZPH or Use LUSCA on FED , as LUSCA supports ZPH well.

      On the other hand UBUNTU installs squid 2.7 which supports ZPH so no extra work is required.

      It’s all about personnel preference, in past, FEDORA was my first choice, but from past 2 years UBUNTU have manage to become First :) but don’t worry, You can achieve your goal by using any flavor of Linux, doesn’t matter its Ubuntu , Fedora, Centos , FreeBSD , or any OS with Linux as its base OS / Kernel , it will work :)

      Comment by Syed Jahanzaib / Pinochio~:) — March 31, 2012 @ 3:30 PM

  29. thenks for the post, i try he squid 2.7 and storeurl.pl “update 27 March, 2012″, the youtube video are not served from the cache.
    There is not “An error occured, Please try again later”
    regards

    Comment by laziz — April 1, 2012 @ 1:22 AM

  30. hi
    i use the same squid.conf as “Youtube caching with SQUID 2.7 [using storeurl.pl] ” related at the link : http://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/
    regards

    Comment by laziz — April 2, 2012 @ 1:05 AM

  31. ok I understand but you need to use 2 network cards to use the squid?

    Comment by Alfredo — April 2, 2012 @ 3:08 AM

  32. [...] 2.7 [using storeurl.pl]Mikrotik DUAL WAN Load Balancing using PCC method. Complete Script ! by zaiBYoutube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.plMikrotik 4 WAN Load Balancing using PCC method. Complete Script ! by [...]

    Pingback by Youtube caching with SQUID 2.7 [using storeurl.pl] « Syed Jahanzaib Personnel Blog to Share Knowledge ! — April 2, 2012 @ 9:04 AM

  33. thanks. it’s work no problem

    please share squid.conf lusca/squid

    patching games facebook example empires and allies dll

    mode on

    Comment by ahmad — April 2, 2012 @ 9:09 AM

  34. TOP Banget Kakak

    Comment by gembeel — April 2, 2012 @ 12:36 PM

  35. hi syed is this storeurl support to manage video that error when i was cache it, exmaple when i cache some video wtih 360 res, it finish then i changed to 720 res but youtube does’nt reload from begining video and cache the video even just the end of video. Is there any advice to fix my problem? cause this make much trouble for me!

    Comment by Zahrian — April 2, 2012 @ 11:38 PM

    • oh and more question if i used this to cache youtube
      # youtube fix
      } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
      $p_str = $2;
      $tag = “”;
      $alg = “”;
      $id = “”;
      $range = “”;
      if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
      if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
      if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
      if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
      print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;

      is my old video on proxy still can be used because i used this storeurl from chudy or i must to re cache all youtube videos
      if ($X[1] =~ /(youtube|google).*videoplayback\?/){
      @itag = m/[&?](itag=[0-9]*)/;
      @id = m/[&?](id=[^\&]*)/;
      @range = m/(&range=[^\&\s]*)/;
      print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@id&@itag@range\n”;

      Comment by Zahrian — April 2, 2012 @ 11:43 PM

    • No fix for this yet.

      Comment by Syed Jahanzaib / Pinochio~:) — April 3, 2012 @ 8:37 AM

      • thanks for all and let me know by email if you found how to fix it!!

        Comment by Zahrian — April 3, 2012 @ 11:46 AM

      • oh syed one more question what the different by your youtube storeurl and chudy storeurl is there any bug addition on your youtube storeurl

        Comment by Zahrian — April 3, 2012 @ 2:20 PM

      • This storeurl.pl is not made/modified by me. This script was first designed by Chuddy and have been modified by many persons around the internet.

        Comment by Syed Jahanzaib / Pinochio~:) — April 3, 2012 @ 3:11 PM

  36. Hi… thanks for sharing storeurl.pl….
    I have try to integrated the update storeurl.pl into my squid….. but i didn’t see any youtube get cache :(

    Comment by adjie — April 7, 2012 @ 5:25 PM

  37. it’s working better thanks man ,but still gives errors
    btw is it possible to cache facebook games while it is chosen by the user to use secure browsing (https)?
    Did you find a way to cache those stuff? thank you

    Comment by antonio — April 11, 2012 @ 6:45 AM

  38. Basit Khan.
    basit at ambnet.net

    Sir I have tried it on pfsense and works fine for few days after that it crashed badly….I have search alot of things on internet and found that VARNISH Cache is 10x faster than squid, I will highly appricate if you can provide its tutorial or how to..etc..

    Thanks
    Allah bless you!

    Basit Khan

    Comment by BASIT KHAN — April 11, 2012 @ 4:51 PM

  39. WARNING – Queue congestion Help with this Warning

    Comment by Elfoman — April 14, 2012 @ 5:59 AM

    • Working with multi-threaded disk access (AIO) Squid queues the tasks to be performed and lets the disk controller work through it as fast as it can. This allows Squid to work on other processing tasks for the same request without being held up waiting for slow disks.

      The queue starts off at a default length of 8 queue slots per thread. When that queue space is filled up, Squid will spit out the WARNING, and double the available queue length.

      Up to a few of these is OK under very high load. But if you get them very frequently then it’s a sign that either the disk I/O is overloaded or you have run out of CPU cycles to handle it.

      Workaround

      * A few seconds of these after a clean startup can be ignored. They should decrease exponentially as the queue is automatically adjusted to the load.
      * For Squid expected to run on a busy network, increasing the default AIO threads available can reduce the annoyance. Using fast disks is essential.
      * If these continue without decreasing you need faster disks, or to spread the traffic load over more proxies.

      Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 7:47 PM

  40. Thanks for your post Syed, I have copast, so far so good. thanks again

    Comment by magelar — April 17, 2012 @ 1:57 AM

  41. Hi I’m testing the code perl and my question is whether this caching me well squid 2.7 now when I see the video gives me this message again TCP_REFRESH_HIT/304 202 GET http://www.youtube.com/crossdomain.xml – DIRECT/173.194 .42.4 –

    or is this tomendo me of another proxy that this network before mine thanks

    Comment by Nestor — April 17, 2012 @ 7:21 AM

  42. Syed why sometimes youtube still showed some error, is more configuration needed to fix it?

    Comment by Zahrian — April 18, 2012 @ 3:10 PM

    • oh the errors showed up when files had partial video and full video!! and error happen when squid load partial video? any advice for this problem!!

      Comment by Zahrian — April 18, 2012 @ 3:14 PM

  43. consult this file to run perl or now youtube cache can only cache videos with YouTube API?

    Comment by Nestor — April 18, 2012 @ 3:23 PM

  44. One further question, when you change the resolution of a video, you get the video cached in the new resolution from the point where you switched. And when you want to watch the same video it starts from the point where it was cached earlier. So, you can’t watch it entirely.
    Any ideas about that, or how to make that with a resolution change, the Squid always make the caching from the video starting point?

    Comment by Damian — April 19, 2012 @ 5:27 PM

  45. Hola a todos gracias por todo los que comparten información aquí y gracias sobre todo a Pinochio les comento que a mi hasta ahora me funciona bastante bien el storeurl.pl que descargue de aquí

    Hello everyone thanks for all those who share information here, thanks mainly to Pinochio tell them that my so far it works pretty well I downloaded from here storeurl.pl

    http://www.google.com.ar/url?sa=t&rct=j&q=storeurl.pl&source=web&cd=1&ved=0CCYQFjAA&url=http%3A%2F%2Ffaisal-sani-project.googlecode.com%2Ffiles%2Fstoreurl.pl&ei=1auQT5_ZN8OCtgfuzYH6BA&usg=AFQjCNFDMUWrQsZmDsUoEQRQo2Nu6Ecxkw

    Comment by Nestor Piro — April 20, 2012 @ 6:15 AM

  46. dude, is it really copyright of
    # http://aacable.wordpress.com / Syed Jahanzaib
    # Email: aacable@hotmail.com
    is it u write all of those code ??

    dude, im the writer of this line

    #speedtest
    if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
    print $x . “http://www.speedtest.net.SQUIDINTERNAL/speedtest/” . $2 . “\n”;

    #mediafire
    }elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
    print $x . “http://www.mediafire.com.SQUIDINTERNAL/” . $1 .”/” . $2 . “\n”;
    }elsif (m/^http:\/\/205\.196\.12[0-3]\.\d*\/\w{12}\/(\w*)\/(.*)/) {
    print $x . “http://www.mediafire.com.SQUIDINTERNAL/” . $1 .”/” . $2 . “\n”;

    #fileserve
    }elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
    print $x . “http://www.fileserve.com.SQUIDINTERNAL/” . $1 . “./” . $2 . “\n”;

    #filesonic
    }elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
    print $x . “http://www.filesonic.com.SQUIDINTERNAL/” . $1 . “\n”;

    #4shared
    }elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
    print $x . “http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n”;

    #4shared preview
    }elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
    print $x . “http://www.4shared.com.SQUIDINTERNAL/$2\n”;

    #photos-X.ak.fbcdn.net where X a-z
    }elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
    print $x . “http://photos” . $1 . “/” . $2 . $3 . “\n”;

    #YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
    } elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
    print $x . “http://photos.ak.fbcdn.net/” . $1 .”/”. $2 . “\n”;
    }

    AND ALSO THIS LINE

    # youtube fix by th30nly @comstuff.net
    } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
    $p_str = $2;
    $tag = “”;
    $alg = “”;
    $id = “”;
    $range = “”;
    $uri = “http://$1/videoplayback\?$2″;
    if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
    if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
    if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
    if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $uri =~ s/$range//;$range =~ s/-//; $range =~ s/range=//; }
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;

    }

    i’ just curious how SELFISH you are.
    just see http://lusca.info for prove and timestamping.

    regards

    th30nly a.k.a invisible_theater

    Comment by th30nly — April 21, 2012 @ 12:11 AM

    • Dear th30nly,

      Please accept my appologies, I never said that this or that script is written by ME. i added the chuddy_fernandez email address on top, I added my line only for tracking purposes, it is not copy righted by me.
      Anyhow thank you for pointing the right thing, I have added following note in the post, Please check it and do let me know if there is any further amendments required.
      =========================================================
      This script is NOT written by me, I only copy pasted it from the internet.
      It was originally Written by chudy_fernandez@yahoo.com & Have been modified by various persons over the net to fix/add various functions.
      For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube

      =========================================================

      As you said, you have added few functions to it, please do let me know your email or name, So I will add them to your section lines.

      Once again, storeurl.pl is not my creation, It was just copy pasted and I did mentioned the original author name at the top. However I have now added more info on it. I hope the picture is clear now.

      Regard’s

      Comment by Syed Jahanzaib / Pinochio~:) — April 21, 2012 @ 10:38 AM

  47. you did not mentioned that you write the script, but you may not modify by omiting the header lines (remaks) which is mentioned th30nly had fixed or modify the original script.

    the name of th30nly was omitted from the whole script remarks and then you put your name was mentioned in the header under the original writer, I doubt it for tracking purposes since the remaks lines which mentioned fixed by th30nly still not revised yet

    check these lines below:

    #!/usr/bin/perl
    # writen by chudy_fernandez@yahoo.com
    # http://aacable.wordpress.com /Syed Jahanzaib
    # Email: aacable@hotmail.com
    .
    .
    .
    # youtube fix
    } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
    $p_str = $2;
    $tag = “”;
    $alg = “”;
    $id = “”;
    $range = “”;
    if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
    if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
    if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
    if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;

    .
    .
    these are in our storeulr.pl which is copy paste by you.

    #!/usr/bin/perl
    # store rewrite originally writen by chudy_fernandez@yahoo.com
    # modified by member of comstuff.net to satisfy common and dynamic content.
    #
    .
    .
    .
    # youtube fix by th30nly @comstuff.net
    } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
    $p_str = $2;
    $tag = “”;
    $alg = “”;
    $id = “”;
    $range = “”;
    if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
    if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
    if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
    if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
    .
    .
    .

    if you need the storeurl.pl script writen by th30nly, you may get it free directly from th30nly @comstuff,net

    regards,
    felix yt @comstuff.net

    ps. thank you for your visit at comstuff.net

    Comment by felix yt — April 21, 2012 @ 5:25 PM

  48. Thanks for this storeurl.pl

    Comment by setting-mikrotik.com — April 23, 2012 @ 1:40 AM

  49. thxx for all this tutorials that u have provided 2 us .
    i just want to ask u something as a network it

    1. i’ve dedicated internet access 30Mbps Down/ 30Mbps up but on youtube i’m having problems on caching youtube with your script
    could it be because my servers provider has added cache servers into his internet access
    on squid i see something like that :
    0 80.80.171.19 TCP_HIT/200 657 GET http://o-o.preferred.ipko-prn1.v21.lscache3.c.youtube.com/crossdomain.xml – NONE/- text/x-cross-domain-policy

    what could be the reasen that squid isn’t caching youtube ?

    Comment by N-or — April 25, 2012 @ 4:04 AM

    • I have tested Youtube caching this morning and its working fine at my end using this storeurl.pl
      youtube have also introduced new .webm format which is not cached by default. test various videos

      Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 11:01 AM

  50. i’m not saying that it doesnt work , the squid works great everywhere else just in youtube i see a link like that
    TCP_HIT/200 657 GET http://o-o.preferred.ipko-prn1.v21.lscache3.c.youtube.com/crossdomain.xml – NONE/- text/x-cross-domain-policy

    the link isn’t from youtube it’s from my service provided and i see just crossdomain.xml
    could it be because my provider has a cache server and i can’t get it from his cache server :S

    Comment by N-or — April 26, 2012 @ 1:29 AM

    • post the output of this command

      ps aux |grep squid

      It must display something like this

      root@zaib_linux:/scripts# ps aux |grep squid

      proxy 9106 1.2 1.0 12096 5412 ? Ssl 16:07 0:00 /usr/sbin/squid -N -D
      proxy 9107 0.2 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9108 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9109 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9110 0.2 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9111 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9112 0.2 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      proxy 9113 0.0 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
      root 9133 0.0 0.1 3324 816 pts/0 S+ 16:07 0:00 grep –color=auto squid

      Comment by Syed Jahanzaib / Pinochio~:) — April 26, 2012 @ 4:10 PM

  51. thank mr. syed…i am from indoesia…i am glad and happpy…my caching youtube it`s work because of shared you storeurl…sory my english bad but thank so much…….

    Comment by beni jauhari — April 26, 2012 @ 1:42 AM

  52. waduh ada mastah2 dari forum m*******k …aampun mastah…apa emang dia dapat kopi paste dari orang lain….

    Comment by beni jauhari — April 26, 2012 @ 2:35 AM

  53. by me its caching just a few videos most of videos i cant cache i see just this crossdomain.xml /text – cross domain policy
    does anybody else have this problem or not :)

    thx by the way Syed for every tutorial and information that u have added in this website :)
    i

    Comment by N-or — April 26, 2012 @ 1:06 PM

  54. dont forget to chmod +x storeurl.pl and chown to the user and group of the squid. mine is chown proxy:proxy storeurl.pl.

    Comment by 9m2pju — April 26, 2012 @ 1:59 PM

  55. [...] To fix youtube loading error on squid lusca, we need to modify the storeurl.pl file. Here is the fix by Syed Jahanzaib, http://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-l… [...]

    Pingback by Ubuntu developers: Mohamad Faizul Zulkifli: Squid Lusca Youtube Error Fix | Linux-Support.com — April 26, 2012 @ 2:45 PM

  56. Thanks Syed Jahanzaib for your help ,
    I have a problem with youtube now only cache 1.69 MB about 1 Min from any youtube video first it was work perfect now i don’t know what happen

    Comment by SquidY — April 28, 2012 @ 3:18 AM

  57. thx Syed :) it’s almost perfect ur blog :)

    i’ve tryed the command and i see all that :)
    ur video that u watched i’ve seen others video i get this crossdomain.xml as tcp hit not the link of the video :)
    but i’m letting it so , better to dont cache youtube :)

    thx for everything :) what i coulnd find in this blog is how to configure the mrtg graphs in ubuntu :)
    could u add a tutorial please to add some graphs for squid

    Comment by n-orix@live.it — April 28, 2012 @ 12:30 PM

  58. Hola a todos comento como tengo mi squid uso centos 5 y el storeurl de aqui

    http://pastebin.com/e3TUtigH

    para no tener el error de videos en squid.conf puse

    storeurl_rewrite_program /etc/squid/storeurl.pl
    storeurl_rewrite_children 7
    storeurl_rewrite_concurrency 5000

    fue la forma de estabilizar los videos y cachea todo

    saludos a todos

    Hi all I have commented as squid use my centos 5 and storeurl from here

    http://pastebin.com/e3TUtigH

    to avoid the mistake of videos put squid.conf

    storeurl_rewrite_program /etc/squid/storeurl.pl
    storeurl_rewrite_children 7
    storeurl_rewrite_concurrency 5000

    was how to stabilize the videos and caches all

    Greetings to all

    Comment by Nestor — April 28, 2012 @ 5:48 PM

  59. what’s the way to restrict the server just for lan users ( of public ip’s )
    for example 82.82.162.17-80.80.171.30
    what do i need to change to acl amigos

    Comment by Bledi — April 28, 2012 @ 9:20 PM

  60. Hi Sayed , i test new storeurl.pl and the same problem not solved i can’t caching youtube , because youtube separate video file with extention FLV to 1.69 MB so that youtube doesn’t cache but it cache with new extention WEBM from youtube .

    Comment by SquidY — May 1, 2012 @ 3:54 PM

    • I tested this storeurl.pl yesterday and I was pleased with he results. It was able to cache all videos including 1 – 20 minutes Length. However I haven’t checked webm format.

      Comment by Syed Jahanzaib / Pinochio~:) — May 1, 2012 @ 4:20 PM

  61. i tested this storeurl.pl, and it’s work…..thx

    Comment by nailancool — May 2, 2012 @ 7:28 AM

  62. just cache until 50 second only,….

    Comment by detrackmx — May 3, 2012 @ 1:17 PM

  63. One further question, when you change the resolution of a video, you get the video cached in the new resolution from the point where you switched. And when you want to watch the same video it starts from the point where it was cached earlier. So, you can’t watch it entirely.
    Any ideas about that, or how to make that with a resolution change, the Squid always make the caching from the video starting point?

    Comment by Damian — May 3, 2012 @ 7:01 PM

  64. ni orang tukang COPAS….sebagaian karya ane juga diembat…..parah

    Comment by copas — May 4, 2012 @ 9:20 PM

    • kenapa mas kalo copas??
      kok gini ya orang Indonesia??
      malu2in indonesia aja lu

      orang ini jg membagikan pengetahuannya..
      diatas jg dia menulis

      (Note: I am not author of this script, Please use the links/emails provided here to contact the real authors, I only tested it and the problem seemed to be better up as compare to the previous versions of this script)

      Comment by ivvel — May 5, 2012 @ 11:15 PM

  65. hola, yo uso esta y no tengo problemas,
    if (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
    $p_str = $2;
    $tag = “”;
    $alg = “”;
    $id = “”;
    $range = “”;
    $begin = “”;
    $redirect = “”;
    if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
    if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
    if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
    if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
    if ($p_str =~ m/(begin=[^\&\s]*)/){$begin = “&”.$1}
    if ($p_str =~ m/(redirect_counter=[^\&]*)/){$redirect = “&”.$1}
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . $begin . $redirect . “\n”;

    mantengo la variable $tag para mantener la compatibilidad con los videos que ya tenia cacheados, pero la pueden eliminar si quieren. para evitar el problema de “An error occured, Please try again later“ parchen su lusca o squid con el loop 3xx o sino definan su minimum_object_size mayor o igual a 1k.

    espero les sirva.

    Comment by xspartacus — May 4, 2012 @ 9:30 PM

  66. are the different this storelurl.pl and your old storeurl.pl
    just this code??

    if ($X[1] =~ /(youtube|google).*videoplayback\?/){
    @itag = m/[&?](itag=[0-9]*)/;
    @id = m/[&?](id=[^\&]*)/;
    @range = m/[&?](range=[^\&\s]*)/;
    @begin = m/[&?](begin=[^\&\s]*)/;
    @redirect = m/[&?](redirect_counter=[^\&]*)/;
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@begin@redirect\n”;

    why i can’t applied your new update storeurl.pl to my machine??
    the result of my squid is like this

    2012/05/04 23:23:05| Accepting HTCP messages on port 4827, FD 23.
    2012/05/04 23:23:05| Accepting SNMP messages on port 3401, FD 24.
    2012/05/04 23:23:05| WCCP Disabled.
    2012/05/04 23:23:05| Ready to serve requests.
    2012/05/04 23:23:05| WARNING: store_rewriter #1 (FD 12) exited
    2012/05/04 23:23:05| Too few store_rewriter processes are running
    FATAL: The store_rewriter helpers are crashing too rapidly, need help!

    I’ve been try other way
    like insert updated code to my storeurl.pl
    but the result is same :(

    need help here..
    but i dont know why
    my machine just cache youtube only 51 second
    without update the code

    sorry for my bad english :)

    Comment by ivvel — May 4, 2012 @ 9:53 PM

    • Yes youtube have changed something at there end, therefore this storeurl.pl is caching only 51 seconds of videos.
      I am currently busy in some projects therefore no time for R&D. I will see it later.

      Comment by Syed Jahanzaib / Pinochio~:) — May 8, 2012 @ 3:22 PM

  67. IIs it true your statement is “Some research update (Someone please confirm and post your comments)
    YOUTUBE has split its videos into segments of 1.5 MB each the which is the approximation of the 51 seconds. YOUTUBE I am sure have taken this step to Prevent people from caching entire videos. If you have a video the which is 100 Mb large, it will be split into segments about 55-60.
    As of right now, storeurl.pl wont be Able to cache it. Currently VIDEOCACHE plugin is doing a full cache of youtube but at higher $ $ $ cost ”

    Then how the solution from you? the problem is I can only cache / hit early in about 50 seconds and so miss.
    So if I watch a video on youtube with more than 3 minutes just cache / hit miss 50 seconds onwards.

    I’m using ubuntu proxy with 2 GB memory, 250 GB hard drive, processor Inter Core i3 + mikrotik

    Comment by duckterkru29 — May 11, 2012 @ 12:31 PM

    • No solution for 51 seconds yet !
      YOUTUBE has split its videos into segments of 1.5 Mb each which is the approximation of the 51 seconds. I am sure YOUTUBE have taken this step to prevent people from caching entire videos. If you have a video which is 100 Mb large, it will be split into about 55-60 segments.
      As of right now, storeurl.pl wont be able to cache it. Currently VIDEOCACHE plugin is doing full cache of youtube but at higher $$$ cost :)

      Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2012 @ 8:59 AM

      • Syed Jahanzaib and the others that are facing problem with youtube cache.. just do this 2 things.. set minimum object to 512 and prevent youtube to send range request.. today(14/05/2012) all my youtube files are working news and old one.. one thing that i’ve observed is that spliting the file help us in bandwidth saving, just check access.log after mods and check for HIT, if you’re facing problems, check store.log and see if you hav a swapout in youtube files with content length size = / = size

        Comment by Daniel Echizen — May 14, 2012 @ 11:53 PM

      • just to know.. to prevent youtube range 206 request
        acl 206 regex -i \.youtube.com/.*
        http_access Range deny 206

        Comment by Daniel Echizen — May 14, 2012 @ 11:54 PM

      • its header_access not http_access in the rule

        Comment by Daniel Echizen — May 16, 2012 @ 5:34 PM

      • I’ve set my squid.conf
        at point
        minimum_object_size 512 bytes
        maximum_object_size 100 MB

        and still youtube just cached at 51second :(

        can u explain this @daniel echizen?
        check store.log and see if you hav a swapout in youtube files with content length size = / = size
        please tel me how step by step to do that? :)

        Comment by ivvel — May 18, 2012 @ 4:09 PM

      • using storeurl.pl, its not possible to cache beyond 51 seconds of youtube. try yt-cache, its still working. search google

        Comment by Syed Jahanzaib / Pinochio~:) — May 20, 2012 @ 8:20 PM

      • sorry about the delay to answer .. it’s something about this.. if you file get cached from squid you’ll get a swapout in store.log on the file..try to look for this first. Youtube splits the video in files but you still can get all parts cached, just check store.log and see whats going on when squid try to store the files

        Comment by Daniel Echizen — May 25, 2012 @ 1:21 AM

  68. i just wanted to ask , after how many days does squid remove some old cache data (for example one month data)( or it doesn’t do it after all) cuz i’ve 1TB hdd and after 10 days i got 200GB on cache

    Comment by Nori — May 11, 2012 @ 12:47 PM

  69. thank you sayid, for sharing this kind of knowledge, keep posting…

    Comment by lambok — May 11, 2012 @ 10:06 PM

  70. great, i having this problem too..
    Do you know how to load balance 2 squid server ? right now im using load balancing software haproxy, and virtual ip form ucarp .

    Comment by ibrahim — May 16, 2012 @ 6:31 AM

  71. An error occurred is back again today…. hope for a solution

    Comment by Bash — May 18, 2012 @ 4:18 AM

  72. An error occurred back. so what i must to do

    Comment by vicky — May 23, 2012 @ 9:18 PM

  73. smartvideo addon for FF & GC..solves your problem..^__^

    Comment by cheonne — May 24, 2012 @ 8:42 AM

  74. we w8 for the free fix

    Comment by kyohisro — May 25, 2012 @ 5:57 PM

  75. Hola.
    Aqui una solucion, disculpen si esta mal traducido, use Google Translate.

    To save the full video on youtube.
    within its squid config folder create a file called redirector.pl and write the following:
    #! / usr / bin / perl
    $ | = 1;
    while () {
    chomp;
    @ X = split;
    $ url = $ X [1];
    if ($ url = ~ / (youtube). * videoplayback \?. * /) {
    $ url = ~ s / & range = 13-1781759 / /;
    print $ X [0]. “$ url \ n”;
    Else {}
    print $ X [0]. “\ n”;
    }
    }
    Save it and give execution permission.
    then your squid.conf add the following lines.

    url_rewrite_program / ​​etc / squid / redirector.pl
    url_rewrite_children 1
    url_rewrite_concurrency 1

    and can only restart the squid
    this is a first and simple redirector, I hope the community will improve.

    – Tested in squid 2.7, squid 3, Lusca.
    – Need of storeurl.pl to save the videos.
    – This can be improved for use in combination with http://rg3.github.com/youtube-dl/ to default to open videos in lower quality and thus speed up loading.

    Comment by xspartacus — May 26, 2012 @ 4:01 AM

    • can you share your redirector.pl and squid.config for this script?

      Comment by cheonne — May 26, 2012 @ 1:13 PM

      • redirector.pl in squid directory

        #!/usr/bin/perl
        $|=1;
        while () {
        chomp;
        @X = split;
        $url = $X[1];
        if ($url =~ /(youtube).*videoplayback\?.*/) {
        $url =~ s/&range=13-1781759//;
        print $X[0].” $url\n”;
        } else {
        print $X[0].” \n”;
        }
        }

        Add in squid.conf

        url_rewrite_program /etc/squid/redirector.pl
        url_rewrite_children 1
        url_rewrite_concurrency 1

        Comment by xspartacus — May 29, 2012 @ 12:26 AM

      • it crashing my squid.

        Comment by 9m2pju — May 29, 2012 @ 7:38 AM

  76. can you tell me how make ubunto bridge ???? to make my net work

    isp1+isp2======>mikrotik load balance and hotspot ======> ubunto +squid “bridged” ======>user

    can you explain it ???

    thanks

    Comment by angle net — May 26, 2012 @ 2:40 PM

    • If you want to run squid on bridge, Place it between Internet and Mikrotik.
      Like
      ISP1+ISP2 ======> Ubuntu + squid “bridged” ======> mikrotik load balance and hotspot ======> USERS Network

      don’t place it directly in between users and mikrotik.

      Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:06 PM

      • thanks MR.syed
        but ? how make it ??? can you tell me configuration ubuntu to make it bridge between internet and mikrotik ??

        Comment by angle net — May 29, 2012 @ 4:57 AM

  77. Completely watched videos are being saved into and delivered from cache. If you exit in middle of video it wont be cached.

    Comment by backupsite — May 27, 2012 @ 11:12 AM

    • Still the issue is 51 seconds caching only. Even if the user have watched complete video, squid is serving only 51 seconds of video.
      Looking for prominent solution for this issue. . . .

      Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:04 PM

  78. succes installing squid for caching youtube but got problem it can`t start automatically after computer start must manually start using command squid -DF any solution

    Comment by andreans — May 27, 2012 @ 2:55 PM

    • add it in /etc/rc.local to automatically execute it at every restart.

      Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:02 PM

      • what to add… i already add squid.sh but when boot it says squid.sh not found so i have to manually start squid

        Comment by andreans — May 28, 2012 @ 12:10 PM

  79. succes for caching youtobe but it can not for caching 4shared video, why ?

    Comment by Marga Utama — May 28, 2012 @ 4:23 PM

  80. I followed few posts just above but it won’t cache – or it will mostly be “an error occurred”. so there is no final solution yet I guess !!! … SADDDDD

    Comment by bash — May 29, 2012 @ 11:18 PM

    • same like me

      Comment by andreans — May 30, 2012 @ 1:09 AM

  81. good blog sir ,i have some problem with my proxy,some time cant cache you tube and must be reboot my proxy and can caching again ,so what is my problem sir ,i’m using your squid and store url.i’m waiting your reply soonesty.i’m sory my language is very poor .thank’s be fore

    Comment by aryobahak — June 3, 2012 @ 7:07 PM

  82. Mr Syed,
    when do you expect to have a reliable script? as you know videocache script is working like a charm….

    thank you

    Comment by Maroon — June 4, 2012 @ 5:56 PM

  83. storeurl.pl algorithm modifications that I managed to cache youtube videos 100% up and running very smoothly so far. So I do NOT agree with you …

    Comment by detrackmx — June 5, 2012 @ 12:23 PM

  84. @ detrackmx
    please post your script or PM to my e-mail address itvolution@gmail.com

    man thanks

    Comment by Maroon — June 5, 2012 @ 3:18 PM

  85. ok thank you sir, so how about if i want to make high priority to game online example pointblank,lost saga . do you have some rule on squid for that because some time lost saga get red ping so make lag for that game ,i use 2mb and i share only 10 pc only in my internet cafe and 1 server sir, please give me some explain to my problem sir .

    Comment by aryobahak — June 5, 2012 @ 4:10 PM

  86. Thanks U bro, excellent job… u are the best, my youtube not erorr again… and good work thanks ..thanks… :)

    Comment by noemail — June 8, 2012 @ 8:18 AM

  87. videocache is a reliable one on pfsense. anyone knows how to install it on pfsense? there are guides on pfsense but really i don’t understand how to install it. regards

    Comment by kamote — June 8, 2012 @ 1:45 PM

  88. i was working on a better solution then store_url_rewrite for quite a long time and had the idea but not it’s workable using icap and a database.
    it can also be implemented using a url_rewriter but using icap is faster and can be maintained almost without reload/restarting any server.

    this solution is using the new capabilities of squid3+ branches.

    my github page:

    https://github.com/elico/squid-helpers

    and the post on squid-users mailing list

    http://squid-web-proxy-cache.1019090.n4.nabble.com/Youtube-dynamic-content-caching-with-squid-3-2-DONE-td4655311.html

    Eliezer

    Comment by Eliezer — June 10, 2012 @ 6:41 AM

  89. Dear Sir, i have the problem when i install the lusca_cache pakges and giveing me the reslt

    Beginning package installation for lusca-cache… Downloading package configuration file… done. Saving updated package information… done. Downloading lusca-cache and its dependencies… Checking for package installation…

    Downloading http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz … could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz.
    of squid_radius_auth-1.10 failed!

    Installation aborted.Backing up libraries… Removing package… Starting package deletion for lusca-cache-r14850…done. Starting package deletion for perl-5.10…done. Starting package deletion for squid_radius_auth-1.10…done. Starting package deletion for libwww-5.4.0_4…done. Removing lusca-cache components… Tabs items… done. Menu items… done. Services… done. Loading package instructions… Include file squid.inc could not be found for inclusion. Deinstall commands… Not executing custom deinstall hook because an include is missing. Removing package instructions…done. Auxiliary files… done. Package XML… done. Configuration… done. Cleaning up… Failed to install package.

    Installation halted.
    plz sir can you help me

    Comment by Ahmad — June 13, 2012 @ 1:23 AM

  90. youtube has again updated their site i think, this is not working anymore for me… is there a fix????

    Comment by tulino — June 14, 2012 @ 12:07 PM

  91. Dear Mr. Syed

    Thanks a lot for sharing

    I need some help for one of my projects.

    I have installed squid for a network of 20 users only with a 30mbps fibre optic link. Ive tried squid script shared by you without youtube caching (cause dont need that) and tried debian also with squid3

    Everything works fine but after 5 – 6 hours normally, the RAM gets fully used and Ubuntu / Debian got stuck. In debian i can restart service but in ubuntu the system crashes.

    After restart, everything works perfect for few hours again

    Hardware is P4 3.0, 1GB RAM, 1TB HD.

    Please advise if its because of hardware issue as I dont see any issues while running except this

    Regards

    Comment by ADDICTED — June 22, 2012 @ 5:03 PM

    • upgrade your RAM, 1GB is not enough

      Comment by Wissam — June 26, 2012 @ 1:42 AM

  92. I wanna say Thank’s. Following the guide here now I’m able to install it on Mac OS…
    Thank You for sharing this!

    Comment by nauval2007 — June 24, 2012 @ 1:10 PM

  93. how to squid 3.2 ?? what can caching youtube

    Comment by fahmi — June 27, 2012 @ 4:29 PM

  94. well all youtube format include webm cachable now tks to the guys who post that here is my work im not that exelent but i understand the concept i test it meny time tks again
    } elsif ($X[1] =~ /(youtube|google).*videoplayback\?/){
    @itag = m/[&?](itag=[0-9]*)/;
    @id = m/[&?](id=[^\&]*)/;
    ####@range = m/[&?](range=[^\&\s]*)/;
    $X[1] = ~ s / & range = 13-1781759 / /;
    @begin = m/[&?](begin=[^\&\s]*)/;
    @redirect = m/[&?](redirect_counter=[^\&]*)/;
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&$X[1]@begin@redirect\n”;
    enjoy

    Comment by joe lawand — June 28, 2012 @ 2:15 AM

    • is it working?

      Comment by anonymoues — June 28, 2012 @ 9:41 AM

  95. again sorry for the last post i should do more test befor posting any way if one of the programmer can help doing this to match variable insted of fixed number becaus it change
    $X[1] = ~ s / & range = 13-1781759 / /;
    range=1781760-3563519
    is this will do??? $X[1] = ~ s / & range =[*] / /;

    Comment by joe lawand — June 28, 2012 @ 2:42 PM

  96. Finaly i find very very good article..thanks a lot…its very very help full

    Comment by omay komara — June 29, 2012 @ 7:37 PM

  97. new method, full file not range

    Comment by Syaifuddin JW — July 2, 2012 @ 5:37 PM

  98. i hait ppl making $$$ from wat ever ppl post to help here is the solution almost 90% all format range non range working of corse need more testing and some not all some of the webM that has content from diseny need more testing
    pls post your test and if somone can help making it beter less bug will app….tks for ppls they share and help
    storeurl_rewrite_children 1
    storeurl_rewrite_concurrency 100

    if ($X[1] =~ /(youtube|google).*videoplayback\?/){
    @itag = m/[&?](itag=[0-9]*)/;
    @id = m/[&?](id=[^\&]*)/;
    @range = m/[&?](id=[^\&]*)/;
    @ptk = m/[&?](ptk=[^\&]*)/;
    @oid = m/[&?](oid=[^\&]*)/;
    @ptchn = m/[&?](ptchn=[^\&]*)/;
    @pltype = m/[&?](pltype=[^\&]*)/;
    @begin = m/[&?](begin=[^\&\s]*)/;
    @redirect = m/[&?](redirect_counter=[^\&]*)/;

    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@ptk@oid@ptchn@pltype@begin@redirect\n”;

    acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
    acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \.webm$ \
    —–
    refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.(flv|x-flv|swf|webm)?) 10080 80% 10080 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims store-stale negative-ttl=0

    Comment by joe lawand — July 3, 2012 @ 1:59 AM

  99. replace my prev… post print stuf withthis one untillsomone figur wat is going on with webM diseny content stuff
    the normal webm is working jus replace to this
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@begin@redirect\n”;

    Comment by joe lawand — July 3, 2012 @ 3:13 AM

  100. print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range\n”;

    Comment by Syaifuddin JW — July 3, 2012 @ 3:29 PM

  101. Yeah some ppl do make money off other ppl code. So is there anyone gentle enough to forget about money for one second and provide us the code to make Youtube caching possible. When you throw the food to the sea to feed the fish – if the fish don’t note your good intention, God do count your good deed.
    PEACE,

    Comment by Namo — July 3, 2012 @ 4:16 PM

  102. is there a way to force the player to use 360p only force the client not to use higher then that pls tks

    Comment by joe lawand — July 4, 2012 @ 6:00 PM

  103. Thank’s :)

    Comment by Novasandi — July 15, 2012 @ 5:02 PM

  104. WARNING: store_rewriter #1 (FD 6) exited

    how to fix it…. thank`s for tutorial

    Comment by drw82ca — July 19, 2012 @ 8:11 AM

  105. is there any one who can post a perfect script/way to cache youtube videos if yes then please post it because my squid still caching only 51sec video from youtube. if any one get solution for perfect video caching then plz plz plzzzzz
    post it here becoz it can’t be hide by anyone. knowledge populated by sharing.

    Comment by Rehmat ali — July 31, 2012 @ 6:59 AM

  106. Syed i’m having a issue with ubuntu .
    After a crash or a power issue , i need always to press enter to start ubuntu properly .
    is there a way to do it automatic if i don’t press enter to choose the first solutions on the boot loader

    there are some choises : ubuntu , ubuntu recovery mode

    Comment by N-Ori — August 5, 2012 @ 2:48 PM

    • You need to modify the /etc/grub.d/00_header , Open it and look for following section

      if [ "${recordfail}" = 1 ]; then
      set timeout=-1

      change timeout = -1 to 0

      -1 means manual selection.

      After modifying, save & exit. Also remember to run update-grub command afterward.
      Then test it again.

      Do let me know if it helps.

      Comment by Syed Jahanzaib / Pinochio~:) — August 6, 2012 @ 9:22 AM

  107. Here is some clue for youtube non-range HIT “youtube range url regex create in squid acl, after that deny and it will become a one file/content non-range” and in storeurl.pl you can use the lastest ini this blog.

    Comment by odon — August 7, 2012 @ 6:57 AM

  108. my proxy untill now still good working ,i using your squid lusca and storeurl.pl . so i have one question sir , how to make clear cache automate using squid konfiguration.
    i’m waithing your reply sonesty.thanks

    Comment by aryo — September 1, 2012 @ 1:12 PM

    • Over a period of time, the allocated space for the caching directories starts to fill up. Squid starts deleting cached objects from the cache once the occupied space by the objects crosses a certain threshold, which is determined by using the cache_swap_low and cache_swap_high directives. These directives take integral values between 0 and 100.

      cache_swap_low 96
      cache_swap_high 97

      So, in accordance with these values, when the space occupied for a cache directory crosses 96 percent, Squid will start deleting objects from the cache and will try to maintain the utilization near 96 percent. However, if the incoming rate is high and the space utilization starts to touch the high limit (97 percent), the deletion becomes quite frequent until utilization moves towards the lower limit.

      Squid’s defaults for low and high limits are 90 percent and 95 percent respectively, which are good if the size of cache directory is low (like 10 GB). However, if we have a large amount of space for caching (such as a few hundreds GBs), we can push the limits a bit higher and closer because even 1 percent will mean a difference of more than a gigabyte.

      Comment by Syed Jahanzaib / Pinochio~:) — September 3, 2012 @ 9:05 AM

      • the storeurl doest not work now..mr syed i mean since in the midlle mar – april 2013…open youtube but get wrong video playing….any help please !!

        Comment by rafi — April 7, 2013 @ 8:25 AM

      • if you read the beginning paragraph of this article, you will notice that its not working and the date is also mentioned. So no solution at a moment, try nginx method, it may be working

        Comment by Syed Jahanzaib / Pinochio~:) — April 7, 2013 @ 3:45 PM

  109. is there a way to always force the player to use 240p as a default resolution?
    Thanks

    Comment by Maroon — September 3, 2012 @ 7:19 PM

  110. ok mr thanks for your reply .so how to make scrip in squid if i want to cache this :
    1.tcp_miss/200 588 GET http://5-jw-w.channel.facebook.com/puul?
    2.tcp_miss/200 223 GET http://www.facebook.com/ai.php?

    only this is nothing caching in my squid ,so you can help me to my problem ,thank before

    Comment by aryo — September 3, 2012 @ 11:35 PM

  111. hi, sometimes i donwload a .exe file and fast download but a few minutes later same file download at normal speed, could you give me some clues? thanks in advance

    Comment by joel — September 21, 2012 @ 8:05 AM

  112. i would apreciate this

    Comment by joel — September 21, 2012 @ 8:06 AM

  113. Please I can spend the new link as hya storeurl videos that I then try to pull through Nestor mistake

    Comment by Syed Jahanzaib / Pinochio~:) — September 25, 2012 @ 7:23 AM

  114. thanks for ur kindness, please still share again

    Comment by hidayat — December 23, 2012 @ 7:44 AM

  115. seeking youtube error,lagi sibuk cari copas lum ketemu ya syed :P

    Comment by lengob — January 2, 2013 @ 1:26 AM

    • bukan sibuk nyari copas… udah setahun lebih youtube di blokir dari pakistan mas bro..

      Comment by jokopenthol — March 25, 2013 @ 11:33 PM

      • pasti tdk ada yg tau range full untuk youtube…………. Kalau yang berani kasih range full bukan 0,51 pasti saya jempol dia…..wkwkwkwkwkwkwkwk

        Comment by andhyka — June 4, 2013 @ 11:23 AM

      • kalau ada yang berani kasih range full untuk youtube, pasti saya jempol dia……. wkwkwkwkwkwkwkwk

        Comment by ANDHYKA PRADANA - MAKASSAR — June 4, 2013 @ 11:28 AM

  116. Storeurl.pl berjalan baik dan normal untuk Lusca http://mr-ekoapriadi.blogspot.com/2013/04/info-storeurlpl-berjalan-baik-dan.html

    Comment by Adam — May 5, 2013 @ 10:48 PM

  117. I want to update everybody about StoreID which is the alternative to store_url_rewrite.
    it comes in the new squid 3.4 and now in squid head which can be compiled manually.
    feel free to contact me about it.

    http://wiki.squid-cache.org/Features/StoreID

    Comment by Eliezer — May 6, 2013 @ 2:23 PM

    • Thank You Eliezer :) , But dueto ban on youtube in our country, we cant do any testing on it.

      Comment by Syed Jahanzaib / Pinochio~:) — May 13, 2013 @ 8:42 AM

      • Well I have one nice server to test it if you want.
        we can also use AWS tiny server for that.. use some ssh\ssl\vpn to get into the aws…

        Comment by Eliezer — June 4, 2013 @ 3:36 PM

  118. Thanks , your post really helped me…………. :)

    Comment by Deon — May 26, 2013 @ 4:41 PM

  119. range full youtube Non 0.51 second……

    Comment by BOCCO — June 4, 2013 @ 11:25 AM

  120. Hello Sir Syed, I’m using Pfsense+Lusca and using your storeurl.pl seems to be working very well but can you please post updated refresh pattern as well to match up ur storeurl.pl? thank you.

    Comment by Oliver — October 20, 2013 @ 9:02 PM

  121. ##### crontab untuk menghapus file yg sudah tidak terpakai lebih dari 1 jam yang lalu
    ## crontab perbaikan dari warnet ersa pati ( pak lutfi )
    0 * * * * find /var/log/squid/ -maxdepth 1 ! -name “*.log” -type f -mmin +60 -delete >> /dev/null 2>&1

    ############# Squid Config

    acl youtube url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204) .*$
    acl youtube url_regex -i \.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
    storeurl_access allow youtube

    ############ storeurl.pl ( squid-2.7.Stable9

    } elsif ($X[1] =~ m/^http(|s)\:\/\/.*youtube.*(ptracking|stream_204|player_204|gen_204).*(video_id|docid|v)\=([^\&\s]*).*/){
    $vid = $4 ;
    @cpn = m/[&?]cpn\=([^\&\s]*)/;
    $fn = “/var/log/squid/@cpn”;
    unless (-e $fn) {
    open FH,”>”.$fn ;
    print FH “$vid\n”;
    close FH;
    }
    print $x . $X[1] . “\n”;

    } elsif ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
    @itag = m/[&?](itag=[0-9]*)/;
    @ids = m/[&?]id\=([^\&\s]*)/;
    @mime = m/[&?](mime\=[^\&\s]*)/;
    @cpn = m/[&?]cpn\=([^\&\s]*)/;
    if (defined($cpn[0])) {
    $fn = “/var/log/squid/@cpn”;
    if (-e $fn) {
    open FH,”<".$fn ;
    $id = ;
    chomp $id ;
    close FH ;
    } else {
    $id = $ids[0] ;
    }
    } else {
    $id = $ids[0] ;
    }
    @range = m/[&?](range=[^\&\s]*)/;
    print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range@mime\n”;

    ########## Store-ID.pl ( Squid-3.4 atau Squid-3.HEAD )

    if ($x =~ m/^http(|s)\:\/\/.*youtube.*(ptracking|stream_204|player_204|gen_204).*(video_id|docid|v)\=([^\&\s]*).*/){
    $vid = $4 ;
    @cpn = m/[&?]cpn\=([^\&\s]*)/;
    $fn = “/var/log/squid/@cpn”;
    unless (-e $fn) {
    open FH,”>”.$fn ;
    print FH “$vid\n”;
    close FH;
    }
    $out = $x . “\n”;

    } elsif ($x =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
    @itag = m/[&?](itag=[0-9]*)/;
    @ids = m/[&?]id\=([^\&\s]*)/;
    @mime = m/[&?](mime\=[^\&\s]*)/;
    @cpn = m/[&?]cpn\=([^\&\s]*)/;
    if (defined($cpn[0])) {
    $fn = “/var/log/squid/@cpn”;
    if (-e $fn) {
    open FH,”<".$fn ;
    $id = ;
    chomp $id ;
    close FH ;
    } else {
    $id = $ids[0] ;
    }
    } else {
    $id = $ids[0] ;
    }
    @range = m/[&?](range=[^\&\s]*)/;
    $out = “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range@mime”;

    Comment by Syaifuddin JW — October 24, 2013 @ 10:09 PM

  122. Tried copy paste the Storeurl.pl above on my Pfsense 2.0.1 + Lusca cache + squidguard.

    i deleted my old storeurl and pasted the above code. still experiencing Youtube error.. is my method correct?

    or i’m missing other methods? Thank you!!!

    Comment by Edward Caezar Pababero — January 28, 2014 @ 8:23 AM

    • try to use new squid.conf

      Comment by Syed Jahanzaib / Pinochio~:) — January 28, 2014 @ 3:07 PM

      • Sir do i have to erase all codes written here /usr/local/etc/squid/squid.conf?

        Here’s the code written:

        # Do not edit manually !
        http_port 192.168.1.1:3128 transparent
        http_port 127.0.0.1:80 transparent
        icp_port 0

        pid_filename /var/run/squid.pid
        cache_effective_user proxy
        cache_effective_group proxy
        error_directory /usr/local/etc/squid/errors/English
        icon_directory /usr/local/etc/squid/icons
        visible_hostname localhost
        cache_mgr admin@localhost
        access_log none
        cache_log /var/squid/logs/cache.log
        cache_store_log none
        shutdown_lifetime 0 seconds
        # Allow local network(s) on interface(s)
        acl localnet src 192.168.1.0/255.255.255.0
        uri_whitespace strip

        cache_mem 8 MB
        maximum_object_size_in_memory 4 KB
        memory_replacement_policy heap GDSF
        cache_replacement_policy heap LFUDA
        cache_dir coss /var/squid/coss 50 max-size=4096 block-size=512
        cache_dir aufs /var/squid/cache 420000 16 256 min-size=4096
        minimum_object_size 0 KB
        maximum_object_size 1024 MB
        offline_mode off
        cache_swap_low 98
        cache_swap_high 99
        acl donotcache dstdomain “/var/squid/acl/donotcache.acl”
        cache deny donotcache
        # No redirector configured

        # Setup some default acls
        acl all src 0.0.0.0/0.0.0.0
        acl localhost src 127.0.0.1/255.255.255.255
        acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 1025-65535
        acl sslports port 443 563
        acl manager proto cache_object
        acl purge method PURGE
        acl connect method CONNECT
        acl partialcontent_req req_header Range .*
        #acl dynamic urlpath_regex cgi-bin \?
        include /usr/local/etc/squid/include.conf
        #cache deny dynamic
        http_access allow manager localhost

        http_access deny manager
        http_access allow purge localhost
        http_access deny purge
        http_access deny !safeports
        http_access deny CONNECT !sslports

        # Always allow localhost connections
        http_access allow localhost

        quick_abort_min 32 KB
        quick_abort_max 128 KB
        quick_abort_pct 75
        range_offset_limit 0 MB
        request_body_max_size 0 allow all
        reply_body_max_size 0 deny all

        delay_pools 1
        delay_class 1 2
        delay_parameters 1 -1/-1 30720/30720
        delay_initial_bucket_level 100
        delay_access 1 allow all

        # Custom options
        negative_ttl 0 seconds
        redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
        redirector_bypass off
        url_rewrite_children 5

        # Allow local network(s) on interface(s)
        http_access allow localnet

        # Default block all to be sure
        http_access deny all

        Do i have to erase that and replace it with the new squid.conf? by the way im using Pfsense 2.0.1 +Lusca cache

        Comment by Edward Caezar Pababero — January 28, 2014 @ 4:28 PM

  123. youtube not cache, can you new update stroreurl.pl ?

    Comment by machmud — March 7, 2014 @ 12:14 PM

  124. thanks … great job .. very help full with this…. my You
    Tube cache running well

    Comment by banaspati — March 10, 2014 @ 9:50 AM

  125. not working on speedtest.net pls help.. thnx

    Comment by jv — March 25, 2014 @ 12:35 PM

    • you need to add custom rule for speedtest.net, something like these just for example, you may need to edit to your need

      -> in squid.conf
      #speedtest
      refresh_pattern -i speedtest.net|speedtest.san.fastserv.com/mini/speedtest/latency.txt|speedtest.san.fastserv.com/mini/speedtest/random350x350.jpg|speedtest.san.fastserv.com/mini/speedtest/upload.php 10080 90% 999999 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth

      #speedtest
      acl speedtest url_regex -i speedtest.net|speedtest.*.com/mini/speedtest/random350x350.jpg

      #this may cause slow when testing upload
      #|speedtest.san.fastserv.com/mini/speedtest/upload.php|speedtest.san.fastserv.com/mini/speedtest/latency.txt

      #acl speedtest url_regex -i speedtest.san.fastserv.com|speedtest.net
      cache allow speedtest

      storeurl_access allow speedtest

      -> in storeurl.pl
      #speedtest
      }elsif (m/^http:\/\/(.*)\/speedtest\.net\/(.*\.(jpg|txt))\?(.*)/) {
      $out=”http://www.speedtest.net.SQUIDINTERNAL/speedtest/” . $2 . “”;

      }elsif (m/^http:\/\/(.*)\/speedtest\.san\.fastserv\.com\/(.*\.(jpg|txt|php))\?(.*)/) {
      $out=”http://www.speedtest.san.fastserv.com.SQUIDINTERNAL/mini/speedtest/” . $2 . “”;

      Comment by nauval2007 — March 27, 2014 @ 2:40 PM

  126. Hi, Goodmorning,

    My pfsense + Lusca cache is not caching youtube anymore, since march 2014. is this applicable for lusca cache?

    Thanks…

    Comment by Edward Caezar Pababero — April 1, 2014 @ 6:28 AM

  127. no funciona mas los storeurl.pl antes andaba muy bien pero ahora ya no anda y gracias por los aporte men

    Comment by jose — April 23, 2014 @ 8:26 PM

  128. Espero master Alastaz y el genio Alaskoid Ayuda en efectivo

    Me Mstab Apinto 10.4.4

    Y trabajado todos los pasos anteriores explicación y Lai viabilidad red de malla temprana

    Posible utilizar

    Si es posible, las órdenes y todos los archivos necesarios 03/01/19

    Waukon Shaker Avdhalkm
    Mizo hermano de Egipto

    Comment by mezoosaidosaid — May 5, 2014 @ 9:43 PM


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 2,361 other followers

%d bloggers like this: