The last updated guide 22nd April, 2014 about youtube caching is as follows …
=========================================================================================================
Last updated > 20th January, 2014
If you are caching YouTube using storeurl.pl method
EXAMPLE:
https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/
and you encounter following error while watching any cached video “An error occured, Please try again later”
As showed in the image below . . .
Then try using the following January, 2014 updated storeurl.pl
#!/usr/bin/perl # This script is NOT written or modified by me, I only copy pasted it from the internet. # It was First originally Written by chudy_fernandez@yahoo.com # & Have been modified by various persons over the net to fix/add various functions. # Like For Example modified by member of comstuff.net to satisfy common and dynamic content. # th30nly @comstuff.net a.k.a invisible_theater , and possibly other people too. # For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube # Syed Jahanzaib / aacable@hotmail.com # https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/ ##################### #### REFERENCES ##### http://www2.fh-lausitz.de/launic/comp/misc/squid/projekt_youtube/ ##################### ##################### ## storeurl.pl-130411-0.05 ## s. 130411.howto.squid.youtube.html ## CHANGES ## 130411 hl+mf ## - fix youtube loop redir ## - rel. 0.05 ## 130409 hl ## - add usleep ## 120726 hl ## - add printtimenow ## 120725 hl ## - remove html-tags ## - add debug ## - disable unused rules ## - rel. 0.04 ## 120130 aacable ## - fix match variable.domain.com ## https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-later-solved/ ## 120111 aacable ## https://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/ #### notes ## - config (s. 130411.howto.squid.youtube.html) ## squid.conf:storeurl_rewrite_program /etc/squid/storeurl.pl ## - test ## $ cat squid.diag.youtube.url | awk '{print "0",$7,"10.0.0.1/- - GET - myip=10.0.0.2 myport=3128"}' | ./storeurl.pl # # ################################################################################################ ## UPDATED on 20 January, 2014 **** thanks to Mr. Safatah Purwonoto [Indonesia] / Syed Jahanzaib ################################################################################################ #### var use IO::File; $|=1; STDOUT->autoflush(1); $debug=1; ## recommended:0 $bypassallrules=0; ## recommended:0 $sucks=""; ## unused $sucks="sucks" if ($debug>=1); $timenow=""; $printtimenow=1; ## print timenow: 0|1 my $logfile = '/tmp/storeurl.log'; open my $logfh, '>>', $logfile or die "Couldn't open $logfile for appending: $!\n" if $debug; $logfh->autoflush(1) if $debug; #### main while (<>) { $timenow=time()." " if ($printtimenow); print $logfh "$timenow"."in : $_" if ($debug>=1); chop; ## strip eol @X = split; $x = $X[0]; ## 0 $u = $X[1]; ## url $_ = $u; ## url if ($bypassallrules){ $out="$u"; ## map 1:1 #youtube with range (YOUTUBE has split its videos into segments) }elsif (m/(youtube|google).*videoplayback\?.*range/ ){ @itag = m/[&?](itag=[0-9]*)/; @id = m/[&?](id=[^\&]*)/; @range = m/[&?](range=[^\&\s]*)/; @begin = m/[&?](begin=[^\&\s]*)/; @redirect = m/[&?](redirect_counter=[^\&]*)/; $out="http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range&@redirect"; #sleep(1); ## delay loop #youtube without range }elsif (m/(youtube|google).*videoplayback\?/ ){ @itag = m/[&?](itag=[0-9]*)/; @id = m/[&?](id=[^\&]*)/; @redirect = m/[&?](redirect_counter=[^\&]*)/; $out="http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@redirect"; #sleep(1); ## delay loop #speedtest }elsif (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) { $out="http://www.speedtest.net.SQUIDINTERNAL/speedtest/" . $2 . ""; #mediafire }elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) { $out="http://www.mediafire.com.SQUIDINTERNAL/" . $1 ."/" . $2 . ""; #fileserve }elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) { $out="http://www.fileserve.com.SQUIDINTERNAL/" . $1 . "./" . $2 . ""; #filesonic }elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) { $out="http://www.filesonic.com.SQUIDINTERNAL/" . $1 . ""; #4shared }elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) { $out="http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3"; #4shared preview }elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) { $out="http://www.4shared.com.SQUIDINTERNAL/$2"; #photos-X.ak.fbcdn.net where X a-z }elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) { $out="http://photos" . $1 . "/" . $2 . $3 . ""; #YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z } elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) { $out="http://photos.ak.fbcdn.net/" . $1 ."/". $2 . ""; #maps.google.com } elsif (m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) { $out="http://" . $1 . $2 . ""; # compatibility for old cached get_video?video_id } elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) { $z = $2; $z =~ s/video_id=/get_video?video_id=/; $out="http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . ""; #sleep(1); ## delay loop } elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) { $out="http://www.google-analytics.com/__utm.gif"; #Cache High Latency Ads } elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) { $y = $3;$z = $2; for ($y) { s/pixel;.*/pixel/; s/activity;.*/activity/; s/(imgad[^&]*).*/\1/; s/;ord=[?0-9]*//; s/;×tamp=[0-9]*//; s/[&?]correlator=[0-9]*//; s/&cookie=[^&]*//; s/&ga_hid=[^&]*//; s/&ga_vid=[^&]*//; s/&ga_sid=[^&]*//; # s/&prev_slotnames=[^&]*// # s/&u_his=[^&]*//; s/&dt=[^&]*//; s/&dtd=[^&]*//; s/&lmt=[^&]*//; s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/; s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/; s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/; s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/; s/[;&?]ord=[?0-9]*//; s/[;&]mpvid=[^&;]*//; s/&xpc=[^&]*//; # yieldmanager s/\?clickTag=[^&]*//; s/&u=[^&]*//; s/&slotname=[^&]*//; s/&page_slots=[^&]*//; } $out="http://" . $1 . $2 . $y . ""; #cache high latency ads } elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) { $out="http://" . $1 . "/" . $2 . ""; # spicific servers starts here.... } elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) { $out="http://" . $1 . ""; #cdn, varialble 1st path } elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) { @y = ($1,$2,$4,$5); $y[0] =~ s/[a-z0-9]{2,5}/cdn./; $out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . ""; #rapidshare } elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) { $out="http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . ""; } elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) { $out="http://" . $1 . "/SQUIDINTERNAL/" . $3 . ""; #like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end } elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) { $out="http://cdn." . $4 . $6 . ""; #...spicific servers end here. #photos-X.ak.fbcdn.net where X a-z } elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) { $out="http://photos.ak.fbcdn.net/" . $1 . ""; #for yimg.com video } elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) { $out="http://cdn.yimg.com//" . $3 . ""; #for yimg.com doubled } elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) { $out="http://cdn.yimg.com/" . $3 . ""; #for yimg.com with &sig= } elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) { @y = ($1,$2); $y[0] =~ s/[a-z]+[0-9]+/cdn/; $y[1] =~ s/&sig=.*//; $out="http://" . $y[0] . ".yimg.com/" . $y[1] . ""; #youjizz. We use only domain and filename } elsif (($u =~ /media[0-9]{2,5}\.youjizz/) && (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})((\?|\%).*)?$/)) { @y = ($1,$2,$4,$5); $y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/; $out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . ""; #general purpose for cdn servers. add above your specific servers. } elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) { $out="http://squid-cdn-url//" . $2 . "." . $3 . ""; #generic http://variable.domain.com/path/filename."ex" "ext" or "exte" with or withour "? or %" } elsif (m/^http:\/\/(.*)(\.[^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{2,4})((\?|\%).*)?$/) { @y = ($1,$2,$3,$4); $y[0] =~ s/(([a-zA-Z]+[0-9]+(-[a-zA-Z])?$)|(.*cdn.*)|(.*cache.*))/cdn/; $out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . ""; } else { $out="$u"; ##$X[2]="$sucks"; } print $logfh "$timenow"."out: $x $out $X[2] $X[3] $X[4] $X[5] $X[6] $X[7]\n" if ($debug>=1); print "$x $out $X[2] $X[3] $X[4] $X[5] $X[6] $X[7]\n"; } close $logfh if ($debug);
.
Please post your comments regarding updated storeurl.pl
Regard’s
SYED JAHANZAIB
squid is not running, after i update. why ?
thanks.
LikeLike
Comment by neo — January 31, 2012 @ 8:10 AM
any solutions on pfsense+lusca, i have the same problem regarding with the youtube error.. i tried reinstalling everthing from scratch but still the same — youtube error
LikeLike
Comment by Incognito — January 31, 2012 @ 7:30 PM
So far I have not found any 100% solution for this issue. However after adding negative_ttl 0 seconds , situation gets much better almot 80-90% problem solved. Try it
I also heard from some folks that php method works 100% without error. you can try that, google for it.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — January 31, 2012 @ 7:43 PM
Thanks for your post Syed. Where do i put or which file do i edit to add the “negative_ttl 0 seconds?.. when i tried opening the file squid.conf, i didn’t see any.. any guides? thanks again.. 🙂
LikeLike
Comment by Incognito — February 3, 2012 @ 6:33 AM
add negative_ttl 0 seconds any where , preferably before Refresh Patterns.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 3, 2012 @ 6:52 PM
Thanks Syed for the quick feedback.. You’re very helpful. thank you….
LikeLike
Comment by Incognito — February 3, 2012 @ 9:22 PM
Hi Syed. I have the same issues and adding that line in squid, situation gets better but sometimes the error returns. Would you like to contact me so we can find he solution together?
LikeLike
Comment by Mario — February 3, 2012 @ 8:06 PM
This issue occurs when youtube loads video partially. I looked on youtube url video part by part, and saw &range= at the end. So I tried to change regex on storeurl.pl and seems working until now. And all parts of a video are cached and ‘hit’ normally.
sorry for my bad english
LikeLike
Comment by lif2k3 — February 4, 2012 @ 4:59 PM
That would be wonderful, But try it for 2-3 days on testing and then do let me know,
Post the updated storeurl.pl , maybe I will be able to fine tune it too 🙂
Thx for the contribution
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 4, 2012 @ 5:58 PM
this picture may explain how youtube loads video partialy :
and this image shows my access log while logging a youtube video part by part.
as you can see, several parts of a video are succesfully load from cache (HIT), then rest part of it is still ‘miss’. However, a video could be loaded part by part or normally. So, a video could take double size on cache swap (partially video and normal video).
Again, sorry for my bad english 😦
LikeLike
Comment by lif2k3 — February 5, 2012 @ 2:03 PM
Can you confirm if the problem have been resolved ?
Can you post the updated storeurl.pl ??
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 5, 2012 @ 3:45 PM
No problem appear since 2 days ago.. and running well. No “an error occured’ again.
hmm.. do u have IM contact? YM, Gtalk, or skype?
We can talk about this later, if u mind.
LikeLike
Comment by lif2k3 — February 5, 2012 @ 5:59 PM
The problem is I rarely use MSN.
My email is aacable [at] hotmail.com
You can email me for any discussion.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 5, 2012 @ 9:07 PM
Hi, is there a new way to fix youtube error?.. after i add negative_ttl 0 seconds to squid.conf, problem still occurs as of now. Can you guide me with the new way to fix the problem? Thanks in advance.
LikeLike
Comment by Incognito — February 8, 2012 @ 8:51 AM
negative_ttl only better the situation, but not 100%
There is no absolute fix for this problem at a moment, At least I am not aware of it 🙂
If you find one, do let me know.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 8, 2012 @ 8:57 AM
maybe you can use my storeurl. i put my store url in pastebin and here the link
http://pastebin.com/e3TUtigH
http://www.forummikrotik.com/showthread.php/21089-New-Storeurl-Untuk-Youtube-file-range
http://clearos-indonesia.com/frm/?topic=1482.0
LikeLike
Comment by syaifuddinjw — February 8, 2012 @ 4:02 PM
What’s the result of this storeurl.pl
Is there any improvement in it ?
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 9, 2012 @ 9:00 AM
now, video youtube with file range, so if you use your old storeurl that error acure come to you. sorry if my english not good. take a look at my capture log
LikeLike
Comment by syaifuddinjw — February 10, 2012 @ 12:00 AM
Hmmm,, can you please post updated storeurl.pl , or email me. It would be great help for our open source community. 🙂
Ok Can you provide me with your email address for communication.
My email is aacable at hotmail.com
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2012 @ 9:02 AM
Okay, Tnx Syed, i’ll post some info regarding this problem if I could find one.. this one is good at the moment – at least..
Thanks for the big help. 🙂
LikeLike
Comment by Incognito — February 8, 2012 @ 4:05 PM
with the squid.conf file you provided, can you also cache software and large files? any type of file?
Thank you
LikeLike
Comment by Try — February 9, 2012 @ 10:41 PM
You can define file size in squid.conf
Everything which is cache-able goes in cache.
Or you can use some advance scripting like storeurl.pl to force cache for some non cacheable objects, for example you-tube videos, antivirus updates, windows updates etc. Yes its possible.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 10, 2012 @ 8:59 AM
hai syed, if you ask about new storeurl, so you can take a look at my pastebin
http://pastebin.com/e3TUtigH
LikeLike
Comment by syaifuddinjw — February 10, 2012 @ 1:16 PM
Yes its working fine so far.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 11, 2012 @ 11:45 AM
Heh, a little bug with new storeurl.pl script, or maybe squid… If you fastforward song somewhere after reload it starts from that place… i hope that you understand me… 😉
LikeLike
Comment by nenad — February 11, 2012 @ 1:04 AM
Well I have configured the new updated storeurl.pl at two different cable.network setup, and so far , its working great. much mroe better then its previous version, so I can confirm that its working good.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 11, 2012 @ 11:43 AM
Did you test with Lusca or squid or both ?
LikeLike
Comment by Damir — February 16, 2012 @ 8:01 PM
Yes I have tested it with LUSCA and SQUID 2.7
Both works fine.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 17, 2012 @ 8:52 AM
maybe you can try to reconfigure your squid config
quick_abort_min
quick_abort_max
and read the manual
LikeLike
Comment by syaifuddinjw — February 11, 2012 @ 7:23 PM
very big thanks to syed there is not more error on youtube …. works very fine with me
LikeLike
Comment by hamid — February 15, 2012 @ 4:06 AM
Great. Glad it helped you 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 15, 2012 @ 9:08 AM
sou do brasil, por favor me ajudem a instalar o squid no Debian
obrigado 😉
LikeLike
Comment by Diogo Pereira — February 15, 2012 @ 6:49 PM
English Please !
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 15, 2012 @ 7:59 PM
I’m from Brazil, please help me install squid on Debian
thank you 😉
LikeLike
Comment by Diogo Pereira — February 16, 2012 @ 2:36 AM
Syed
i Would like to know your opinion to tell me what hardware can i use for approximatelly
200 active users every one having a bantwith 384kbps-512kbps
with a network of 25Mbps and a bantwith of 1.5TB within a month of traffic used .
I want to build a squid without caching .flv files and files larger than 150Mb .
What do u thing , what hardware computer do u prefer for me like DELL Servers , IBM and which type of that if u like any of them .
By the way i would like to buy any without needing much energy (if any exist with at least 500W)
LikeLike
Comment by nori — February 17, 2012 @ 8:47 PM
It doesn’t matter what brand you go for.
Any Dual base processor machine with enough RAM (min of 4 GB or Plus) can easily accommodate your 200+ users load. For example
Xeon 2.4G hz OR 3.0 GHz Dual / 4GB RAM / 2 HDD on proxy, one for OS and Logs and one reserved for caching , like 500GB x 2 HDD
It all depends on your network scenario and Budget you have. Mikrotik doesn’t requires much heavier hardware for this amount of users, Also Mikrotik supports only 2 GB of ram maximum.
but for proxy, the better hardware you put on it, better performance you will get from it.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 18, 2012 @ 1:00 PM
Im runing mikrotik on intel atom (1,6Ghz) with 2 GB RAM , with proxy , lot of firewall rules … and cpu is only 5% with 20 users 🙂
LikeLike
Comment by Damir — February 18, 2012 @ 1:20 PM
thx men 🙂 i’ve more than 8 mikrotiks all RB433 or Rb435 and i want to build a squid proxy server for my clients
with just one lan card with 200users 😀
now i’ll buy a xeon 3.0 Dual with 8GB Ram and 1TB HDD 🙂
what i want to know if possible is :
Why are some servers with 2 Power Supply ?
Do i need to connect both of them in power or can i connect just one and if the Server has 750W
Does it mean that the server will take that power or if i have jut one HDD and just 8GB ram it will take just less than 750W , because sometimes we have a power cut into my town and it can stay for 3 hours without power :S
thx anyway for everything 😀
LikeLike
Comment by Nori — February 18, 2012 @ 4:26 PM
Some server have 2 power supplies, because they provide fault tolerance, redundancy , means if one power source fails, the second will keep working , like UPS working.
(Provided power is coming on second when first failure occurs)
Usually heavy servers requires more power.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 19, 2012 @ 7:39 PM
thanks bro, for this perl scripts
LikeLike
Comment by http://mandailingnatal.com — February 19, 2012 @ 11:20 PM
You are welcome.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 20, 2012 @ 9:31 AM
hahaha…. tonggo ….
aku baca situse orang luar negeri, tapi ternyata resolvnya dari dalam negeri.
Bravo, Congratulation My Brother
LikeLike
Comment by riez — February 20, 2012 @ 7:46 AM
English please 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 20, 2012 @ 9:25 AM
hehehehe… i’m sorry my english language is still poor
LikeLike
Comment by riez — February 21, 2012 @ 9:09 PM
welah dalah, yok opo cak, iki mung bagi bagi sing tak copas kok cak
LikeLike
Comment by Syaifuddin JW — February 22, 2012 @ 6:03 AM
Thx syed :=) wish u all the best 🙂 thx for sharing knowledge
LikeLike
Comment by Nori — February 22, 2012 @ 2:19 AM
Thank you 🙂
but from now on, I won’t be able to participate much due to my new job. Now I am highly involved with MICRSOFT / IBM world 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 22, 2012 @ 8:54 AM
Gracias.
LikeLike
Comment by Spartacus — March 13, 2012 @ 7:48 PM
I modified the lines below, but now storeurl.pl Squid increased hits, but now the storeurl.pl is consuming 100% cpu.
$url = ;
while ($url) {
while () {
LikeLike
Comment by Moises Souza — March 22, 2012 @ 7:28 PM
youtube error occured again..it started yesterday…T_T
LikeLike
Comment by kjshoot2ill — March 23, 2012 @ 2:08 PM
True dont know why, probably google change something… If someone find new storeurl please post here… Thanks
LikeLike
Comment by nenad — March 25, 2012 @ 4:42 AM
[…] https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-la… […]
LikeLike
Pingback by Howto Cache Youtube with SQUID / LUSCA and bypass Cached Videos from Mikrotik Queue « Syed Jahanzaib Personnel Blog to Share Knowledge ! — March 28, 2012 @ 10:38 AM
Updated New STOREURL.PL
update and comment.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 28, 2012 @ 10:53 AM
Testing testing… and tell me please, ignore-must-revalidate, lusca dont give me error on that.
thanks
LikeLike
Comment by nenad — March 28, 2012 @ 1:05 PM
Remove all instances of
ignore-must-revalidate
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 29, 2012 @ 7:20 PM
Ok I have posted new storeurl.pl today (27th March)
So far result is Fine. Comment its updates . . .
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 28, 2012 @ 3:43 PM
Wow… Amazing…. I’d tried & all problem fix…. Thank’s so much. and I’ll try it for more days…!!! Hope it work well… 🙂
LikeLike
Comment by Reds Ryan (Rilyandi) (@Golden_Cikampak) — March 28, 2012 @ 6:25 PM
Thanks Bro, it is working now, or… for now 🙂 i have one suggestion, maybe we should support WebM format for youtube…
Thanks again
LikeLike
Comment by nenad — March 28, 2012 @ 9:21 PM
any help, when i paste/edit from any source storeurl.pl end restarting squid, my squid wont work, but if paste again with originial storeurl.pl its work again, i’m edit storeurl.pl with notepad++ and save as perl, and try edit with perl editor too, and try from this too just copy paste not edit http://pastebin.com/e3TUtigH, error again, please help, me still newbie
LikeLike
Comment by tono — March 29, 2012 @ 9:38 AM
There must be some copy pasting error , either due to blogs coding, or your notepad saving changing codes.
use putty to login to your squid box, then from the browser , directly copy paste the content in storeurl.pl in putty
Start SQUID in debug mode , with following parameters, it will show exactly for what reasons SQUID not restarting.
squid -d1n
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 29, 2012 @ 7:18 PM
make a new storeurl.pl and execute chmod 777 storeurl.pl.
It’s all.
LikeLike
Comment by newbie — March 29, 2012 @ 10:26 PM
thanks for posting because it really helped me.
LikeLike
Comment by Abenk Ones Skd — March 30, 2012 @ 12:15 PM
I m glad it helped u 😀
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 31, 2012 @ 3:32 PM
Greetings, I apologize for that necessarily have ubuntu with 2 network cards? I ask because I use mikrotik and thundercache used previously only used a network card … thanks in advance
LikeLike
Comment by Alfredo — March 31, 2012 @ 10:48 AM
Most of my posts related to Linux are Ubuntu base, because I found Ubuntu most easy to use Linux flavor. You or other may found any other flavor more suitable for various reasons, As I have my own 🙂
For example: By default FEDORA installs SQUID 3 which doesn’t support ZPH by default, you have to compile Squid 2.7 on Fedora in order to use ZPH or Use LUSCA on FED , as LUSCA supports ZPH well.
On the other hand UBUNTU installs squid 2.7 which supports ZPH so no extra work is required.
It’s all about personnel preference, in past, FEDORA was my first choice, but from past 2 years UBUNTU have manage to become First 🙂 but don’t worry, You can achieve your goal by using any flavor of Linux, doesn’t matter its Ubuntu , Fedora, Centos , FreeBSD , or any OS with Linux as its base OS / Kernel , it will work 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 31, 2012 @ 3:30 PM
thenks for the post, i try he squid 2.7 and storeurl.pl “update 27 March, 2012”, the youtube video are not served from the cache.
There is not “An error occured, Please try again later”
regards
LikeLike
Comment by laziz — April 1, 2012 @ 1:22 AM
check your squid.conf
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 1, 2012 @ 12:38 PM
hi
i use the same squid.conf as “Youtube caching with SQUID 2.7 [using storeurl.pl] ” related at the link : https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/
regards
LikeLike
Comment by laziz — April 2, 2012 @ 1:05 AM
ok I understand but you need to use 2 network cards to use the squid?
LikeLike
Comment by Alfredo — April 2, 2012 @ 3:08 AM
Not necessary that you must have 2 interfaces in SQUID. But its better to have 2 for various reasons.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 2, 2012 @ 8:38 AM
[…] 2.7 [using storeurl.pl]Mikrotik DUAL WAN Load Balancing using PCC method. Complete Script ! by zaiBYoutube Caching Problem : An error occured. Please try again later. [SOLVED] updated storeurl.plMikrotik 4 WAN Load Balancing using PCC method. Complete Script ! by […]
LikeLike
Pingback by Youtube caching with SQUID 2.7 [using storeurl.pl] « Syed Jahanzaib Personnel Blog to Share Knowledge ! — April 2, 2012 @ 9:04 AM
thanks. it’s work no problem
please share squid.conf lusca/squid
patching games facebook example empires and allies dll
mode on
LikeLike
Comment by ahmad — April 2, 2012 @ 9:09 AM
TOP Banget Kakak
LikeLike
Comment by gembeel — April 2, 2012 @ 12:36 PM
hi syed is this storeurl support to manage video that error when i was cache it, exmaple when i cache some video wtih 360 res, it finish then i changed to 720 res but youtube does’nt reload from begining video and cache the video even just the end of video. Is there any advice to fix my problem? cause this make much trouble for me!
LikeLike
Comment by Zahrian — April 2, 2012 @ 11:38 PM
oh and more question if i used this to cache youtube
# youtube fix
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = “”;
$alg = “”;
$id = “”;
$range = “”;
if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
is my old video on proxy still can be used because i used this storeurl from chudy or i must to re cache all youtube videos
if ($X[1] =~ /(youtube|google).*videoplayback\?/){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@range = m/(&range=[^\&\s]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@id&@itag@range\n”;
LikeLike
Comment by Zahrian — April 2, 2012 @ 11:43 PM
I am not sure about it, I guess it will pick old cache videos fine. Test it
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 3, 2012 @ 8:37 AM
No fix for this yet.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 3, 2012 @ 8:37 AM
thanks for all and let me know by email if you found how to fix it!!
LikeLike
Comment by Zahrian — April 3, 2012 @ 11:46 AM
oh syed one more question what the different by your youtube storeurl and chudy storeurl is there any bug addition on your youtube storeurl
LikeLike
Comment by Zahrian — April 3, 2012 @ 2:20 PM
This storeurl.pl is not made/modified by me. This script was first designed by Chuddy and have been modified by many persons around the internet.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 3, 2012 @ 3:11 PM
Hi… thanks for sharing storeurl.pl….
I have try to integrated the update storeurl.pl into my squid….. but i didn’t see any youtube get cache 😦
LikeLike
Comment by adjie — April 7, 2012 @ 5:25 PM
it’s working better thanks man ,but still gives errors
btw is it possible to cache facebook games while it is chosen by the user to use secure browsing (https)?
Did you find a way to cache those stuff? thank you
LikeLike
Comment by antonio — April 11, 2012 @ 6:45 AM
It’s better not to cache HTTPS for various security reasons.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 11, 2012 @ 8:44 AM
Basit Khan.
basit at ambnet.net
Sir I have tried it on pfsense and works fine for few days after that it crashed badly….I have search alot of things on internet and found that VARNISH Cache is 10x faster than squid, I will highly appricate if you can provide its tutorial or how to..etc..
Thanks
Allah bless you!
Basit Khan
LikeLike
Comment by BASIT KHAN — April 11, 2012 @ 4:51 PM
WARNING – Queue congestion Help with this Warning
LikeLike
Comment by Elfoman — April 14, 2012 @ 5:59 AM
Working with multi-threaded disk access (AIO) Squid queues the tasks to be performed and lets the disk controller work through it as fast as it can. This allows Squid to work on other processing tasks for the same request without being held up waiting for slow disks.
The queue starts off at a default length of 8 queue slots per thread. When that queue space is filled up, Squid will spit out the WARNING, and double the available queue length.
Up to a few of these is OK under very high load. But if you get them very frequently then it’s a sign that either the disk I/O is overloaded or you have run out of CPU cycles to handle it.
Workaround
* A few seconds of these after a clean startup can be ignored. They should decrease exponentially as the queue is automatically adjusted to the load.
* For Squid expected to run on a busy network, increasing the default AIO threads available can reduce the annoyance. Using fast disks is essential.
* If these continue without decreasing you need faster disks, or to spread the traffic load over more proxies.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 14, 2012 @ 7:47 PM
Thanks for your post Syed, I have copast, so far so good. thanks again
LikeLike
Comment by magelar — April 17, 2012 @ 1:57 AM
Great, Update us with your findings when your testing is done.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 17, 2012 @ 8:47 AM
Hi I’m testing the code perl and my question is whether this caching me well squid 2.7 now when I see the video gives me this message again TCP_REFRESH_HIT/304 202 GET http://www.youtube.com/crossdomain.xml – DIRECT/173.194 .42.4 –
or is this tomendo me of another proxy that this network before mine thanks
LikeLike
Comment by Nestor — April 17, 2012 @ 7:21 AM
Syed why sometimes youtube still showed some error, is more configuration needed to fix it?
LikeLike
Comment by Zahrian — April 18, 2012 @ 3:10 PM
oh the errors showed up when files had partial video and full video!! and error happen when squid load partial video? any advice for this problem!!
LikeLike
Comment by Zahrian — April 18, 2012 @ 3:14 PM
It will happen in few videos, but the ratio is very less as compared to its prior versions.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 18, 2012 @ 3:22 PM
consult this file to run perl or now youtube cache can only cache videos with YouTube API?
LikeLike
Comment by Nestor — April 18, 2012 @ 3:23 PM
One further question, when you change the resolution of a video, you get the video cached in the new resolution from the point where you switched. And when you want to watch the same video it starts from the point where it was cached earlier. So, you can’t watch it entirely.
Any ideas about that, or how to make that with a resolution change, the Squid always make the caching from the video starting point?
LikeLike
Comment by Damian — April 19, 2012 @ 5:27 PM
Hola a todos gracias por todo los que comparten información aquí y gracias sobre todo a Pinochio les comento que a mi hasta ahora me funciona bastante bien el storeurl.pl que descargue de aquí
Hello everyone thanks for all those who share information here, thanks mainly to Pinochio tell them that my so far it works pretty well I downloaded from here storeurl.pl
http://www.google.com.ar/url?sa=t&rct=j&q=storeurl.pl&source=web&cd=1&ved=0CCYQFjAA&url=http%3A%2F%2Ffaisal-sani-project.googlecode.com%2Ffiles%2Fstoreurl.pl&ei=1auQT5_ZN8OCtgfuzYH6BA&usg=AFQjCNFDMUWrQsZmDsUoEQRQo2Nu6Ecxkw
LikeLike
Comment by Nestor Piro — April 20, 2012 @ 6:15 AM
dude, is it really copyright of
# https://aacable.wordpress.com / Syed Jahanzaib
# Email: aacable@hotmail.com
is it u write all of those code ??
dude, im the writer of this line
#speedtest
if (m/^http:\/\/(.*)\/speedtest\/(.*\.(jpg|txt))\?(.*)/) {
print $x . “http://www.speedtest.net.SQUIDINTERNAL/speedtest/” . $2 . “\n”;
#mediafire
}elsif (m/^http:\/\/199\.91\.15\d\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . “http://www.mediafire.com.SQUIDINTERNAL/” . $1 .”/” . $2 . “\n”;
}elsif (m/^http:\/\/205\.196\.12[0-3]\.\d*\/\w{12}\/(\w*)\/(.*)/) {
print $x . “http://www.mediafire.com.SQUIDINTERNAL/” . $1 .”/” . $2 . “\n”;
#fileserve
}elsif (m/^http:\/\/fs\w*\.fileserve\.com\/file\/(\w*)\/[\w-]*\.\/(.*)/) {
print $x . “http://www.fileserve.com.SQUIDINTERNAL/” . $1 . “./” . $2 . “\n”;
#filesonic
}elsif (m/^http:\/\/s[0-9]*\.filesonic\.com\/download\/([0-9]*)\/(.*)/) {
print $x . “http://www.filesonic.com.SQUIDINTERNAL/” . $1 . “\n”;
#4shared
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/download\/(.*)\/(.*\..*)\?.*/) {
print $x . “http://www.4shared.com.SQUIDINTERNAL/download/$2\/$3\n”;
#4shared preview
}elsif (m/^http:\/\/[a-zA-Z]{2}\d*\.4shared\.com(:8080|)\/img\/(\d*)\/\w*\/dlink__2Fdownload_2F(\w*)_3Ftsid_3D[\w-]*\/preview\.mp3\?sId=\w*/) {
print $x . “http://www.4shared.com.SQUIDINTERNAL/$2\n”;
#photos-X.ak.fbcdn.net where X a-z
}elsif (m/^http:\/\/photos-[a-z](\.ak\.fbcdn\.net)(\/.*\/)(.*\.jpg)/) {
print $x . “http://photos” . $1 . “/” . $2 . $3 . “\n”;
#YX.sphotos.ak.fbcdn.net where X 1-9, Y a-z
} elsif (m/^http:\/\/[a-z][0-9]\.sphotos\.ak\.fbcdn\.net\/(.*)\/(.*)/) {
print $x . “http://photos.ak.fbcdn.net/” . $1 .”/”. $2 . “\n”;
}
AND ALSO THIS LINE
# youtube fix by th30nly @comstuff.net
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = “”;
$alg = “”;
$id = “”;
$range = “”;
$uri = “http://$1/videoplayback\?$2”;
if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $uri =~ s/$range//;$range =~ s/-//; $range =~ s/range=//; }
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
}
i’ just curious how SELFISH you are.
just see http://lusca.info for prove and timestamping.
regards
th30nly a.k.a invisible_theater
LikeLike
Comment by th30nly — April 21, 2012 @ 12:11 AM
Dear th30nly,
Please accept my appologies, I never said that this or that script is written by ME. i added the chuddy_fernandez email address on top, I added my line only for tracking purposes, it is not copy righted by me.
Anyhow thank you for pointing the right thing, I have added following note in the post, Please check it and do let me know if there is any further amendments required.
=========================================================
This script is NOT written by me, I only copy pasted it from the internet.
It was originally Written by chudy_fernandez@yahoo.com & Have been modified by various persons over the net to fix/add various functions.
For more info, http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
=========================================================
As you said, you have added few functions to it, please do let me know your email or name, So I will add them to your section lines.
Once again, storeurl.pl is not my creation, It was just copy pasted and I did mentioned the original author name at the top. However I have now added more info on it. I hope the picture is clear now.
Regard’s
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 21, 2012 @ 10:38 AM
you did not mentioned that you write the script, but you may not modify by omiting the header lines (remaks) which is mentioned th30nly had fixed or modify the original script.
the name of th30nly was omitted from the whole script remarks and then you put your name was mentioned in the header under the original writer, I doubt it for tracking purposes since the remaks lines which mentioned fixed by th30nly still not revised yet
check these lines below:
#!/usr/bin/perl
# writen by chudy_fernandez@yahoo.com
# https://aacable.wordpress.com /Syed Jahanzaib
# Email: aacable@hotmail.com
.
.
.
# youtube fix
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = “”;
$alg = “”;
$id = “”;
$range = “”;
if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
.
.
these are in our storeulr.pl which is copy paste by you.
#!/usr/bin/perl
# store rewrite originally writen by chudy_fernandez@yahoo.com
# modified by member of comstuff.net to satisfy common and dynamic content.
#
.
.
.
# youtube fix by th30nly @comstuff.net
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = “”;
$alg = “”;
$id = “”;
$range = “”;
if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . “\n”;
.
.
.
if you need the storeurl.pl script writen by th30nly, you may get it free directly from th30nly @comstuff,net
regards,
felix yt @comstuff.net
ps. thank you for your visit at comstuff.net
LikeLike
Comment by felix yt — April 21, 2012 @ 5:25 PM
I have added the comstuff.net details in the script. Please Check.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 21, 2012 @ 7:34 PM
Thanks for this storeurl.pl
LikeLike
Comment by setting-mikrotik.com — April 23, 2012 @ 1:40 AM
thxx for all this tutorials that u have provided 2 us .
i just want to ask u something as a network it
1. i’ve dedicated internet access 30Mbps Down/ 30Mbps up but on youtube i’m having problems on caching youtube with your script
could it be because my servers provider has added cache servers into his internet access
on squid i see something like that :
0 80.80.171.19 TCP_HIT/200 657 GET http://o-o.preferred.ipko-prn1.v21.lscache3.c.youtube.com/crossdomain.xml – NONE/- text/x-cross-domain-policy
what could be the reasen that squid isn’t caching youtube ?
LikeLike
Comment by N-or — April 25, 2012 @ 4:04 AM
I have tested Youtube caching this morning and its working fine at my end using this storeurl.pl
youtube have also introduced new .webm format which is not cached by default. test various videos
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 25, 2012 @ 11:01 AM
i’m not saying that it doesnt work , the squid works great everywhere else just in youtube i see a link like that
TCP_HIT/200 657 GET http://o-o.preferred.ipko-prn1.v21.lscache3.c.youtube.com/crossdomain.xml – NONE/- text/x-cross-domain-policy
the link isn’t from youtube it’s from my service provided and i see just crossdomain.xml
could it be because my provider has a cache server and i can’t get it from his cache server :S
LikeLike
Comment by N-or — April 26, 2012 @ 1:29 AM
post the output of this command
ps aux |grep squid
It must display something like this
root@zaib_linux:/scripts# ps aux |grep squid
proxy 9106 1.2 1.0 12096 5412 ? Ssl 16:07 0:00 /usr/sbin/squid -N -D
proxy 9107 0.2 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9108 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9109 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9110 0.2 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9111 0.0 0.3 3684 1624 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9112 0.2 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
proxy 9113 0.0 0.3 3684 1628 ? Ss 16:07 0:00 /usr/bin/perl /etc/squid/storeurl.pl
root 9133 0.0 0.1 3324 816 pts/0 S+ 16:07 0:00 grep –color=auto squid
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 26, 2012 @ 4:10 PM
thank mr. syed…i am from indoesia…i am glad and happpy…my caching youtube it`s work because of shared you storeurl…sory my english bad but thank so much…….
LikeLike
Comment by beni jauhari — April 26, 2012 @ 1:42 AM
You are welcome.
I am not the author/writer of the storeurl.pl . I only share it on my blog for easy finding 🙂
Thanks to the real authors of storeurl.pl
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 26, 2012 @ 3:49 PM
English please 🙂 .
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 26, 2012 @ 3:56 PM
waduh ada mastah2 dari forum m*******k …aampun mastah…apa emang dia dapat kopi paste dari orang lain….
LikeLike
Comment by beni jauhari — April 26, 2012 @ 2:35 AM
by me its caching just a few videos most of videos i cant cache i see just this crossdomain.xml /text – cross domain policy
does anybody else have this problem or not 🙂
thx by the way Syed for every tutorial and information that u have added in this website 🙂
i
LikeLike
Comment by N-or — April 26, 2012 @ 1:06 PM
dont forget to chmod +x storeurl.pl and chown to the user and group of the squid. mine is chown proxy:proxy storeurl.pl.
LikeLike
Comment by 9m2pju — April 26, 2012 @ 1:59 PM
[…] To fix youtube loading error on squid lusca, we need to modify the storeurl.pl file. Here is the fix by Syed Jahanzaib, https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-l… […]
LikeLike
Pingback by Ubuntu developers: Mohamad Faizul Zulkifli: Squid Lusca Youtube Error Fix | Linux-Support.com — April 26, 2012 @ 2:45 PM
Thanks Syed Jahanzaib for your help ,
I have a problem with youtube now only cache 1.69 MB about 1 Min from any youtube video first it was work perfect now i don’t know what happen
LikeLike
Comment by SquidY — April 28, 2012 @ 3:18 AM
I haven’t tested squid youtube caching since few weeks. I will test it again and will let you know the results .
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 28, 2012 @ 4:15 PM
thx Syed 🙂 it’s almost perfect ur blog 🙂
i’ve tryed the command and i see all that 🙂
ur video that u watched i’ve seen others video i get this crossdomain.xml as tcp hit not the link of the video 🙂
but i’m letting it so , better to dont cache youtube 🙂
thx for everything 🙂 what i coulnd find in this blog is how to configure the mrtg graphs in ubuntu 🙂
could u add a tutorial please to add some graphs for squid
LikeLike
Comment by n-orix@live.it — April 28, 2012 @ 12:30 PM
I haven’t tested squid youtube caching since few weeks. I will test it again and will let you know the results.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 28, 2012 @ 4:14 PM
Hola a todos comento como tengo mi squid uso centos 5 y el storeurl de aqui
http://pastebin.com/e3TUtigH
para no tener el error de videos en squid.conf puse
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 5000
fue la forma de estabilizar los videos y cachea todo
saludos a todos
Hi all I have commented as squid use my centos 5 and storeurl from here
http://pastebin.com/e3TUtigH
to avoid the mistake of videos put squid.conf
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 7
storeurl_rewrite_concurrency 5000
was how to stabilize the videos and caches all
Greetings to all
LikeLike
Comment by Nestor — April 28, 2012 @ 5:48 PM
Thanks for the Tip !
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 29, 2012 @ 10:15 AM
what’s the way to restrict the server just for lan users ( of public ip’s )
for example 82.82.162.17-80.80.171.30
what do i need to change to acl amigos
LikeLike
Comment by Bledi — April 28, 2012 @ 9:20 PM
Hi Sayed , i test new storeurl.pl and the same problem not solved i can’t caching youtube , because youtube separate video file with extention FLV to 1.69 MB so that youtube doesn’t cache but it cache with new extention WEBM from youtube .
LikeLike
Comment by SquidY — May 1, 2012 @ 3:54 PM
I tested this storeurl.pl yesterday and I was pleased with he results. It was able to cache all videos including 1 – 20 minutes Length. However I haven’t checked webm format.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 1, 2012 @ 4:20 PM
i tested this storeurl.pl, and it’s work…..thx
LikeLike
Comment by nailancool — May 2, 2012 @ 7:28 AM
just cache until 50 second only,….
LikeLike
Comment by detrackmx — May 3, 2012 @ 1:17 PM
One further question, when you change the resolution of a video, you get the video cached in the new resolution from the point where you switched. And when you want to watch the same video it starts from the point where it was cached earlier. So, you can’t watch it entirely.
Any ideas about that, or how to make that with a resolution change, the Squid always make the caching from the video starting point?
LikeLike
Comment by Damian — May 3, 2012 @ 7:01 PM
No solution for this yet in this storeurl.pl
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 8, 2012 @ 3:22 PM
ni orang tukang COPAS….sebagaian karya ane juga diembat…..parah
LikeLike
Comment by copas — May 4, 2012 @ 9:20 PM
kenapa mas kalo copas??
kok gini ya orang Indonesia??
malu2in indonesia aja lu
orang ini jg membagikan pengetahuannya..
diatas jg dia menulis
(Note: I am not author of this script, Please use the links/emails provided here to contact the real authors, I only tested it and the problem seemed to be better up as compare to the previous versions of this script)
LikeLike
Comment by ivvel — May 5, 2012 @ 11:15 PM
hola, yo uso esta y no tengo problemas,
if (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com)\/videoplayback\?(.*)/) {
$p_str = $2;
$tag = “”;
$alg = “”;
$id = “”;
$range = “”;
$begin = “”;
$redirect = “”;
if ($p_str =~ m/(itag=[0-9]*)/){$tag = “&”.$1}
if ($p_str =~ m/(algorithm=[a-z]*\-[a-z]*)/){$alg = “&”.$1}
if ($p_str =~ m/(id=[a-zA-Z0-9]*)/){$id = “&”.$1}
if ($p_str =~ m/(range=[0-9\-]*)/){$range = “&”.$1; $range =~ s/-//; $range =~ s/range=//; }
if ($p_str =~ m/(begin=[^\&\s]*)/){$begin = “&”.$1}
if ($p_str =~ m/(redirect_counter=[^\&]*)/){$redirect = “&”.$1}
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/” . $tag . “&” . $alg . “&” . $id . “&” . $range . $begin . $redirect . “\n”;
mantengo la variable $tag para mantener la compatibilidad con los videos que ya tenia cacheados, pero la pueden eliminar si quieren. para evitar el problema de “An error occured, Please try again later“ parchen su lusca o squid con el loop 3xx o sino definan su minimum_object_size mayor o igual a 1k.
espero les sirva.
LikeLike
Comment by xspartacus — May 4, 2012 @ 9:30 PM
are the different this storelurl.pl and your old storeurl.pl
just this code??
if ($X[1] =~ /(youtube|google).*videoplayback\?/){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@range = m/[&?](range=[^\&\s]*)/;
@begin = m/[&?](begin=[^\&\s]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@begin@redirect\n”;
why i can’t applied your new update storeurl.pl to my machine??
the result of my squid is like this
2012/05/04 23:23:05| Accepting HTCP messages on port 4827, FD 23.
2012/05/04 23:23:05| Accepting SNMP messages on port 3401, FD 24.
2012/05/04 23:23:05| WCCP Disabled.
2012/05/04 23:23:05| Ready to serve requests.
2012/05/04 23:23:05| WARNING: store_rewriter #1 (FD 12) exited
2012/05/04 23:23:05| Too few store_rewriter processes are running
FATAL: The store_rewriter helpers are crashing too rapidly, need help!
I’ve been try other way
like insert updated code to my storeurl.pl
but the result is same
😦
need help here..
but i dont know why
my machine just cache youtube only 51 second
without update the code
sorry for my bad english
🙂
LikeLike
Comment by ivvel — May 4, 2012 @ 9:53 PM
Yes youtube have changed something at there end, therefore this storeurl.pl is caching only 51 seconds of videos.
I am currently busy in some projects therefore no time for R&D. I will see it later.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 8, 2012 @ 3:22 PM
IIs it true your statement is “Some research update (Someone please confirm and post your comments)
YOUTUBE has split its videos into segments of 1.5 MB each the which is the approximation of the 51 seconds. YOUTUBE I am sure have taken this step to Prevent people from caching entire videos. If you have a video the which is 100 Mb large, it will be split into segments about 55-60.
As of right now, storeurl.pl wont be Able to cache it. Currently VIDEOCACHE plugin is doing a full cache of youtube but at higher $ $ $ cost ”
Then how the solution from you? the problem is I can only cache / hit early in about 50 seconds and so miss.
So if I watch a video on youtube with more than 3 minutes just cache / hit miss 50 seconds onwards.
I’m using ubuntu proxy with 2 GB memory, 250 GB hard drive, processor Inter Core i3 + mikrotik
LikeLike
Comment by duckterkru29 — May 11, 2012 @ 12:31 PM
No solution for 51 seconds yet !
YOUTUBE has split its videos into segments of 1.5 Mb each which is the approximation of the 51 seconds. I am sure YOUTUBE have taken this step to prevent people from caching entire videos. If you have a video which is 100 Mb large, it will be split into about 55-60 segments.
As of right now, storeurl.pl wont be able to cache it. Currently VIDEOCACHE plugin is doing full cache of youtube but at higher $$$ cost 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2012 @ 8:59 AM
Syed Jahanzaib and the others that are facing problem with youtube cache.. just do this 2 things.. set minimum object to 512 and prevent youtube to send range request.. today(14/05/2012) all my youtube files are working news and old one.. one thing that i’ve observed is that spliting the file help us in bandwidth saving, just check access.log after mods and check for HIT, if you’re facing problems, check store.log and see if you hav a swapout in youtube files with content length size = / = size
LikeLike
Comment by Daniel Echizen — May 14, 2012 @ 11:53 PM
just to know.. to prevent youtube range 206 request
acl 206 regex -i \.youtube.com/.*
http_access Range deny 206
LikeLike
Comment by Daniel Echizen — May 14, 2012 @ 11:54 PM
its header_access not http_access in the rule
LikeLike
Comment by Daniel Echizen — May 16, 2012 @ 5:34 PM
I’ve set my squid.conf
at point
minimum_object_size 512 bytes
maximum_object_size 100 MB
and still youtube just cached at 51second
😦
can u explain this @daniel echizen?
check store.log and see if you hav a swapout in youtube files with content length size = / = size
please tel me how step by step to do that?
🙂
LikeLike
Comment by ivvel — May 18, 2012 @ 4:09 PM
using storeurl.pl, its not possible to cache beyond 51 seconds of youtube. try yt-cache, its still working. search google
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 20, 2012 @ 8:20 PM
sorry about the delay to answer .. it’s something about this.. if you file get cached from squid you’ll get a swapout in store.log on the file..try to look for this first. Youtube splits the video in files but you still can get all parts cached, just check store.log and see whats going on when squid try to store the files
LikeLike
Comment by Daniel Echizen — May 25, 2012 @ 1:21 AM
i just wanted to ask , after how many days does squid remove some old cache data (for example one month data)( or it doesn’t do it after all) cuz i’ve 1TB hdd and after 10 days i got 200GB on cache
LikeLike
Comment by Nori — May 11, 2012 @ 12:47 PM
It depends on the rotation setting and refresh pattern.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 14, 2012 @ 9:01 AM
thank you sayid, for sharing this kind of knowledge, keep posting…
LikeLike
Comment by lambok — May 11, 2012 @ 10:06 PM
great, i having this problem too..
Do you know how to load balance 2 squid server ? right now im using load balancing software haproxy, and virtual ip form ucarp .
LikeLike
Comment by ibrahim — May 16, 2012 @ 6:31 AM
You can use squid builtin SIBLING option to load balance squid request.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 22, 2012 @ 3:16 PM
An error occurred is back again today…. hope for a solution
LikeLike
Comment by Bash — May 18, 2012 @ 4:18 AM
An error occurred back. so what i must to do
LikeLike
Comment by vicky — May 23, 2012 @ 9:18 PM
smartvideo addon for FF & GC..solves your problem..^__^
LikeLike
Comment by cheonne — May 24, 2012 @ 8:42 AM
It is for user desktop PC only.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 25, 2012 @ 8:51 AM
http://www.pinoydsl.net/threads/51-seconds-youtube-caching-fix.27632/
LikeLike
Comment by kyohisro — May 25, 2012 @ 4:41 AM
to: kyohisro
Unfortunately this fix is not FREE :s
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 25, 2012 @ 8:45 AM
we w8 for the free fix
LikeLike
Comment by kyohisro — May 25, 2012 @ 5:57 PM
http://www.ulop.net/forum/8-networkingweb/167169-pfsense-users?limit=10&start=2120
LikeLike
Comment by kyohisro — May 25, 2012 @ 6:29 PM
Hola.
Aqui una solucion, disculpen si esta mal traducido, use Google Translate.
To save the full video on youtube.
within its squid config folder create a file called redirector.pl and write the following:
#! / usr / bin / perl
$ | = 1;
while () {
chomp;
@ X = split;
$ url = $ X [1];
if ($ url = ~ / (youtube). * videoplayback \?. * /) {
$ url = ~ s / & range = 13-1781759 / /;
print $ X [0]. “$ url \ n”;
Else {}
print $ X [0]. “\ n”;
}
}
Save it and give execution permission.
then your squid.conf add the following lines.
url_rewrite_program / etc / squid / redirector.pl
url_rewrite_children 1
url_rewrite_concurrency 1
and can only restart the squid
this is a first and simple redirector, I hope the community will improve.
– Tested in squid 2.7, squid 3, Lusca.
– Need of storeurl.pl to save the videos.
– This can be improved for use in combination with http://rg3.github.com/youtube-dl/ to default to open videos in lower quality and thus speed up loading.
LikeLike
Comment by xspartacus — May 26, 2012 @ 4:01 AM
can you share your redirector.pl and squid.config for this script?
LikeLike
Comment by cheonne — May 26, 2012 @ 1:13 PM
redirector.pl in squid directory
#!/usr/bin/perl
$|=1;
while () {
chomp;
@X = split;
$url = $X[1];
if ($url =~ /(youtube).*videoplayback\?.*/) {
$url =~ s/&range=13-1781759//;
print $X[0].” $url\n”;
} else {
print $X[0].” \n”;
}
}
Add in squid.conf
url_rewrite_program /etc/squid/redirector.pl
url_rewrite_children 1
url_rewrite_concurrency 1
LikeLike
Comment by xspartacus — May 29, 2012 @ 12:26 AM
it crashing my squid.
LikeLike
Comment by 9m2pju — May 29, 2012 @ 7:38 AM
http://www.ulop.net/forum/8-networkingweb/167169-pfsense-users?limit=10&start=2180#533283
LikeLike
Comment by kyohisro — May 26, 2012 @ 7:17 AM
can you tell me how make ubunto bridge ???? to make my net work
isp1+isp2======>mikrotik load balance and hotspot ======> ubunto +squid “bridged” ======>user
can you explain it ???
thanks
LikeLike
Comment by angle net — May 26, 2012 @ 2:40 PM
If you want to run squid on bridge, Place it between Internet and Mikrotik.
Like
ISP1+ISP2 ======> Ubuntu + squid “bridged” ======> mikrotik load balance and hotspot ======> USERS Network
don’t place it directly in between users and mikrotik.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:06 PM
thanks MR.syed
but ? how make it ??? can you tell me configuration ubuntu to make it bridge between internet and mikrotik ??
LikeLike
Comment by angle net — May 29, 2012 @ 4:57 AM
Completely watched videos are being saved into and delivered from cache. If you exit in middle of video it wont be cached.
LikeLike
Comment by backupsite — May 27, 2012 @ 11:12 AM
Still the issue is 51 seconds caching only. Even if the user have watched complete video, squid is serving only 51 seconds of video.
Looking for prominent solution for this issue. . . .
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:04 PM
succes installing squid for caching youtube but got problem it can`t start automatically after computer start must manually start using command squid -DF any solution
LikeLike
Comment by andreans — May 27, 2012 @ 2:55 PM
add it in /etc/rc.local to automatically execute it at every restart.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 28, 2012 @ 12:02 PM
what to add… i already add squid.sh but when boot it says squid.sh not found so i have to manually start squid
LikeLike
Comment by andreans — May 28, 2012 @ 12:10 PM
succes for caching youtobe but it can not for caching 4shared video, why ?
LikeLike
Comment by Marga Utama — May 28, 2012 @ 4:23 PM
I followed few posts just above but it won’t cache – or it will mostly be “an error occurred”. so there is no final solution yet I guess !!! … SADDDDD
LikeLike
Comment by bash — May 29, 2012 @ 11:18 PM
same like me
LikeLike
Comment by andreans — May 30, 2012 @ 1:09 AM
good blog sir ,i have some problem with my proxy,some time cant cache you tube and must be reboot my proxy and can caching again ,so what is my problem sir ,i’m using your squid and store url.i’m waiting your reply soonesty.i’m sory my language is very poor .thank’s be fore
LikeLike
Comment by aryobahak — June 3, 2012 @ 7:07 PM
There are some issues with storeurl.pl for youtube. Wait for some reliable method to caching youtube.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — June 4, 2012 @ 10:07 AM
Mr Syed,
when do you expect to have a reliable script? as you know videocache script is working like a charm….
thank you
LikeLike
Comment by Maroon — June 4, 2012 @ 5:56 PM
storeurl.pl algorithm modifications that I managed to cache youtube videos 100% up and running very smoothly so far. So I do NOT agree with you …
LikeLike
Comment by detrackmx — June 5, 2012 @ 12:23 PM
@ detrackmx
please post your script or PM to my e-mail address itvolution@gmail.com
man thanks
LikeLike
Comment by Maroon — June 5, 2012 @ 3:18 PM
ok thank you sir, so how about if i want to make high priority to game online example pointblank,lost saga . do you have some rule on squid for that because some time lost saga get red ping so make lag for that game ,i use 2mb and i share only 10 pc only in my internet cafe and 1 server sir, please give me some explain to my problem sir .
LikeLike
Comment by aryobahak — June 5, 2012 @ 4:10 PM
Thanks U bro, excellent job… u are the best, my youtube not erorr again… and good work thanks ..thanks… 🙂
LikeLike
Comment by noemail — June 8, 2012 @ 8:18 AM
videocache is a reliable one on pfsense. anyone knows how to install it on pfsense? there are guides on pfsense but really i don’t understand how to install it. regards
LikeLike
Comment by kamote — June 8, 2012 @ 1:45 PM
i was working on a better solution then store_url_rewrite for quite a long time and had the idea but not it’s workable using icap and a database.
it can also be implemented using a url_rewriter but using icap is faster and can be maintained almost without reload/restarting any server.
this solution is using the new capabilities of squid3+ branches.
my github page:
https://github.com/elico/squid-helpers
and the post on squid-users mailing list
http://squid-web-proxy-cache.1019090.n4.nabble.com/Youtube-dynamic-content-caching-with-squid-3-2-DONE-td4655311.html
Eliezer
LikeLike
Comment by Eliezer — June 10, 2012 @ 6:41 AM
1. Goto Diagnostics > Command Prompt > Execute
2. enter this: pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/squid_radius_auth-1.10.tbz
note: mabilis lang po ito basta may internet connection.
3. enter this: pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/libwww-5.4.0_4.tbz
4. enter this: fetch http://pfsense-cacheboy.googlecode.com/svn/trunk/script/package.sh && chmod +x package.sh && ./package.sh
5. Now you may install lusca-cache at System > Packages.
Remember Keep open source alive!
LikeLike
Comment by tags — July 20, 2012 @ 7:13 PM
Dear Sir, i have the problem when i install the lusca_cache pakges and giveing me the reslt
Beginning package installation for lusca-cache… Downloading package configuration file… done. Saving updated package information… done. Downloading lusca-cache and its dependencies… Checking for package installation…
Downloading http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz … could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz.
of squid_radius_auth-1.10 failed!
Installation aborted.Backing up libraries… Removing package… Starting package deletion for lusca-cache-r14850…done. Starting package deletion for perl-5.10…done. Starting package deletion for squid_radius_auth-1.10…done. Starting package deletion for libwww-5.4.0_4…done. Removing lusca-cache components… Tabs items… done. Menu items… done. Services… done. Loading package instructions… Include file squid.inc could not be found for inclusion. Deinstall commands… Not executing custom deinstall hook because an include is missing. Removing package instructions…done. Auxiliary files… done. Package XML… done. Configuration… done. Cleaning up… Failed to install package.
Installation halted.
plz sir can you help me
LikeLike
Comment by Ahmad — June 13, 2012 @ 1:23 AM
For installation halt
1. Goto Diagnostics > Command Prompt > Execute
2. enter this: pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/squid_radius_auth-1.10.tbz
note: mabilis lang po ito basta may internet connection.
3. enter this: pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/libwww-5.4.0_4.tbz
4. enter this: fetch http://pfsense-cacheboy.googlecode.com/svn/trunk/script/package.sh && chmod +x package.sh && ./package.sh
5. Now you may install lusca-cache at System > Packages.
Remember Keep open source alive!
LikeLike
Comment by tags — July 20, 2012 @ 7:14 PM
youtube has again updated their site i think, this is not working anymore for me… is there a fix????
LikeLike
Comment by tulino — June 14, 2012 @ 12:07 PM
they did not updated it.
they have 2-3 ways or more to get the data:
http://ytdom/some_data&id=…&itag…&range…
http://ytdom/some_data&itag…&range…&id=…
etc…
and also
http://ytdom/some_data&itag…&id=…&begin…
or without a begin or range at all.
http://ytdom/some_data&itag…&id=…
http://ytdom/some_data&id=…&itag…
so you must match each of these separate cases with a one match case of id and sub cases of the order in the file.
you can have a look at this specific code i have written in ruby for that:
https://github.com/elico/squid-helpers/blob/master/testers/test3.rb
works great with an explicit acl to allow store_url_rewrite only for youtube cache domains,
LikeLike
Comment by Eliezer — June 14, 2012 @ 8:55 PM
Dear Mr. Syed
Thanks a lot for sharing
I need some help for one of my projects.
I have installed squid for a network of 20 users only with a 30mbps fibre optic link. Ive tried squid script shared by you without youtube caching (cause dont need that) and tried debian also with squid3
Everything works fine but after 5 – 6 hours normally, the RAM gets fully used and Ubuntu / Debian got stuck. In debian i can restart service but in ubuntu the system crashes.
After restart, everything works perfect for few hours again
Hardware is P4 3.0, 1GB RAM, 1TB HD.
Please advise if its because of hardware issue as I dont see any issues while running except this
Regards
LikeLike
Comment by ADDICTED — June 22, 2012 @ 5:03 PM
upgrade your RAM, 1GB is not enough
LikeLike
Comment by Wissam — June 26, 2012 @ 1:42 AM
[…] https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-la… […]
LikeLike
Pingback by Squid 2.7 for Caching Dynamic Content on Mac OS - Staidbands™ — June 24, 2012 @ 1:06 PM
I wanna say Thank’s. Following the guide here now I’m able to install it on Mac OS…
Thank You for sharing this!
LikeLike
Comment by nauval2007 — June 24, 2012 @ 1:10 PM
how to squid 3.2 ?? what can caching youtube
LikeLike
Comment by fahmi — June 27, 2012 @ 4:29 PM
try my solution at this comment:
LikeLike
Comment by Eliezer — June 28, 2012 @ 2:40 PM
well all youtube format include webm cachable now tks to the guys who post that here is my work im not that exelent but i understand the concept i test it meny time tks again
} elsif ($X[1] =~ /(youtube|google).*videoplayback\?/){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
####@range = m/[&?](range=[^\&\s]*)/;
$X[1] = ~ s / & range = 13-1781759 / /;
@begin = m/[&?](begin=[^\&\s]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&$X[1]@begin@redirect\n”;
enjoy
LikeLike
Comment by joe lawand — June 28, 2012 @ 2:15 AM
is it working?
LikeLike
Comment by anonymoues — June 28, 2012 @ 9:41 AM
again sorry for the last post i should do more test befor posting any way if one of the programmer can help doing this to match variable insted of fixed number becaus it change
$X[1] = ~ s / & range = 13-1781759 / /;
range=1781760-3563519
is this will do??? $X[1] = ~ s / & range =[*] / /;
LikeLike
Comment by joe lawand — June 28, 2012 @ 2:42 PM
Finaly i find very very good article..thanks a lot…its very very help full
LikeLike
Comment by omay komara — June 29, 2012 @ 7:37 PM
new method, full file not range
LikeLike
Comment by Syaifuddin JW — July 2, 2012 @ 5:37 PM
if you interest, please contac me via ym ucok_karnad
LikeLike
Comment by Syaifuddin JW — July 2, 2012 @ 5:39 PM
i hait ppl making $$$ from wat ever ppl post to help here is the solution almost 90% all format range non range working of corse need more testing and some not all some of the webM that has content from diseny need more testing
pls post your test and if somone can help making it beter less bug will app….tks for ppls they share and help
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 100
—
if ($X[1] =~ /(youtube|google).*videoplayback\?/){
@itag = m/[&?](itag=[0-9]*)/;
@id = m/[&?](id=[^\&]*)/;
@range = m/[&?](id=[^\&]*)/;
@ptk = m/[&?](ptk=[^\&]*)/;
@oid = m/[&?](oid=[^\&]*)/;
@ptchn = m/[&?](ptchn=[^\&]*)/;
@pltype = m/[&?](pltype=[^\&]*)/;
@begin = m/[&?](begin=[^\&\s]*)/;
@redirect = m/[&?](redirect_counter=[^\&]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@ptk@oid@ptchn@pltype@begin@redirect\n”;
acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \.webm$ \
—–
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.(flv|x-flv|swf|webm)?) 10080 80% 10080 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims store-stale negative-ttl=0
LikeLike
Comment by joe lawand — July 3, 2012 @ 1:59 AM
replace my prev… post print stuf withthis one untillsomone figur wat is going on with webM diseny content stuff
the normal webm is working jus replace to this
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range@begin@redirect\n”;
LikeLike
Comment by joe lawand — July 3, 2012 @ 3:13 AM
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/@itag&@id&@range\n”;
LikeLike
Comment by Syaifuddin JW — July 3, 2012 @ 3:29 PM
Yeah some ppl do make money off other ppl code. So is there anyone gentle enough to forget about money for one second and provide us the code to make Youtube caching possible. When you throw the food to the sea to feed the fish – if the fish don’t note your good intention, God do count your good deed.
PEACE,
LikeLike
Comment by Namo — July 3, 2012 @ 4:16 PM
is there a way to force the player to use 360p only force the client not to use higher then that pls tks
LikeLike
Comment by joe lawand — July 4, 2012 @ 6:00 PM
Thank’s 🙂
LikeLike
Comment by Novasandi — July 15, 2012 @ 5:02 PM
WARNING: store_rewriter #1 (FD 6) exited
how to fix it…. thank`s for tutorial
LikeLike
Comment by drw82ca — July 19, 2012 @ 8:11 AM
is there any one who can post a perfect script/way to cache youtube videos if yes then please post it because my squid still caching only 51sec video from youtube. if any one get solution for perfect video caching then plz plz plzzzzz
post it here becoz it can’t be hide by anyone. knowledge populated by sharing.
LikeLike
Comment by Rehmat ali — July 31, 2012 @ 6:59 AM
Try with LUSCA cache. I read somewhere its working with pfsense/lusca.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 3, 2012 @ 7:52 AM
Syed i’m having a issue with ubuntu .
After a crash or a power issue , i need always to press enter to start ubuntu properly .
is there a way to do it automatic if i don’t press enter to choose the first solutions on the boot loader
there are some choises : ubuntu , ubuntu recovery mode
LikeLike
Comment by N-Ori — August 5, 2012 @ 2:48 PM
You need to modify the /etc/grub.d/00_header , Open it and look for following section
if [ “${recordfail}” = 1 ]; then
set timeout=-1
change timeout = -1 to 0
-1 means manual selection.
After modifying, save & exit. Also remember to run update-grub command afterward.
Then test it again.
Do let me know if it helps.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 6, 2012 @ 9:22 AM
Here is some clue for youtube non-range HIT “youtube range url regex create in squid acl, after that deny and it will become a one file/content non-range” and in storeurl.pl you can use the lastest ini this blog.
LikeLike
Comment by odon — August 7, 2012 @ 6:57 AM
my proxy untill now still good working ,i using your squid lusca and storeurl.pl . so i have one question sir , how to make clear cache automate using squid konfiguration.
i’m waithing your reply sonesty.thanks
LikeLike
Comment by aryo — September 1, 2012 @ 1:12 PM
Over a period of time, the allocated space for the caching directories starts to fill up. Squid starts deleting cached objects from the cache once the occupied space by the objects crosses a certain threshold, which is determined by using the cache_swap_low and cache_swap_high directives. These directives take integral values between 0 and 100.
cache_swap_low 96
cache_swap_high 97
So, in accordance with these values, when the space occupied for a cache directory crosses 96 percent, Squid will start deleting objects from the cache and will try to maintain the utilization near 96 percent. However, if the incoming rate is high and the space utilization starts to touch the high limit (97 percent), the deletion becomes quite frequent until utilization moves towards the lower limit.
Squid’s defaults for low and high limits are 90 percent and 95 percent respectively, which are good if the size of cache directory is low (like 10 GB). However, if we have a large amount of space for caching (such as a few hundreds GBs), we can push the limits a bit higher and closer because even 1 percent will mean a difference of more than a gigabyte.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — September 3, 2012 @ 9:05 AM
the storeurl doest not work now..mr syed i mean since in the midlle mar – april 2013…open youtube but get wrong video playing….any help please !!
LikeLike
Comment by rafi — April 7, 2013 @ 8:25 AM
if you read the beginning paragraph of this article, you will notice that its not working and the date is also mentioned. So no solution at a moment, try nginx method, it may be working
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 7, 2013 @ 3:45 PM
is there a way to always force the player to use 240p as a default resolution?
Thanks
LikeLike
Comment by Maroon — September 3, 2012 @ 7:19 PM
ok mr thanks for your reply .so how to make scrip in squid if i want to cache this :
1.tcp_miss/200 588 GET http://5-jw-w.channel.facebook.com/puul?
2.tcp_miss/200 223 GET http://www.facebook.com/ai.php?
only this is nothing caching in my squid ,so you can help me to my problem ,thank before
LikeLike
Comment by aryo — September 3, 2012 @ 11:35 PM
hi, sometimes i donwload a .exe file and fast download but a few minutes later same file download at normal speed, could you give me some clues? thanks in advance
LikeLike
Comment by joel — September 21, 2012 @ 8:05 AM
i would apreciate this
LikeLike
Comment by joel — September 21, 2012 @ 8:06 AM
Please I can spend the new link as hya storeurl videos that I then try to pull through Nestor mistake
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — September 25, 2012 @ 7:23 AM
thanks for ur kindness, please still share again
LikeLike
Comment by hidayat — December 23, 2012 @ 7:44 AM
seeking youtube error,lagi sibuk cari copas lum ketemu ya syed 😛
LikeLike
Comment by lengob — January 2, 2013 @ 1:26 AM
bukan sibuk nyari copas… udah setahun lebih youtube di blokir dari pakistan mas bro..
LikeLike
Comment by jokopenthol — March 25, 2013 @ 11:33 PM
pasti tdk ada yg tau range full untuk youtube…………. Kalau yang berani kasih range full bukan 0,51 pasti saya jempol dia…..wkwkwkwkwkwkwkwk
LikeLike
Comment by andhyka — June 4, 2013 @ 11:23 AM
kalau ada yang berani kasih range full untuk youtube, pasti saya jempol dia……. wkwkwkwkwkwkwkwk
LikeLike
Comment by ANDHYKA PRADANA - MAKASSAR — June 4, 2013 @ 11:28 AM
[…] Read More: https://aacable.wordpress.com/2012/01/30/youtube-caching-problem-an-error-occured-please-try-again-la… […]
LikeLike
Pingback by Get Latest News Around The World — February 2, 2013 @ 3:39 AM
Storeurl.pl berjalan baik dan normal untuk Lusca http://mr-ekoapriadi.blogspot.com/2013/04/info-storeurlpl-berjalan-baik-dan.html
LikeLike
Comment by Adam — May 5, 2013 @ 10:48 PM
I want to update everybody about StoreID which is the alternative to store_url_rewrite.
it comes in the new squid 3.4 and now in squid head which can be compiled manually.
feel free to contact me about it.
http://wiki.squid-cache.org/Features/StoreID
LikeLike
Comment by Eliezer — May 6, 2013 @ 2:23 PM
Thank You Eliezer 🙂 , But dueto ban on youtube in our country, we cant do any testing on it.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — May 13, 2013 @ 8:42 AM
Well I have one nice server to test it if you want.
we can also use AWS tiny server for that.. use some ssh\ssl\vpn to get into the aws…
LikeLike
Comment by Eliezer — June 4, 2013 @ 3:36 PM
Thanks , your post really helped me…………. 🙂
LikeLike
Comment by Deon — May 26, 2013 @ 4:41 PM
range full youtube Non 0.51 second……
LikeLike
Comment by BOCCO — June 4, 2013 @ 11:25 AM
Hello Sir Syed, I’m using Pfsense+Lusca and using your storeurl.pl seems to be working very well but can you please post updated refresh pattern as well to match up ur storeurl.pl? thank you.
LikeLike
Comment by Oliver — October 20, 2013 @ 9:02 PM
provide more details.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — October 21, 2013 @ 8:29 AM
##### crontab untuk menghapus file yg sudah tidak terpakai lebih dari 1 jam yang lalu
## crontab perbaikan dari warnet ersa pati ( pak lutfi )
0 * * * * find /var/log/squid/ -maxdepth 1 ! -name “*.log” -type f -mmin +60 -delete >> /dev/null 2>&1
############# Squid Config
acl youtube url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204) .*$
acl youtube url_regex -i \.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
storeurl_access allow youtube
############ storeurl.pl ( squid-2.7.Stable9
} elsif ($X[1] =~ m/^http(|s)\:\/\/.*youtube.*(ptracking|stream_204|player_204|gen_204).*(video_id|docid|v)\=([^\&\s]*).*/){
$vid = $4 ;
@cpn = m/[&?]cpn\=([^\&\s]*)/;
$fn = “/var/log/squid/@cpn”;
unless (-e $fn) {
open FH,”>”.$fn ;
print FH “$vid\n”;
close FH;
}
print $x . $X[1] . “\n”;
} elsif ($X[1] =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
@itag = m/[&?](itag=[0-9]*)/;
@ids = m/[&?]id\=([^\&\s]*)/;
@mime = m/[&?](mime\=[^\&\s]*)/;
@cpn = m/[&?]cpn\=([^\&\s]*)/;
if (defined($cpn[0])) {
$fn = “/var/log/squid/@cpn”;
if (-e $fn) {
open FH,”<".$fn ;
$id = ;
chomp $id ;
close FH ;
} else {
$id = $ids[0] ;
}
} else {
$id = $ids[0] ;
}
@range = m/[&?](range=[^\&\s]*)/;
print $x . “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range@mime\n”;
########## Store-ID.pl ( Squid-3.4 atau Squid-3.HEAD )
if ($x =~ m/^http(|s)\:\/\/.*youtube.*(ptracking|stream_204|player_204|gen_204).*(video_id|docid|v)\=([^\&\s]*).*/){
$vid = $4 ;
@cpn = m/[&?]cpn\=([^\&\s]*)/;
$fn = “/var/log/squid/@cpn”;
unless (-e $fn) {
open FH,”>”.$fn ;
print FH “$vid\n”;
close FH;
}
$out = $x . “\n”;
} elsif ($x =~ m/^http\:\/\/.*(youtube|google).*videoplayback.*/){
@itag = m/[&?](itag=[0-9]*)/;
@ids = m/[&?]id\=([^\&\s]*)/;
@mime = m/[&?](mime\=[^\&\s]*)/;
@cpn = m/[&?]cpn\=([^\&\s]*)/;
if (defined($cpn[0])) {
$fn = “/var/log/squid/@cpn”;
if (-e $fn) {
open FH,”<".$fn ;
$id = ;
chomp $id ;
close FH ;
} else {
$id = $ids[0] ;
}
} else {
$id = $ids[0] ;
}
@range = m/[&?](range=[^\&\s]*)/;
$out = “http://video-srv.youtube.com.SQUIDINTERNAL/id=” . $id . “&@itag@range@mime”;
LikeLike
Comment by Syaifuddin JW — October 24, 2013 @ 10:09 PM
https://code.google.com/p/tempat-sampah/source/browse/storeurl.pl
LikeLike
Comment by Syaifuddin JW — October 24, 2013 @ 10:11 PM
Tried copy paste the Storeurl.pl above on my Pfsense 2.0.1 + Lusca cache + squidguard.
i deleted my old storeurl and pasted the above code. still experiencing Youtube error.. is my method correct?
or i’m missing other methods? Thank you!!!
LikeLike
Comment by Edward Caezar Pababero — January 28, 2014 @ 8:23 AM
try to use new squid.conf
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — January 28, 2014 @ 3:07 PM
Sir do i have to erase all codes written here /usr/local/etc/squid/squid.conf?
Here’s the code written:
# Do not edit manually !
http_port 192.168.1.1:3128 transparent
http_port 127.0.0.1:80 transparent
icp_port 0
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_directory /usr/local/etc/squid/errors/English
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log none
cache_log /var/squid/logs/cache.log
cache_store_log none
shutdown_lifetime 0 seconds
# Allow local network(s) on interface(s)
acl localnet src 192.168.1.0/255.255.255.0
uri_whitespace strip
cache_mem 8 MB
maximum_object_size_in_memory 4 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir coss /var/squid/coss 50 max-size=4096 block-size=512
cache_dir aufs /var/squid/cache 420000 16 256 min-size=4096
minimum_object_size 0 KB
maximum_object_size 1024 MB
offline_mode off
cache_swap_low 98
cache_swap_high 99
acl donotcache dstdomain “/var/squid/acl/donotcache.acl”
cache deny donotcache
# No redirector configured
# Setup some default acls
acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 1025-65535
acl sslports port 443 563
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
acl partialcontent_req req_header Range .*
#acl dynamic urlpath_regex cgi-bin \?
include /usr/local/etc/squid/include.conf
#cache deny dynamic
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports
# Always allow localhost connections
http_access allow localhost
quick_abort_min 32 KB
quick_abort_max 128 KB
quick_abort_pct 75
range_offset_limit 0 MB
request_body_max_size 0 allow all
reply_body_max_size 0 deny all
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 30720/30720
delay_initial_bucket_level 100
delay_access 1 allow all
# Custom options
negative_ttl 0 seconds
redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass off
url_rewrite_children 5
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny all
Do i have to erase that and replace it with the new squid.conf? by the way im using Pfsense 2.0.1 +Lusca cache
LikeLike
Comment by Edward Caezar Pababero — January 28, 2014 @ 4:28 PM
youtube not cache, can you new update stroreurl.pl ?
LikeLike
Comment by machmud — March 7, 2014 @ 12:14 PM
Try this one.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — March 7, 2014 @ 4:22 PM
thanks … great job .. very help full with this…. my You
Tube cache running well
LikeLike
Comment by banaspati — March 10, 2014 @ 9:50 AM
not working on speedtest.net pls help.. thnx
LikeLike
Comment by jv — March 25, 2014 @ 12:35 PM
you need to add custom rule for speedtest.net, something like these just for example, you may need to edit to your need
-> in squid.conf
#speedtest
refresh_pattern -i speedtest.net|speedtest.san.fastserv.com/mini/speedtest/latency.txt|speedtest.san.fastserv.com/mini/speedtest/random350x350.jpg|speedtest.san.fastserv.com/mini/speedtest/upload.php 10080 90% 999999 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private ignore-auth
#speedtest
acl speedtest url_regex -i speedtest.net|speedtest.*.com/mini/speedtest/random350x350.jpg
#this may cause slow when testing upload
#|speedtest.san.fastserv.com/mini/speedtest/upload.php|speedtest.san.fastserv.com/mini/speedtest/latency.txt
#acl speedtest url_regex -i speedtest.san.fastserv.com|speedtest.net
cache allow speedtest
storeurl_access allow speedtest
-> in storeurl.pl
#speedtest
}elsif (m/^http:\/\/(.*)\/speedtest\.net\/(.*\.(jpg|txt))\?(.*)/) {
$out=”http://www.speedtest.net.SQUIDINTERNAL/speedtest/” . $2 . “”;
}elsif (m/^http:\/\/(.*)\/speedtest\.san\.fastserv\.com\/(.*\.(jpg|txt|php))\?(.*)/) {
$out=”http://www.speedtest.san.fastserv.com.SQUIDINTERNAL/mini/speedtest/” . $2 . “”;
LikeLike
Comment by nauval2007 — March 27, 2014 @ 2:40 PM
Hi, Goodmorning,
My pfsense + Lusca cache is not caching youtube anymore, since march 2014. is this applicable for lusca cache?
Thanks…
LikeLike
Comment by Edward Caezar Pababero — April 1, 2014 @ 6:28 AM
the settings are intended to be used on squid 2.7.9
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — April 1, 2014 @ 10:59 AM
caching youtube not working anymore, i’ve been using storeurl.pl from this site https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/..any update?thanks sir..
LikeLike
Comment by rcp — April 1, 2014 @ 2:36 PM
no funciona mas los storeurl.pl antes andaba muy bien pero ahora ya no anda y gracias por los aporte men
LikeLike
Comment by jose — April 23, 2014 @ 8:26 PM
Espero master Alastaz y el genio Alaskoid Ayuda en efectivo
Me Mstab Apinto 10.4.4
Y trabajado todos los pasos anteriores explicación y Lai viabilidad red de malla temprana
Posible utilizar
Si es posible, las órdenes y todos los archivos necesarios 03/01/19
Waukon Shaker Avdhalkm
Mizo hermano de Egipto
LikeLike
Comment by mezoosaidosaid — May 5, 2014 @ 9:43 PM