I was having a problem that if a user download any file like .exe using IDM, it never cached by squid proxy, So I did some digging and found out that IDM was splitting the file into multiple pieces and requesting each of those pieces as different HTTP requests & the Squid cannot cache partial files by default, only whole file can be cached.
As you can clearly see in the below image that no matter how many times I download the same file using IDM, it always show TCP_MISS in the squid access logs.
.
I solved this by adding following two directives in squid.conf and then restart squid service.
range_offset_limit -1
Now I again downloaded the file twice using IDM, and this time I can see TCP_HIT and the cache hit worked like a charm. As showed in the image below.
.
CAUTION:Use this directive with care, because in some cases it can give you undesirable results including excessive bandwidth usage, e.g: If a user starts downloading any file of any size , lets suppose 600 MB, and he cancel the download in middle or after 10MB, the SQUID will continue to download the whole file in the cache. This will cause a bigger problem , if user downloading a file with
download manager ,let’s say in 6 segments , squid will start 6 download threads for the same file each one from its beginning. & it will consume 6 times bandwidth than really needed. so it can really become bottleneck in some cases, specially if you have small bandwidth pipe.So be careful using this option. Most people prefer the opposite, to make Squid abort downloads immediately.because this kind of continued downloads can easily chew up a considerable amount of bandwidth for a very log hit ratio.
Regard’s
Syed Jahanzaib
Awesome,
I was going to dig about this problem but you beat me to it.
Thanks,
LikeLike
Comment by Badr — August 24, 2012 @ 6:35 PM
Thanks Syed Jahanzaib so much i was search about this problem 🙂
LikeLike
Comment by SquidY — August 24, 2012 @ 8:37 PM
Sir if we set quick_abort to -1 then squid will fetch whole file from internet either user is downloading or canceled the download. Similarly if any user quit youtube or any other big file in the middle of transfer then squid will fetch complete file which causes bandwidth bottleneck.
LikeLike
Comment by smsaqib — August 25, 2012 @ 12:04 AM
Note from squid official site.
For range_offset_limit:
The range request wil be fully fetched from start to finish regardless of the client actions(START/CANCEL the download). This affects bandwidth usage.
For qucik_abort_min:
The cache by default continues downloading aborted requests which are almost completed (less than 16 KB remaining). This may be undesirable on slow links and/or very busy caches. Impatient users may tie up file descriptors and bandwidth by repeatedly requesting and immediately aborting downloads.
LikeLike
Comment by smsaqib — August 25, 2012 @ 12:40 AM
thankss you man
LikeLike
Comment by malak — August 25, 2012 @ 4:28 AM
NOTE: If range_offset_limit ist set to -1 the quick abort options will NOT work
source: http://doc.pfsense.org/index.php/Squid_Package_Tuning
LikeLike
Comment by incubus14 — August 25, 2012 @ 12:00 PM
is there any downside or negative effect using this parameter? because i think in the background it will always download the file which may cause excessive used of bandwidth even if you abort the download of the file.
thanks!
LikeLike
Comment by incubus14 — August 25, 2012 @ 12:11 PM
set quick_abort_min = 512 KB or 1 MB
set the range_offset_limit = 10 MB or file size you want to downloaded by cache regardless of user actions.
As i set =10 MB and squid box downloads 10 mb file from start to end to cache it. following is the results.
1345895797.893 8628 10.10.10.129 TCP_HIT/206 701075 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
1345895797.968 8684 10.10.10.129 TCP_HIT/206 682715 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
1345895799.504 10806 10.10.10.129 TCP_HIT/200 1122558 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
1345895799.782 11049 10.10.10.129 TCP_HIT/206 1044793 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
one more thing about setting quick_abort_min = -1. Consider a user is watching video of file size 300mb-600mb and then user quits the video after few minutes but squid will download whole file 300mb-600mb(waste of bandwidth speacially for 1,2,4 mb link).
LikeLike
Comment by smsaqib — August 25, 2012 @ 5:08 PM
Dear Saqib,
Thank you for sharing your experiences with us. Please keep sharing 🙂
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 26, 2012 @ 10:02 AM
Choosing correct value for quick_abort_min:
If a user cancel any file retrieval or download squid perform following action;
1. if remaining part of file is less than the value of quick_abort_min then squid completes the download
2. else if the remaining part of file is greater than quick_abort_min then squid terminate the download
3. else if quick_abort_min = -1 then squid always completes the download regardless of user action.
squid by default uses quick abort min=16KB
here comes mathematical part
1Mb = 128 KB
2Mb = 256 KB
4Mb = 512 KB
8Mb = 1 MB
10Mb= 1280 KB
choose right hand side values of above equations for quick_abort_min. If statement “1” is true then squid will saturate your link for one second to finish the retrieval. if you want squid to saturate link more than one second then multiply the chosen value by no. of seconds.
e.g. i want squid to saturate link for 5 seconds and i have 1mb connection,
5 X 128 KB = 640 KB = Quick abort min.
Assumptions:
You have a large amount of users.
Your cache is very busy cache.
if above two conditions are not true for you then you can use twice or four times greater value.
Choosing correct value for range_offset_limit:
set the range_offset_limit = 10 MB or file size you want to downloaded by cache regardless of user actions.
Warning: if you set range_offset_limit to -1 then it will override quick_abort_min = -1 and squid will consume large amount bandwidth + rapidly when you have impatient users.
LikeLike
Comment by smsaqib — August 26, 2012 @ 1:36 PM
True!
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 27, 2012 @ 8:49 AM
after adding range_offset_limit 10 MB. My net is too slow and i have set quick_abort_min 512 KB. My idm is caching now but its not creating peers. so becoz of this speed is too slow
LikeLike
Comment by Farhan — August 27, 2012 @ 11:59 AM
IDM makes range requests(multiple connection) of a same file to download from server.
when you set range_offset_limit = XX MB then squid checks that if file size is less than XX MB then squid make single connection with server to download itand cache it. if file size is greater than XX MB then squid passes the same range requests (multiple connections to same file) to server as received from IDM.
LikeLike
Comment by smsaqib — August 27, 2012 @ 6:02 PM
helperHandleRead: unexpected reply on channel -1 from store_rewriter #1 ”
squidaio_queue_request: WARNING – Queue congestion
how to solve this two warnings/errors in cache log?
LikeLike
Comment by incubus14 — August 31, 2012 @ 12:44 PM
quick_abort_pct 70;
To ensure that a file is only downloaded if a user actually receives 70% or more of it. Otherwise if a user requests a file and then aborts, it will download the whole file.
NOTE: If range_offset_limit ist set to -1 the quick abort options will NOT work
LikeLike
Comment by int21 — August 27, 2012 @ 8:13 PM
Dear Jahanzaib there is also same problem with mikrotik builtin web proxy is there any solution for that if yes Kindly share it , Regards
LikeLike
Comment by Silent Wishes — August 28, 2012 @ 12:36 AM
not yet.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 28, 2012 @ 9:20 AM
Dear jahanzaib bhai i got this error when i try to enable url rewriter program.
m using Centos 5.5 kindly advise me how can i resolve this issue.
2012/08/29 10:44:10| Ready to serve requests.
2012/08/29 10:44:10| WARNING: url_rewriter #1 (FD 6) exited
2012/08/29 10:44:10| WARNING: url_rewriter #2 (FD 8) exited
2012/08/29 10:44:10| WARNING: url_rewriter #3 (FD 9) exited
2012/08/29 10:44:10| Too few url_rewriter processes are running
FATAL: The url_rewriter helpers are crashing too rapidly, need help!
LikeLike
Comment by owais — August 29, 2012 @ 10:55 AM
helperHandleRead: unexpected reply on channel -1 from store_rewriter #1 ”
squidaio_queue_request: WARNING – Queue congestion
how to solve this two warnings/errors in cache log?
thanks!
LikeLike
Comment by incubus14 — August 31, 2012 @ 12:44 PM
increase the file descriptor for more efficient performance compile the squid with as per your sessions
LikeLike
Comment by owais — September 5, 2012 @ 5:53 PM
My Squid was hacked from Chinese IP (61.0.0.0/8). How can I secure it ?
LikeLike
Comment by cm — September 12, 2012 @ 1:44 AM
use iptables to protect your squid/linux. close all ports for external interface, and allow only fewer ports on inernal LAN that are required. also if possible allow only specific ips on specific port for access.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — September 12, 2012 @ 10:33 AM
salam
i have a problem “watching” youtube when enabling squid , i can only watch the first part of the video then it keep waiting forever !! I know this is due to the new policy of youtube but is there any solution for it ? i tried rang_offset_limit -1 , and my god it’s eating the whole bandwidth !!!!! i’m about to surrender and shutdown my squid , this is a bigger problem , cause it’s really saving an expensive bandwidth , but i don’t think that there is no solution i’m sure i have somthing wrong in my config , please i need your help , i tried many solutions but it doesn’t help ..
LikeLike
Comment by sadook — September 14, 2012 @ 6:11 AM
salam,
all work fine but have a question that when client hit to mikrotik and client bandwidth limit is 768kb and its request forward to squid by mikrotik squid consume full bandwidth to complete request, bcoz squid does not know the bandwidth shaping of mikrotik so how i can able to squid will knows the bandwidth usage. or any QOS method to make squid intelligent.
LikeLike
Comment by rehmat ali — September 24, 2012 @ 7:55 PM
It’s the default behavior of squid that it consumes all available bandwidth to fulfill the client request..
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — September 29, 2012 @ 4:27 PM
ok then plzz post a best QOS scrpit for 1st priority browsing then others. thxz for ur reply
LikeLike
Comment by Rehmat ali — October 2, 2012 @ 9:31 AM
dear janzaib
3 days ago hotmail and facebook making problem
1. hotmail
when i open hotmail in browser and enter my username and password and click ok then page is going hung up inboc not opening.
2.facebook
facebook is login fine but it also hang up after login, means fb only show its 1st page and when i m click any tak or any friend name page hangup.
aur haan jab mai mikrotik sai dstNAT disable kerta houn only srcNAT pe hoota hai rule tu hotmail aur facebook dono sahi chaltey hien plz koi acha sa pattren batien is masley k liye.
i did not change any code in linux
thxzzzzzzzzzzzz
LikeLike
Comment by Rehmat ali — November 11, 2012 @ 5:38 AM
plzzz reply as soon as possible
LikeLike
Comment by Rehmat ali — November 11, 2012 @ 8:34 PM
hello syed
what is the last modified for this issue till now so that idm dont take your bandwidth ?
why we cant cache idm in storurl
like we do for youtube ??
thank you 🙂
LikeLike
Comment by Hussein — November 1, 2013 @ 6:55 PM
salam
i have LUSCA_HEAD-r14942 and its working well but i cant cache IDM downloaded file i tried to change range_offset_limit to -1 and to 10 MB but no luck any one have idea how to fix it Thanks
LikeLike
Comment by Mike — December 27, 2014 @ 3:33 AM
make sure the download isnt coming from any dynamic site like ? or https, as http caching or some dynamic sites are not supported.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — December 29, 2014 @ 9:19 AM
Thanks Syed Jahanzaib
i want to ask did you find any way to cache youtube or https websites
LikeLike
Comment by Mike — January 27, 2015 @ 11:12 PM
its possible with IMPORT LOCAL CERTIFICATE, but not recommended
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — January 28, 2015 @ 3:08 PM
Sir…………..i want block idm cache any files in squid
how to do this…………?
ALLAH aap ko jaza e khair dey………….!
LikeLike
Comment by Salman — April 6, 2015 @ 11:06 PM
Can you please share a working squid SSO Config with AD
LikeLike
Comment by maqsoodahmad1570 — April 22, 2019 @ 10:02 AM