Syed Jahanzaib – Personal Blog to Share Knowledge !

August 24, 2012

Squid don’t cache IDM downloads ?

Filed under: Linux Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 4:49 PM

I was having a problem that if a user download any file like .exe using IDM, it never cached by squid proxy, So I did some digging and found out that IDM was splitting the file into multiple pieces and requesting each of those pieces as different HTTP requests & the Squid cannot cache partial files by default, only whole file can be cached.
As you can clearly see in the below image that  no matter how many times I download the same file using IDM, it always show TCP_MISS in the squid access logs.

Downloads via IDM always MISS

.

I solved this by adding following two directives in squid.conf and then restart squid service.

range_offset_limit -1

Now I again downloaded the file twice using IDM, and this time I can see TCP_HIT and the cache hit worked like a charm. As showed in the image below.

Downloads via IDM gets HIT after tunning squid.conf

.

CAUTION:Use this directive with care, because in some cases it can give you undesirable results including excessive bandwidth usage, e.g: If a user starts downloading any file of any size , lets suppose 600 MB, and he cancel the download in middle or after 10MB, the  SQUID will continue to download the whole file in the cache. This will cause a bigger problem , if user downloading a file with
download manager ,let’s say in 6 segments , squid will start 6 download threads for the same file each one from its beginning. & it will  consume 6 times bandwidth than really needed. so it can really become bottleneck in some cases, specially if you have small bandwidth pipe.So be careful using this option.  Most people prefer the opposite, to make Squid abort downloads immediately.because this kind of continued downloads can easily chew up a considerable amount of bandwidth for a very log hit ratio.

Regard’s
Syed Jahanzaib

35 Comments »

  1. Awesome,
    I was going to dig about this problem but you beat me to it.

    Thanks,

    Like

    Comment by Badr — August 24, 2012 @ 6:35 PM

  2. Thanks Syed Jahanzaib so much i was search about this problem 🙂

    Like

    Comment by SquidY — August 24, 2012 @ 8:37 PM

  3. Sir if we set quick_abort to -1 then squid will fetch whole file from internet either user is downloading or canceled the download. Similarly if any user quit youtube or any other big file in the middle of transfer then squid will fetch complete file which causes bandwidth bottleneck.

    Like

    Comment by smsaqib — August 25, 2012 @ 12:04 AM

  4. Note from squid official site.
    For range_offset_limit:

    The range request wil be fully fetched from start to finish regardless of the client actions(START/CANCEL the download). This affects bandwidth usage.

    For qucik_abort_min:

    The cache by default continues downloading aborted requests which are almost completed (less than 16 KB remaining). This may be undesirable on slow links and/or very busy caches. Impatient users may tie up file descriptors and bandwidth by repeatedly requesting and immediately aborting downloads.

    Like

    Comment by smsaqib — August 25, 2012 @ 12:40 AM

  5. thankss you man

    Like

    Comment by malak — August 25, 2012 @ 4:28 AM

  6. NOTE: If range_offset_limit ist set to -1 the quick abort options will NOT work

    source: http://doc.pfsense.org/index.php/Squid_Package_Tuning

    Like

    Comment by incubus14 — August 25, 2012 @ 12:00 PM

  7. is there any downside or negative effect using this parameter? because i think in the background it will always download the file which may cause excessive used of bandwidth even if you abort the download of the file.

    thanks!

    Like

    Comment by incubus14 — August 25, 2012 @ 12:11 PM

  8. set quick_abort_min = 512 KB or 1 MB
    set the range_offset_limit = 10 MB or file size you want to downloaded by cache regardless of user actions.

    As i set =10 MB and squid box downloads 10 mb file from start to end to cache it. following is the results.

    1345895797.893 8628 10.10.10.129 TCP_HIT/206 701075 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
    1345895797.968 8684 10.10.10.129 TCP_HIT/206 682715 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
    1345895799.504 10806 10.10.10.129 TCP_HIT/200 1122558 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg
    1345895799.782 11049 10.10.10.129 TCP_HIT/206 1044793 GET http://sound18.mp3pk.com/indian/ekthatiger/%5BSongs.PK%5D%20Ek%20Tha%20Tiger%20-%2002%20-%20Laapata.mp3 – NONE/- audio/mpeg

    one more thing about setting quick_abort_min = -1. Consider a user is watching video of file size 300mb-600mb and then user quits the video after few minutes but squid will download whole file 300mb-600mb(waste of bandwidth speacially for 1,2,4 mb link).

    Like

    Comment by smsaqib — August 25, 2012 @ 5:08 PM

  9. Choosing correct value for quick_abort_min:
    If a user cancel any file retrieval or download squid perform following action;

    1. if remaining part of file is less than the value of quick_abort_min then squid completes the download
    2. else if the remaining part of file is greater than quick_abort_min then squid terminate the download
    3. else if quick_abort_min = -1 then squid always completes the download regardless of user action.

    squid by default uses quick abort min=16KB

    here comes mathematical part

    1Mb = 128 KB
    2Mb = 256 KB
    4Mb = 512 KB
    8Mb = 1 MB
    10Mb= 1280 KB

    choose right hand side values of above equations for quick_abort_min. If statement “1” is true then squid will saturate your link for one second to finish the retrieval. if you want squid to saturate link more than one second then multiply the chosen value by no. of seconds.
    e.g. i want squid to saturate link for 5 seconds and i have 1mb connection,
    5 X 128 KB = 640 KB = Quick abort min.

    Assumptions:
    You have a large amount of users.
    Your cache is very busy cache.

    if above two conditions are not true for you then you can use twice or four times greater value.
    Choosing correct value for range_offset_limit:

    set the range_offset_limit = 10 MB or file size you want to downloaded by cache regardless of user actions.

    Warning: if you set range_offset_limit to -1 then it will override quick_abort_min = -1 and squid will consume large amount bandwidth + rapidly when you have impatient users.

    Like

    Comment by smsaqib — August 26, 2012 @ 1:36 PM

  10. after adding range_offset_limit 10 MB. My net is too slow and i have set quick_abort_min 512 KB. My idm is caching now but its not creating peers. so becoz of this speed is too slow

    Like

    Comment by Farhan — August 27, 2012 @ 11:59 AM

    • IDM makes range requests(multiple connection) of a same file to download from server.
      when you set range_offset_limit = XX MB then squid checks that if file size is less than XX MB then squid make single connection with server to download itand cache it. if file size is greater than XX MB then squid passes the same range requests (multiple connections to same file) to server as received from IDM.

      Like

      Comment by smsaqib — August 27, 2012 @ 6:02 PM

      • helperHandleRead: unexpected reply on channel -1 from store_rewriter #1 ”
        squidaio_queue_request: WARNING – Queue congestion

        how to solve this two warnings/errors in cache log?

        Like

        Comment by incubus14 — August 31, 2012 @ 12:44 PM

  11. quick_abort_pct 70;

    To ensure that a file is only downloaded if a user actually receives 70% or more of it. Otherwise if a user requests a file and then aborts, it will download the whole file.

    NOTE: If range_offset_limit ist set to -1 the quick abort options will NOT work

    Like

    Comment by int21 — August 27, 2012 @ 8:13 PM

  12. Dear Jahanzaib there is also same problem with mikrotik builtin web proxy is there any solution for that if yes Kindly share it , Regards

    Like

    Comment by Silent Wishes — August 28, 2012 @ 12:36 AM

  13. Dear jahanzaib bhai i got this error when i try to enable url rewriter program.
    m using Centos 5.5 kindly advise me how can i resolve this issue.

    2012/08/29 10:44:10| Ready to serve requests.
    2012/08/29 10:44:10| WARNING: url_rewriter #1 (FD 6) exited
    2012/08/29 10:44:10| WARNING: url_rewriter #2 (FD 8) exited
    2012/08/29 10:44:10| WARNING: url_rewriter #3 (FD 9) exited
    2012/08/29 10:44:10| Too few url_rewriter processes are running
    FATAL: The url_rewriter helpers are crashing too rapidly, need help!

    Like

    Comment by owais — August 29, 2012 @ 10:55 AM

  14. helperHandleRead: unexpected reply on channel -1 from store_rewriter #1 ”
    squidaio_queue_request: WARNING – Queue congestion

    how to solve this two warnings/errors in cache log?

    thanks!

    Like

    Comment by incubus14 — August 31, 2012 @ 12:44 PM

    • increase the file descriptor for more efficient performance compile the squid with as per your sessions

      Like

      Comment by owais — September 5, 2012 @ 5:53 PM

  15. My Squid was hacked from Chinese IP (61.0.0.0/8). How can I secure it ?

    Like

    Comment by cm — September 12, 2012 @ 1:44 AM

    • use iptables to protect your squid/linux. close all ports for external interface, and allow only fewer ports on inernal LAN that are required. also if possible allow only specific ips on specific port for access.

      Like

      Comment by Syed Jahanzaib / Pinochio~:) — September 12, 2012 @ 10:33 AM

  16. salam

    i have a problem “watching” youtube when enabling squid , i can only watch the first part of the video then it keep waiting forever !! I know this is due to the new policy of youtube but is there any solution for it ? i tried rang_offset_limit -1 , and my god it’s eating the whole bandwidth !!!!! i’m about to surrender and shutdown my squid , this is a bigger problem , cause it’s really saving an expensive bandwidth , but i don’t think that there is no solution i’m sure i have somthing wrong in my config , please i need your help , i tried many solutions but it doesn’t help ..

    Like

    Comment by sadook — September 14, 2012 @ 6:11 AM

  17. salam,
    all work fine but have a question that when client hit to mikrotik and client bandwidth limit is 768kb and its request forward to squid by mikrotik squid consume full bandwidth to complete request, bcoz squid does not know the bandwidth shaping of mikrotik so how i can able to squid will knows the bandwidth usage. or any QOS method to make squid intelligent.

    Like

    Comment by rehmat ali — September 24, 2012 @ 7:55 PM

  18. ok then plzz post a best QOS scrpit for 1st priority browsing then others. thxz for ur reply

    Like

    Comment by Rehmat ali — October 2, 2012 @ 9:31 AM

  19. dear janzaib
    3 days ago hotmail and facebook making problem
    1. hotmail
    when i open hotmail in browser and enter my username and password and click ok then page is going hung up inboc not opening.
    2.facebook
    facebook is login fine but it also hang up after login, means fb only show its 1st page and when i m click any tak or any friend name page hangup.

    aur haan jab mai mikrotik sai dstNAT disable kerta houn only srcNAT pe hoota hai rule tu hotmail aur facebook dono sahi chaltey hien plz koi acha sa pattren batien is masley k liye.
    i did not change any code in linux
    thxzzzzzzzzzzzz

    Like

    Comment by Rehmat ali — November 11, 2012 @ 5:38 AM

  20. plzzz reply as soon as possible

    Like

    Comment by Rehmat ali — November 11, 2012 @ 8:34 PM

  21. hello syed
    what is the last modified for this issue till now so that idm dont take your bandwidth ?
    why we cant cache idm in storurl
    like we do for youtube ??
    thank you 🙂

    Like

    Comment by Hussein — November 1, 2013 @ 6:55 PM

  22. salam

    i have LUSCA_HEAD-r14942 and its working well but i cant cache IDM downloaded file i tried to change range_offset_limit to -1 and to 10 MB but no luck any one have idea how to fix it Thanks

    Like

    Comment by Mike — December 27, 2014 @ 3:33 AM

  23. Thanks Syed Jahanzaib
    i want to ask did you find any way to cache youtube or https websites

    Like

    Comment by Mike — January 27, 2015 @ 11:12 PM

  24. Sir…………..i want block idm cache any files in squid
    how to do this…………?
    ALLAH aap ko jaza e khair dey………….!

    Like

    Comment by Salman — April 6, 2015 @ 11:06 PM

  25. Can you please share a working squid SSO Config with AD

    Like

    Comment by maqsoodahmad1570 — April 22, 2019 @ 10:02 AM


RSS feed for comments on this post. TrackBack URI

Leave a comment