Syed Jahanzaib Personal Blog to Share Knowledge !

August 24, 2012

Squid don’t cache IDM downloads ?

Filed under: Linux Related — Tags: , , — Syed Jahanzaib / Pinochio~:) @ 4:49 PM

I was having a problem that if a user download any file like .exe using IDM, it never cached by squid proxy, So I did some digging and found out that IDM was splitting the file into multiple pieces and requesting each of those pieces as different HTTP requests & the Squid cannot cache partial files by default, only whole file can be cached.
As you can clearly see in the below image that  no matter how many times I download the same file using IDM, it always show TCP_MISS in the squid access logs.

Downloads via IDM always MISS


I solved this by adding following two directives in squid.conf and then restart squid service.

range_offset_limit -1

Now I again downloaded the file twice using IDM, and this time I can see TCP_HIT and the cache hit worked like a charm. As showed in the image below.

Downloads via IDM gets HIT after tunning squid.conf


CAUTION:Use this directive with care, because in some cases it can give you undesirable results including excessive bandwidth usage, e.g: If a user starts downloading any file of any size , lets suppose 600 MB, and he cancel the download in middle or after 10MB, the  SQUID will continue to download the whole file in the cache. This will cause a bigger problem , if user downloading a file with
download manager ,let’s say in 6 segments , squid will start 6 download threads for the same file each one from its beginning. & it will  consume 6 times bandwidth than really needed. so it can really become bottleneck in some cases, specially if you have small bandwidth pipe.So be careful using this option.  Most people prefer the opposite, to make Squid abort downloads immediately.because this kind of continued downloads can easily chew up a considerable amount of bandwidth for a very log hit ratio.

Syed Jahanzaib

%d bloggers like this: