WAN utilization @ 60 mbps / zaib
First of all , intro to what VIDEOCACHE is ?
” Videocache is a Squid URL plugin to cache videos from famous video sharing portals/websites like Youtube, Metacafe etc. It helps you save bandwidth when a particular video is requested more than once from the same network/machine.
Squid can not cache the videos served dynamically. Videocache fits into squid to help it cache the videos as well. The cached videos are stored on your proxy server’s local storage or on a storage server in your network. These cached videos can be served to your clients at a very fast speed saving your upstream bandwidth. “
Following are some working configuration samples for VIDEOCACHE with SQUID on UBUNTU. Installation of VC is fairly simple as mentioned here step by step
http://cachevideos.com/documentation/
I am not showing howto install VIDEOCACHE plugin as its very simple and briefly described at the author site, I am only posting the working configuration files. Also note that videocache is a paid plugin, which costs around 400$ per year.
If you are looking for something free, you may try the following NIGNX method.
https://aacable.wordpress.com/2012/08/13/youtube-caching-with-squid-nginx/
Also I would like to share that for VIDEOCACHE plugin, you *MUST* have some good hardware with *LOTS OF* memory and STORAGE. Just for an idea, I deployed it at a network of 300 Users which had Dual CPU with multi cores server for proxy , 128GB RAM, and 24 TB of external HP Storage, CPU was not much utilized, but at full peak time, the memory usage was around 95%, and the storage usage was around 180 GB in just 1 day.
Also you should have some good amount of bandwidth, otherwise videocache will become bottleneck for your network 🙂 , OR schedule VIDEOCACHE to cache only in specific timings, its supported in /etc/videocache.conf
Following is an working Squid 2.7 on Ubuntu Server 10.4 with videocache support. Please modify necessary directives before using , for example cache_mem, hostname, ACL, Ip’s. Also make sure the CACHE_DIR have been changed to match your local directory structure with proper permissions assgined to user proxy or squid , depend on he nix flavor you are using
# SQUID 2.7 CONFIG FILE WITH VIDEOCACHE PLUGIN FOR SQUID/VIDEOS [ Zaib]
# SQUID 2.7 WORKING CONFIG FILE [August 2013] with Videocache Support
# By - Syed Jahanzaib
# Email: aacable@hotmail.com
# Web : https://aacable.wordpress.com
cache_store_log /var/log/squid/store.log
acl this_machine src 127.0.0.1 192.168.100.1
http_access allow this_machine
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 20
acl vc_deny_url url_regex -i \.blip\.tv\/(.*)filename \.hardsextube\.com\/videothumbs \.xtube\.com\/(.*)(Thumb|videowall)
acl vc_url url_regex -i \/youku\/[0-9A-Z]+\/[0-9A-Z\-]+\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg)
acl vc_url url_regex -i \/(.*)key=[a-z0-9]+(.*)\.flv
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/feeds\/api\/videos\/[0-9a-zA-Z_-]{11}\/
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/(videoplayback|get_video|watch|watch_popup|user_watch)\?
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/(v|e|embed)\/[0-9a-zA-Z_-]{11}
acl vc_dom_r dstdom_regex -i msn\..*\.(com|net)
acl vc_dom_r dstdom_regex -i msnbc\..*\.(com|net)
acl vc_dom_r dstdom_regex -i video\..*\.fbcdn\.net
acl vc_dom_r dstdom_regex -i myspacecdn\..*\.footprint\.net
acl vc_dom dstdomain .stream.aol.com .videos.5min.com msn.com .blip.tv .vid.ec.dmcdn.net .break.com .vimeocdn.com
acl vc_dom dstdomain .cdn.turner.com .dailymotion.com .c.wrzuta.pl .v.imwx.com .mccont.com .myspacecdn.com
acl vc_dom dstdomain .hardsextube.com .public.extremetube.phncdn.com .redtubefiles.com .video.pornhub.phncdn.com
acl vc_dom dstdomain .public.keezmovies.com .public.keezmovies.phncdn.com .slutload-media.com .public.spankwire.com .xtube.com
acl vc_dom dstdomain .public.youporn.phncdn.com .xvideos.com .tube8.com .public.spankwire.phncdn.com .pornhub.com
refresh_pattern \.youtube\.com\/videoplayback\? 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern stream\.aol\.com\/(.*)/[a-zA-Z0-9]+\/(.*)\.(flv|mp4) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern videos\.5min\.com\/(.*)/[0-9_]+\.(mp4|flv) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern proxy[a-z0-9\-]?[a-z0-9]?[a-z0-9]?[a-z0-9]?\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern vid\.akm\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern vid\.ec\.dmcdn\.net\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern video\.(.*)\.fbcdn\.net\/(.*)/[0-9_]+\.(mp4|flv|avi|mkv|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.com\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.(.*)\.footprint\.net\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern c\.wrzuta\.pl\/wv[0-9]+\/[a-z0-9]+/[0-9]+/ 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern c\.wrzuta\.pl\/wa[0-9]+\/[a-z0-9]+ 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern vs[a-z0-9]?[a-z0-9]?[a-z0-9]?\.hardsextube\.com\/(.*)\/(.*)\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache override-expire override-lastmod ignore-private
acl vc_deny_url url_regex -i crossdomain.xml
acl vc_method method GET
acl vc_header req_header X-Requested-With -i videocache
url_rewrite_access deny !vc_method
url_rewrite_access deny vc_header
url_rewrite_access deny vc_deny_url
url_rewrite_access allow vc_dom
url_rewrite_access allow vc_url
url_rewrite_access allow vc_dom_r
redirector_bypass on
#These lines are needed.
acl this_machine src 127.0.0.1
http_access allow this_machine
strip_query_terms off
maximum_object_size 1000 MB
# --END-- videocache config for squid
# PORT and Transparent Option
http_port 8080 transparent
store_dir_select_algorithm round-robin
# Cache folders, change them according to your need, its an example of 500 GB
cache_dir aufs /cache1 512000 16 256
memory_replacement_policy heap LFUDA
cache_replacement_policy heap GDSF
# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off
# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache_log.log
cache_store_log /var/log/squid/store.log
# Block Ads [zaib]
#acl adsites dstdomain url_regex "/etc/squid/adslist.txt"
#http_access deny adsites
#deny_info http://192.168.6.1/psb.htm adsites
#I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 8.8.8.8
ftp_user anonymous@
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on
#ACL Section mylan myacl
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
#acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager all
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
#http_access allow all2
http_reply_access allow all
#http_reply_access allow all2
icp_access allow all
#==========================
# Administrative Parameters
#==========================
#============================================================$
# SNMP , if you want to generate graphs for SQUID via MRTG / zaib
#============================================================$
acl snmppublic snmp_community zaib
snmp_port 3401
snmp_access allow snmppublic all
snmp_access allow all
# I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr SYED_JAHAN_ZAIB
visible_hostname aacable.wordpress.com
unique_hostname aacable@hotmail.com
# Memory , If you have 4 GB of RAM in the server, then use 1 GB
cache_mem 1 GB
minimum_object_size 0 bytes
maximum_object_size_in_memory 1024 KB
# ZPH , to mark content which is in CACHE
tcp_outgoing_tos 0x30 all
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136
acl store_rewrite_list urlpath_regex \/(get_video|videoplayback\?id|videoplayback.*id)
acl store_rewrite_list urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar)\?
acl store_rewrite_list_domain url_regex ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_domain url_regex (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
acl store_rewrite_list_path urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|avc|zip|mp3|3gp|rar|on2|mar|exe)$
acl store_rewrite_list_domain_CDN url_regex \.rapidshare\.com.*\/[0-9]*\/.*\/[^\/]* ^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*) \.doubleclick\.net.*
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$ quantserve\.com
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.co(m|\.id)
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(rar|zip|flv|wm(a|v)|3gp|mp(4|3)|exe|msi|avi|(mp(e?g|a|e|1|2|3|4))|cab|exe)
acl dontrewrite url_regex redbot\.org \.php
acl getmethod method GET
##
refresh_pattern -i \.htm 120 50% 10080 reload-into-ims
refresh_pattern -i \.html 120 50% 10080 reload-into-ims
refresh_pattern ^http://*.facebook.com/* 720 100% 4320
refresh_pattern ^http://mail.yahoo.com/.* 720 100% 4320
refresh_pattern ^http://*.yahoo.*/.* 720 100% 4320
refresh_pattern ^http://*.yimg.*/.* 720 100% 4320
refresh_pattern ^http://*.gmail.*/.* 720 100% 4320
refresh_pattern ^http://*.google.*/.* 720 100% 4320
refresh_pattern ^http://*.kaskus.*/.* 720 100% 4320
refresh_pattern ^http://*.googlesyndication.*/.* 720 100% 4320
refresh_pattern ^http://*.plasa.*/.* 720 100% 4320
refresh_pattern ^http://*.telkom.*/.* 720 100% 4320
##
# 1 year = 525600 mins, 1 month = 43800 mins
refresh_pattern imeem.*\.flv 0 0% 0 override-lastmod override-expire
refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280 90% 161280 ignore-reload
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
#refresh_pattern -i (get_video\?|videoplayback\?id|videoplayback.*id||videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern \.(ico|video-stats) 10800 80% 10800 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth override-lastmod
refresh_pattern \.etology\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern galleries\.video(\?|sz) 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern brazzers\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern \.adtology\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
#refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 10800 20% 10800 ignore-no-cache ignore-private override-expire ignore-reload ignore-auth negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google 10800 80% 10800 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 10800 80% 10800 override-expire ignore-reload ignore-private
refresh_pattern ytimg\.com.*\.jpg 10800 80% 10800 override-expire ignore-reload
refresh_pattern images\.friendster\.com.*\.(png|gif) 10800 80% 10800 override-expire ignore-reload
refresh_pattern garena\.com 10800 80% 10800 override-expire reload-into-ims
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 10800 80% 10800 override-expire ignore-reload
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 10800 80% 10800 ignore-no-cache override-expire override-lastmod
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 10800 80% 10800 reload-into-ims override-expire ignore-private
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 10800 80% 10800 reload-into-ims ignore-no-cache ignore-reload override-expire
refresh_pattern ^http:\/\/www.onemanga.com.*\/ 10800 80% 10800 reload-into-ims ignore-no-cache ignore-reload override-expire
# ANTI VIRUS
refresh_pattern guru.avg.com/.*\.(bin) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern (avgate|avira).*(idx|gz)$ 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern kaspersky.*\.avc$ 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern kaspersky 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern update.nai.com/.*\.(gem|zip|mcs) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern windowsupdate.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern update.microsoft.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
#banner IIX
refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/img.ads.kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/openx.kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
#IIX DOWNLOAD
refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-auth
#All File
refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|divx|dvr-ms) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t)) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
#########################################################################
refresh_pattern (cgi-bin|\?) 0 0% 0
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern ^ftp: 10080 95% 10800 override-lastmod reload-into-ims
refresh_pattern . 180 95% 10800 override-lastmod reload-into-ims
global_internal_static off
retry_on_error on
buffered_logs on
read_ahead_gap 32 KB
#header_access Accept-Encoding deny all
client_persistent_connections off
server_persistent_connections on
half_closed_clients off
strip_query_terms off
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100
vary_ignore_expire on
reload_into_ims on
pipeline_prefetch on
read_timeout 30 minute
client_lifetime 6 hour
positive_dns_ttl 6 hour
pconn_timeout 15 second
request_timeout 1 minute
log_icp_queries off
ipcache_size 16384
ipcache_low 98
ipcache_high 99
log_fqdn off
fqdncache_size 16384
memory_pools off
forwarded_for on
client_db off
max_filedescriptors 8192
☻
☺
if you are using SQUID 3.1.19 (Ubuntu 12.4 default isntallation) then use following, pretty much same hmmm 🙂
SQUID 3.1.19 CONFIG FILE in /etc/squid3/squid.conf
# SQUID 3.1.19 CONFIG FILE WITH VIDEOCACHE PLUGIN FOR SQUID/VIDEOS [ Zaib]
# SQUID 3.1.19 WORKING CONFIG FILE [August 2013] with Videocache Support
# By - Syed Jahanzaib
# Email: aacable@hotmail.com
# Web : https://aacable.wordpress.com
cache_store_log /var/log/squid3/store.log
acl this_machine src 127.0.0.1 192.168.100.1 # Change this ip to match your's
http_access allow this_machine
# --BEGIN-- videocache config for squid
#
# PASTE THIS FILE AT THE TOP OF YOUR SQUID CONFIGURATION FILE
#
# (C) Copyright White Magnet Software Private Limited
# Company Website : http://whitemagnet.com/
# Product Website : http://cachevideos.com/
#
# Videocache Version : Trial
#
# This configuration is for Squid versions 3.x ZABBO
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 20
acl vc_deny_url url_regex -i \.blip\.tv\/(.*)filename \.hardsextube\.com\/videothumbs \.xtube\.com\/(.*)(Thumb|videowall)
acl vc_url url_regex -i \/youku\/[0-9A-Z]+\/[0-9A-Z\-]+\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg)
acl vc_url url_regex -i \/(.*)key=[a-z0-9]+(.*)\.flv
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/feeds\/api\/videos\/[0-9a-zA-Z_-]{11}\/
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/(videoplayback|get_video|watch|watch_popup|user_watch)\?
acl vc_url url_regex -i \.(youtube|youtube-nocookie)\.com\/(v|e|embed)\/[0-9a-zA-Z_-]{11}
acl vc_dom_r dstdom_regex -i msn\..*\.(com|net)
acl vc_dom_r dstdom_regex -i msnbc\..*\.(com|net)
acl vc_dom_r dstdom_regex -i video\..*\.fbcdn\.net
acl vc_dom_r dstdom_regex -i myspacecdn\..*\.footprint\.net
acl vc_dom dstdomain .stream.aol.com .videos.5min.com msn.com .blip.tv .vid.ec.dmcdn.net .break.com .vimeocdn.com
acl vc_dom dstdomain .cdn.turner.com .dailymotion.com .c.wrzuta.pl .v.imwx.com .mccont.com .myspacecdn.com
acl vc_dom dstdomain .hardsextube.com .public.extremetube.phncdn.com .redtubefiles.com .video.pornhub.phncdn.com
acl vc_dom dstdomain .public.keezmovies.com .public.keezmovies.phncdn.com .slutload-media.com .public.spankwire.com .xtube.com
acl vc_dom dstdomain .public.youporn.phncdn.com .xvideos.com .tube8.com .public.spankwire.phncdn.com .pornhub.com
refresh_pattern \.youtube\.com\/videoplayback\? 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern stream\.aol\.com\/(.*)/[a-zA-Z0-9]+\/(.*)\.(flv|mp4) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern videos\.5min\.com\/(.*)/[0-9_]+\.(mp4|flv) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern proxy[a-z0-9\-]?[a-z0-9]?[a-z0-9]?[a-z0-9]?\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern vid\.akm\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern vid\.ec\.dmcdn\.net\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern video\.(.*)\.fbcdn\.net\/(.*)/[0-9_]+\.(mp4|flv|avi|mkv|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.com\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.(.*)\.footprint\.net\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern c\.wrzuta\.pl\/wv[0-9]+\/[a-z0-9]+/[0-9]+/ 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern c\.wrzuta\.pl\/wa[0-9]+\/[a-z0-9]+ 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
refresh_pattern vs[a-z0-9]?[a-z0-9]?[a-z0-9]?\.hardsextube\.com\/(.*)\/(.*)\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 120 80% 180 ignore-no-cache ignore-no-store override-expire override-lastmod ignore-private
acl vc_deny_url url_regex -i crossdomain.xml
acl vc_method method GET
acl vc_header req_header X-Requested-With -i videocache
url_rewrite_access deny !vc_method
url_rewrite_access deny vc_header
url_rewrite_access deny vc_deny_url
url_rewrite_access allow vc_dom
url_rewrite_access allow vc_url
url_rewrite_access allow vc_dom_r
url_rewrite_bypass on
#These lines are needed.
acl this_machine src 127.0.0.1
http_access allow this_machine
strip_query_terms off
maximum_object_size 10 GB
# --END-- videocache config for squid
# SQUID 3.x CONFIG FILE
# PORT and Transparent Option
#http_port 8080
http_port 8080 transparent
icp_port 0
store_dir_select_algorithm round-robin
# Cache Direcotries on external Storage, for sure you should change it
# Each folder is of 1.5 TB in space
cache_dir aufs /cache-1/squid-cache-1 1500000 16 256
cache_dir aufs /cache-2/squid-cache-2 15000000 16 256
cache_dir aufs /cache-3/squid-cache-3 15000000 16 256
cache_dir aufs /cache-4/squid-cache-4 15000000 16 256
cache_dir aufs /cache-5/squid-cache-5 15000000 16 256
cache_dir aufs /cache-6/squid-cache-6 15000000 16 256
cache_dir aufs /cache-7/squid-cache-7 15000000 16 256
cache_dir aufs /cache-8/squid-cache-8 15000000 16 256
cache_dir aufs /cache-9/squid-cache-9 15000000 16 256
cache_dir aufs /cache-10/squid-cache-10 15000000 16 256
cache_dir aufs /cache-11/squid-cache-11 15000000 16 256
cache_dir aufs /cache-12/squid-cache-12 1500000 16 256
#cache_dir ufs /mnt/hdd2/cache2 200000 16 256 # If you have secondary HDD
memory_replacement_policy heap LFUDA
cache_replacement_policy heap GDSF
# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off
# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid3/access.log
cache_log /var/log/squid3/cache_log.log
cache_store_log /var/log/squid3/store.log
# Block Ads [zaib]
#acl adsites dstdomain url_regex "/etc/squid/adslist.txt"
#http_access deny adsites
#deny_info http://192.168.6.1/psb.htm adsites
#I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 8.8.8.8
ftp_user anonymous@
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on
#ACL Section mylan myacl
acl all src 0.0.0.0/0.0.0.0 # ALLOW ALL, you may want to change this /zaib
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager all
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
#http_access allow all2
http_reply_access allow all
#http_reply_access allow all2
icp_access allow all
#==========================
# Administrative Parameters
#==========================
#============================================================$
# SNMP , if you want to generate graphs for SQUID via MRTG
#============================================================$
#acl snmppublic snmp_community zaib
#snmp_port 3401
#snmp_access allow snmppublic all
#snmp_access allow all
# I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr SYED_JAHAN_ZAIB
visible_hostname aacable.wordpress.com
unique_hostname aacable@hotmail.com
# Memory, in this example, teh squid box have 32 B of RAM
cache_mem 8 GB # Change it as per your hardware config, you may want to lower it down
minimum_object_size 0 bytes
maximum_object_size_in_memory 5 MB # Change it as per your hardware config, you may want to lower it down if you have low ram
# 1 year = 525600 mins, 1 month = 43800 mins
refresh_pattern imeem.*\.flv 0 0% 0 override-lastmod override-expire
refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280 90% 161280 ignore-reload
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
#refresh_pattern -i (get_video\?|videoplayback\?id|videoplayback.*id||videodownload\?|\.flv?) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern \.(ico|video-stats) 10800 80% 10800 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth override-lastmod
refresh_pattern \.etology\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern galleries\.video(\?|sz) 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern brazzers\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
refresh_pattern \.adtology\? 10800 80% 10800 override-expire ignore-reload ignore-no-cache
#refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 10800 20% 10800 ignore-no-cache ignore-private override-expire ignore-reload ignore-auth negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google 10800 80% 10800 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth
refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 10800 80% 10800 override-expire ignore-reload ignore-private
refresh_pattern ytimg\.com.*\.jpg 10800 80% 10800 override-expire ignore-reload
refresh_pattern images\.friendster\.com.*\.(png|gif) 10800 80% 10800 override-expire ignore-reload
refresh_pattern garena\.com 10800 80% 10800 override-expire reload-into-ims
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 10800 80% 10800 override-expire ignore-reload
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 10800 80% 10800 ignore-no-cache override-expire override-lastmod
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 10800 80% 10800 reload-into-ims override-expire ignore-private
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 10800 80% 10800 reload-into-ims ignore-no-cache ignore-reload override-expire
refresh_pattern ^http:\/\/www.onemanga.com.*\/ 10800 80% 10800 reload-into-ims ignore-no-cache ignore-reload override-expire
# ANTI VIRUS
refresh_pattern guru.avg.com/.*\.(bin) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern (avgate|avira).*(idx|gz)$ 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern kaspersky.*\.avc$ 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern kaspersky 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern update.nai.com/.*\.(gem|zip|mcs) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern windowsupdate.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern update.microsoft.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe) 10800 80% 10800 ignore-no-cache ignore-reload reload-into-ims
#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 10800 80% 10800 ignore-reload override-expire ignore-no-cache
#banner IIX
refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/img.ads.kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/openx.kompas.com.*\/ 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache
#IIX DOWNLOAD
refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 10800 99999% 10800 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-auth
#All File
refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|divx|dvr-ms) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t)) 10800 80% 10800 ignore-no-cache ignore-private override-expire override-lastmod reload-into-ims
#########################################################################
refresh_pattern (cgi-bin|\?) 0 0% 0
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern ^ftp: 10080 95% 10800 override-lastmod reload-into-ims
refresh_pattern . 180 95% 10800 override-lastmod reload-into-ims
global_internal_static off
retry_on_error on
buffered_logs on
read_ahead_gap 32 KB
#header_access Accept-Encoding deny all
client_persistent_connections off
server_persistent_connections on
half_closed_clients off
strip_query_terms off
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100
vary_ignore_expire on
reload_into_ims on
pipeline_prefetch on
read_timeout 30 minute
client_lifetime 6 hour
positive_dns_ttl 6 hour
pconn_timeout 15 second
request_timeout 1 minute
log_icp_queries off
ipcache_size 16384
ipcache_low 98
ipcache_high 99
log_fqdn off
fqdncache_size 16384
memory_pools off
forwarded_for on
client_db off
max_filedescriptors 8192
# ZAIB
# ZPH for Squid 3.3.19
qos_flows local-hit=0x30
☺
↓
After videocache is installed, following is the fine tunned config file. Don’t forget to change the host ip (192.168.100.1) to match your own server ip
VIDEO CACHE CONFIG FILE in /etc/videocache.conf
[main] # file : /etc/videocache.conf # (C) Copyright White Magnet Software Private Limited # Company Website : http://whitemagnet.com/ # Product Website : http://cachevideos.com/ # WORKING SAMPLE FILE [JULY 2013] - Syed Jahanzaib # Email: aacable@hotmail.com # Web : https://aacable.wordpress.com # version : cxxxxxxb client_email = your_registered_emai_address@hotmail.com cache_host = 192.168.100.1 videocache_user = proxy # This option is used to define the cache directories where Videocache will store # cached videos. We can specify multiple caching directories separated by ‘|’. # Examples: # base_dir = /videocache/ # base_dir = /cache/videocache2/ | /hdd2/videocache/ | /home/videocache/ base_dir = /cache1/vc/ disk_avail_threshold = 15000 disk_cleanup_strategy = 1 enable_videocache = 1 offline_mode = 0 base_dir_selection = 2 enable_store_log_monitoring = 1 squid_store_log = /var/log/squid3/store.log this_proxy = 127.0.0.1:8080 hit_threshold = 1 max_cache_processes = 10 max_cache_speed = 0 max_cache_queue_size = 100000 cache_period = proxy = proxy_username = proxy_password = max_video_size = 0 min_video_size = 0 force_video_size = 1 scheduler_pidfile = /var/run/videocache.pid logdir = /var/log/videocache/ logformat = %tl %p %s %i %w %c %v %m %d scheduler_logformat = %tl %p %s %i %w %c %v %m %d cleaner_logformat = %tl %p %s %w %c %v %m %d timeformat = %d/%b/%Y:%H:%M:%S enable_videocache_log = 1 enable_scheduler_log = 1 enable_cleaner_log = 1 enable_trace_log = 1 logfile = videocache.log scheduler_logfile = scheduler.log cleaner_logfile = cleaner.log tracefile = trace.log max_logfile_size = 10 max_scheduler_logfile_size = 10 max_cleaner_logfile_size = 10 max_tracefile_size = 10 max_logfile_backups = 10 max_scheduler_logfile_backups = 5 max_cleaner_logfile_backups = 1 max_tracefile_backups = 1 enable_youtube_cache = 1 # This options determines if Videocache will cache different YouTube video # formats separately. Please select an appropriate algorithm from the listed below. # Available strategies: # 1 : (disabled) Don't check for YouTube video formats. Cache one of the formats # and serve it for requests for all kinds of formats. # 2 : (strict) Strictly check for YouTube formats and cache all formats separately. # Consumes maximum bandwidth. # 3 : (approximate) Check YouTube formats but with approximation. For example, # if a client asked for a video in 480p format and we already have 360p # format of the same video in cache, then serve 360p format and vice-versa. # Default : 3 enable_youtube_format_support = 3 enable_youtube_html5_videos = 1 enable_youtube_3d_videos = 1 enable_youtube_partial_caching = 1 max_youtube_video_quality = 720p min_youtube_views = 0 enable_aol_cache = 1 enable_bing_cache = 1 enable_bliptv_cache = 1 enable_breakcom_cache = 1 enable_cnn_cache = 1 enable_dailymotion_cache = 1 enable_facebook_cache = 1 enable_megavideo_cache = 1 enable_metacafe_cache = 1 enable_myspace_cache = 1 enable_vimeo_cache = 1 enable_weather_cache = 1 enable_wrzuta_cache = 1 enable_youku_cache = 1 enable_extremetube_cache = 1 enable_hardsextube_cache = 1 enable_keezmovies_cache = 1 enable_pornhub_cache = 1 enable_redtube_cache = 1 enable_slutload_cache = 1 enable_spankwire_cache = 1 enable_tube8_cache = 1 enable_xhamster_cache = 1 enable_xtube_cache = 1 enable_xvideos_cache = 1 enable_youporn_cache = 1 skip_apache_conf = 0 apache_conf_dir = /etc/apache2/conf.d/ hide_cache_dirs = 1 rpc_host = 127.0.0.1 rpc_port = 9100
☺
To restart Videocache and related services, use following commands
vc-update
service apache2 restart
vc-scheduler -s restart
service squid3 restart
To view Video cache Logs in live for info and troubleshooting
tail -f /var/log/videocache/videocache.log
tail -f /var/log/videocache/scheduler.log
tail -f /var/log/squid3/access.log
Sample image of logs at live server
VC-CACHE
☺
☼☼☼☼☼☼☼☼
Regard’s
Syed Jahanzaib
Syed Jahanzaib - Personal Blog to Share Knowledge ! 