Following is an automated script to install LUSCA r14942 for UBUNTU with aggressive content caching support including some video web sites like YOUTUBE and few others as described in my other article @
https://aacable.wordpress.com/2014/04/21/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/
I will add more n more functions as soon as I get some free time, like configurable options via choice menu like cache size, mem, and other variables.
SCRIPT FUNCTIONS . . .
This script will do the following
- Update Ubuntu
- Install some components required for Compilation of Lusca/Squid package
- Backup squid.conf if already in /etc/squid.conf with squid.conf.old, stop any running squid instance
- Download LUSCA r14942 source package to /temp folder and compile it
- Download squid.conf and storeurl.pl from the internet and place them in /etc/squid.conf
- Create cache directory like in /cache-1 and default cache size is 5 GB
- add squid in /etc/rc.local so it may start auto upon system reboot
Note: You should modify all options in /etc/squid.conf after installation , like cache_dir, cache_mem and others as per your network and hardware specifications.
REQUIREMENTS . . .
1- Fresh Installation of UBUNTU OS and Configure Internet Access
2- root access to execute script
3- REMOVE ANY KIND OF PREVIOUSLY INSTALLED SQUID INSTALLATION IF ANY
4- Upload or create script in any folder of Ubuntu box,
or create new script with following commands
mkdir /temp
cd /temp
touch lusca_install.sh
chmod +x lusca_install.sh
nano lusca_install.sh
and paste the following code . . .
#!/bin/bash # Version 1.0 / 2nd July, 2014 # LUSCA r14942 Automated Installation Script for Ubuntu flavor / jz # Syed Jahanzaib / aacable @ hotmail.com / https://aacable.wordpress.com # Setting Variables . . . [JZ] # You can change the URL if default url is not accessible in some cases. #URL=http://aacable.rdo.pt/files/linux_related/lusca URL=http://wifismartzone.com/files/linux_related/lusca SQUID_DIR="/etc/squid" CACHE_DIR="/cache-1" pid=`pidof squid` osver=`cat /etc/issue |awk '{print $1}'` squidlabel="LUSCA_HEAD-r14942" # Colors Config . . . [[ JZ . . . ]] ESC_SEQ="\x1b[" COL_RESET=$ESC_SEQ"39;49;00m" COL_RED=$ESC_SEQ"31;01m" COL_GREEN=$ESC_SEQ"32;01m" # OS checkup for UBUNTU echo -e "$COL_GREEN Lusca r14942 Automated Installation Script ver 1.0 for Ubuntu . . .$COL_RESET" echo -e "$COL_GREEN Checking OS version, as it must be Ubuntu in order to Continue . . .$COL_RESET" if [[ $osver == Ubuntu ]]; then echo echo -e "$COL_GREEN Ubuntu is installed with following information fetched. $COL_RESET" lsb_release -a sleep 3 else echo -e "$COL_RED Sorry, it seems your Linux Distribution is not UBUNTU . Exiting ...$COL_RESET" exit 1 fi # Make sure only root can run our script / Checking if user is root, otherwise exit with error [[Jz]] echo echo -e "$COL_GREEN Verifying if you are logged in with root privileges . . .$COL_RESET" 1>&2 FILE="/tmp/out.$$" GREP="/bin/grep" if [ "$(id -u)" != "0" ]; then echo echo -e "$COL_RED This script must be run as root, switch to root now . . .$COL_RESET" 1>&2 exit 1 fi # Clearing previous download if any in /tmp folder echo echo -e "$COL_GREEN Clearing previous downloads if any in /tmp folder to avoid duplication$COL_RESET" sleep 3 rm -fr /tmp/squid.conf rm -fr /tmp/storeurl.txt rm -fr /tmp/storeurl.pl rm -fr /tmp/LUSCA_HEAD-r14942* # Checking IF $URL is accessible m if YES then continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]] echo echo -e "$COL_GREEN Checking if $URL is accessible in order to proceed further. . .!! $COL_RESET" cd /tmp wget -q $URL/squid.conf { if [ ! -f /tmp/squid.conf ]; then echo echo -e "$COL_RED ERROR: Unable to contact $URL, or possibly internet is not working or your IP is in black list at destination server !! $COL_RESET" echo -e "$COL_RED ERROR: Please check manual if $URL is accessible or not or if it have required files, JZ !! $COL_RESET" exit 0 fi } rm -fr /tmp/squid.conf sleep 6 # Moving further . . . clear echo -e "$COL_GREEN You are logged in with root ID, Ok to proceed further . . .!! $COL_RESET" echo ################################################################## [zaib] echo echo -e "$COL_GREEN Updating Ubuntu first . . . !! $COL_RESET" apt-get update echo echo echo -e "$COL_GREEN Installing required components . . . !! $COL_RESET" sleep 3 apt-get install -y gcc build-essential libstdc++6 unzip bzip2 sharutils ccze libzip-dev automake1.9 libfile-readbackwards-perl dnsmasq # Clearing OLD data files . . . { if [ -f $SQUID_DIR/squid.conf ]; then echo echo echo -e "$COL_RED Previous SQUID configuration file found in $SQUID_DIR ! renaming it for backup purpose . . . $COL_RESET" mv $SQUID_DIR/squid.conf $SQUID_DIR/squid.conf.old else echo echo echo -e "$COL_GREEN No Previous Squid configuration have been found in $SQUID_DIR. Proceeding further $COL_RESET" fi } # Checking SQUID status if its already running - check by PID if [ "$pid" == "" ]; then echo echo echo -e "$COL_GREEN No SQUID instance found in memory , so it seems we are good to GO !!! $COL_RESET" else echo echo -e "$COL_RED SQUID is already running, probably you have some previous copy of SQUID installation, Better to stop and remove all previous squid installation !! $COL_RESET" echo echo -e "$COL_RED KILLING PREVIOUS SQUID INSTANCE by killall -9 squid command !! $COL_RESET" killall -9 squid sleep 3 fi # Downloading Squid source package [zaib] echo echo echo -e "$COL_GREEN Downloading SQUID source package in /tmp folder. . . !! $COL_RESET" sleep 3 # Checking if /tmp folder is previously present or not . . . { if [ ! -d "/tmp" ]; then echo echo echo -e "$COL_RED /tmp folder not found, Creating it so all downloads will be placed here . . . $COL_RESET" mkdir /tmp else echo echo -e "$COL_GREEN /tmp folder is already present , so no need to create it, Proceeding further . . . $COL_RESET" fi } cd /tmp # Checking IF LUSCA_HEAD-r14942.tar.gz installation file have been ALREADY downloaded in /tmp to avoid duplication! [[ JZ .. . .]] { if [ -f /tmp/LUSCA_HEAD-r14942.tar.gz ]; then rm -fr /tmp/LUSCA_HEAD-r14942.tar.gz fi } wget -c http://wifismartzone.com/files/linux_related/lusca/LUSCA_HEAD-r14942.tar.gz # Checking IF LUSCA_HEAD-r14942 installation file have been downloaded properly. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]] { if [ ! -f /tmp/LUSCA_HEAD-r14942.tar.gz ]; then echo echo echo -e "$COL_RED ERROR: SQUID source code package File could not be download or not found in /tmp/ !! $COL_RESET" exit 0 fi } echo echo echo -e "$COL_GREEN Extracting Squid from tar archive. . . !! $COL_RESET" sleep 3 tar zxvf LUSCA_HEAD-r14942.tar.gz cd LUSCA_HEAD-r14942/ mkdir /etc/squid echo -e "$COL_GREEN Executing $squidlabel Compiler [jz] . . . !! $COL_RESET" echo cd /tmp/LUSCA_HEAD-r14942 ./configure --prefix=/usr --exec_prefix=/usr --bindir=/usr/sbin --sbindir=/usr/sbin --libexecdir=/usr/lib/squid --sysconfdir=/etc/squid --localstatedir=/var/spool/squid --datadir=/usr/share/squid --enable-async-io=24 --with-aufs-threads=24 --with-pthreads --enable-storeio=aufs --enable-linux-netfilter --enable-arp-acl --enable-epoll --enable-removal-policies=heap --with-aio --with-dl --enable-snmp --enable-delay-pools --enable-htcp --enable-cache-digests --disable-unlinkd --enable-large-cache-files --with-large-files --enable-err-languages=English --enable-default-err-language=English --enable-referer-log --with-maxfd=65536 echo echo -e "$COL_GREEN Executing MAKE and MAKE INSTALL commands . . . !! $COL_RESET" sleep 3 make make install echo echo echo -e "$COL_GREEN Creating SQUID LOGS folder and assiging permissions . . . !! $COL_RESET" sleep 3 # Checking if log folder is previously present or not . . . { if [ -d "/var/log/squid" ]; then echo echo echo -e "$COL_GREEN LOGS folder found. No need to create, proceeding Further . . . $COL_RESET" else echo echo echo -e "$COL_GREEN Creating LOG Folder in /var/log/squid and setting permissions accordingly (to user proxy) $COL_RESET" mkdir /var/log/squid fi } chown proxy:proxy /var/log/squid ## ** DOWNLOAD SQUID.CONF echo echo echo -e "$COL_GREEN Downloading SQUID.CONF file from $URL and copy it to $SQUID_DIR. . . !! $COL_RESET" sleep 3 # Checking IF SQUID.CONF File have been ALREADY downloaded in /tmp to avoid duplication! [[ JZ .. . .]] { if [ -f /tmp/squid.conf ]; then rm -fr /tmp/squid.conf fi } cd /tmp wget $URL/squid.conf # Checking IF SQUID.CONF file have been downloaded. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]] { if [ ! -f /tmp/squid.conf ]; then echo echo echo -e "$COL_RED ERROR: SQUID.CONF File could not be download or not found in /tmp/ !! $COL_RESET" exit 0 fi } cp -fr squid.conf $SQUID_DIR ## ** DOWNLOAD SQUID.CONF echo echo echo -e "$COL_GREEN Downloading STOREURL.PL file from $URL and copy it to $SQUID_DIR. . . !! $COL_RESET" sleep 3 cd /tmp { if [ -f /tmp/storeurl.txt ]; then rm -fr /tmp/storeurl.txt fi } wget $URL/storeurl.txt { if [ -f /tmp/storeurl.pl ]; then rm -fr /tmp/storeurl.pl fi } mv storeurl.txt storeurl.pl # Checking IF STOREURL.PL file have been downloaded. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]] { if [ ! -f /tmp/storeurl.pl ]; then echo echo echo -e "$COL_RED ERROR: STOREURL.PL File could not be download or not found in /tmp/ !! $COL_RESET" exit 0 fi } cp -fr storeurl.pl $SQUID_DIR echo echo echo -e "$COL_GREEN Setting EXECUTE permission for storeurl.pl . . . !! $COL_RESET" chmod +x $SQUID_DIR/storeurl.pl # Creating CACHE folders echo echo echo -e "$COL_GREEN Creating CACHE directory in $CACHE_DIR , in this example,I used 5GB for cache for test ,Adjust it accordingly . . . !! $COL_RESET" sleep 3 # Checking if /cache-1 folder exist . . . { if [ ! -d "$CACHE_DIR" ]; then echo echo echo -e "$COL_GREEN Creating cache folder in $CACHE_DIR , Default size is 5GB, you should set it accordingly to your requirements . . . $COL_RESET" mkdir $CACHE_DIR chown proxy:proxy $CACHE_DIR chmod 777 -R $CACHE_DIR squid -z else echo echo -e "$COL_RED $CACHE_DIR folder already exists , Clearing it before proceeding. . . $COL_RESET" rm -fr $CACHE_DIR/* chown proxy:proxy $CACHE_DIR echo -e "$COL_GREEN $CACHE_DIR Initializing Cache Directories as per the config . . . $COL_RESET" echo squid -z chmod 777 -R $CACHE_DIR fi } echo echo echo -e "$COL_GREEN Adding squid in /etc/rc.local for auto startup . . . !! $COL_RESET" sed -i '/exit/d' /etc/rc.local sed -i '/[/usr\/sbin\/squid]/d' /etc/rc.local echo /usr/sbin/squid >> /etc/rc.local echo exit 0 >> /etc/rc.local echo echo -e "$COL_GREEN Starting SQUID (and adding 10 seconds Pause for proper initialization). . . !! $COL_RESET" squid sleep 5 # Checking SQUID status via PID [zaib] #if [ "$pid" == "" ]; then #echo #echo -e "$COL_RED ERROR: UNABLE to start SQUID, try to run with -d1N syntax and see where its showing error !! $COL_RESET" #else ps aux |grep squid echo echo -e "$COL_GREEN $squidlabel is Running OK with PID number "$pid", no further action required, EXITING . . .$COL_RESET" echo echo To view squid web access activity log, use command echo -e "$COL_GREEN tail -f /var/log/squid/access.log $COL_RESET" echo OR echo -e "$COL_GREEN tail -f /var/log/squid/access.log |ccze $COL_RESET" echo echo -e "$COL_GREEN Regard's / Syed Jahanzaib . . . !! $COL_RESET" echo
ALL DONE.
now execute the script by running
/temp/lusca_install.sh
It will start installation and will show you the progress with all the action its doing [in colored rows, RED color shows error, Green Color shows Ok/INFO].
TIP:
To start SQUID Server in Debug mode, to check any errors, use following
squid -d1n
if squid is successfully started , you can see its process via PS command
ps aux |grep squid
as showed in the image below …
thank you… good post I will try now
LikeLike
Comment by ven — July 2, 2014 @ 1:06 PM
Dear sir;;
How to block https traffic (facebook.com) by using squid with transparent proxy…
Regards Assad
LikeLike
Comment by Asad — July 2, 2014 @ 6:33 PM
not working sir…
LikeLike
Comment by tamsil — July 2, 2014 @ 6:54 PM
nice share. This should be fresh ubuntu? I’ll try later when I had to reinstall.
thanks
LikeLike
Comment by papaeky — July 3, 2014 @ 12:42 AM
is this Lusca proxy working with youtube https:// traffic recently youtube changed their traffic to https://
LikeLike
Comment by ven — July 3, 2014 @ 1:28 PM
no ….
LikeLike
Comment by smn4all — July 6, 2014 @ 8:34 PM
Thanks ……….sir ALLAH aap ko jaza e khair dey………!
We Check this Script…..
LikeLike
Comment by Salman — July 6, 2014 @ 2:08 AM
Sir…………….ubuntu 64 bit OS Architecture ” section ” & compiler for 64 bit
not mention in this article……..ALLAH aap ko jaza e khair dey.
LikeLike
Comment by Salman — July 7, 2014 @ 6:40 AM
Dear Zaib thanks for your top POST
I want to cache youtube https:// traffic
if this possible with ssl?
LikeLike
Comment by Haroon Rashidi — July 8, 2014 @ 3:04 AM
Please read this.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — July 9, 2014 @ 9:35 AM
Sir…Is this Script also work on ubuntu 64bit & 64bit OS Architecture?
Because you don’t write in the script ( 64 bit compiler of squid )
plz………explain ALLAH aap ko JAZA E KHAIR dey…..
LikeLike
Comment by Salman — July 10, 2014 @ 6:22 AM
Yes this works with 32 bit as well as 64bit too.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — July 10, 2014 @ 9:47 AM
Hi Syed, if i do LUSCHA with youtube for mikrotik. And want to do this script when computer restart. do i need to delete all the configs from LUSCHA and youtube? Thank you
LikeLike
Comment by alex — July 10, 2014 @ 7:52 AM
You dont need to do any thing further.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — July 10, 2014 @ 9:45 AM
Hi Syed,
I get this error
ERROR: Unable to contact http://wifismartzone.com/files/linux_related/lusca, or possibly internet is not working or your IP is in black list at destination server !!
ERROR: Please check manual if http://wifismartzone.com/files/linux_related/lusca is accessible or not or if it have required files, JZ !!
Any possible way to resolve it.
Thanks
LikeLike
Comment by arnoldrodrigues — August 1, 2014 @ 7:16 PM
there is an alternate URL present there in the script. I will update the second URL link it tomorrow and will let u know
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 3, 2014 @ 3:59 PM
Hi, I have installed the same on Ubuntu 12.04 with hardware config of I3-3.2Ghz with 8 GB RAM and 1TD HDD, in which 4GB allocated to squid and 500GB Storage. The problem i face is after 1 day of working, it slows down or doesn’t [the browsing part]. We need to restart the lusca/squid in order to work again any solution for this. The Bandwidth is around 40mb in peak time and average 20-25mbps .
LikeLike
Comment by Sankar S — August 7, 2014 @ 10:52 AM
hello mr syed.. i have a problem when the squid startup it take about 2 to 3 mb for upload and download without redirect any website to squid…can you help me…thanks you
LikeLike
Comment by ismail — August 11, 2014 @ 2:14 PM
hmm there is no such issue in default squid config. You have to check at your router level that what data is being pulled/pushed from the squid. probably someone is downloading OR probably Linux is updating something. You should wait few hours , maybe it will end itself.
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — August 14, 2014 @ 4:02 PM
Hi Syed. First of all, many thanks for your blog!! it has a really helpful for my job!! I would like to ask you about Lusca is not caching youtube content anymore, i have tried on Ubuntu 12.04LTS and 14.04LTS with the same result: TCP_HIT: MISS, I have tried other tools/plugins as Squid Video & Music Booster but nothing. Any piece of advice?
LikeLike
Comment by IgorLVG — August 15, 2014 @ 3:52 AM
hi syed..its not working to caching youtube and facebook videos and applications..but working in other videos sites.. can i do anything to cache videos from youtube and facebook videos?
LikeLike
Comment by ismail — August 18, 2014 @ 5:58 PM
can i add android application to be cached ?
LikeLike
Comment by ismail — August 18, 2014 @ 6:30 PM
Excellent article, congratulations for the work
which rules integration with mikrotik version 6.x with cache full?
Marti Souza
Brazil, Amazonas, Teffe
LikeLike
Comment by Martinelli Souza — August 20, 2014 @ 12:42 AM
thanks for your blogs sayd …and please tell me about tips that cache youtube and facebook sites
cos
all users tell me that face book more slowly!!!
please tell me about that problem
LikeLike
Comment by ahmed — August 29, 2014 @ 5:21 PM
youtube no caching because is https. whats is the solution? thanks
LikeLike
Comment by carlos pirela — September 7, 2014 @ 10:57 AM
Dear zaib please tell us how to cache youtube https and another https ?????
please help us
thank you
LikeLike
Comment by haroonnimroozy — September 8, 2014 @ 7:55 PM
dear sir please help what is dscp this script ? for microtik router
LikeLike
Comment by lalu — September 10, 2014 @ 11:50 PM
i have problem : clientnatlookup nf getsockopt(so_original_dst) failed (92) protocol not available. how can solve?
LikeLike
Comment by syarifudin — October 23, 2014 @ 1:28 PM
Dear Sir, Please let me know that when installing ubuntu do we have to make partitions or it will make it automatically. if we do have to make then please do let me know how and we can make and which partitions. thanks
LikeLike
Comment by Roger — October 25, 2014 @ 1:28 PM
Thanks sir it is working on latest ubuntu 14.04 LTS server, but how can I restart the lusca service without restarting the whole machine? I’ve tried sudo service lusca restart and sudo service squid restart but its not working or the service is not recognized.
Thanks in advance sir,
LikeLike
Comment by Norman Villas — November 15, 2014 @ 12:17 PM
Dear Sir, please tell me how to REMOVE ANY KIND OF PREVIOUSLY INSTALLED SQUID INSTALLATION COMPLETELY.
Thanks b4
dedmax
LikeLike
Comment by dedmax — November 17, 2014 @ 7:17 PM
alhamdulillah! it’s working! but i need help…
my network is like this :
ISP 1 (192.168.1.1) —-|
MIKROTIK —————— LAN Client (192.168.5.0/24) GW 192.168.5.254
ISP 2 (192.168.2.2) ——| |
|
Ubuntu server 12.04 LTS + LUSCA r14942 (192.168.4.2) GW 192.168.4.1
eth01 – WAN (ISP1)
eth02 – WAN (ISP2)
eth03 – disabled
eth04 – Ubuntu + LUSCA
eth05 – LAN
I can’t use your tutorial https://aacable.wordpress.com/2011/07/21/mikrotik-howto-redirect-http-traffic-to-squid-with-original-source-client-ip/
if i’m using DSTNAT..it works fine…but i have to masquerade all interface i had. can you help me?
oh by the way, i’m still using mikrotik 5.xx but theres an error on LUsca says
unrecognized : ‘qos_flows’ and after i used
tcp_outgoing_tos 0x30 “localnet” [ACL name for my local network or 192.168.5.0/24]
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136
another error says : unrecognized tcp_outgoing_tos
pelase, can you help me?
mail me at gonzhilaz@gmail.com
Thank you!
LikeLike
Comment by Bayu — November 21, 2014 @ 6:11 PM
Hi!
when running squid proxy mikrotik graphs are stopped. not changed. please give me solution
LikeLike
Comment by PEDDAREDDY — December 4, 2014 @ 8:38 AM
Thanks ……….sir ALLAH aap ko jaza e khair dey………!
We Check this Script nyc work but sir 1kia lusca may speedtest.ptcl.net ya speedtest.net cache nahi hoti aksar form per frnds ne speedtest ki snaps di hoe hoti hain but mere pas speedtest cache nahi ho rahi…
please tell me kia ye ho sekti hai is me……..???
LikeLike
Comment by Muhammad Furqan Khan — February 5, 2015 @ 5:39 PM
You shouldn’t be caching speedtest.net like sites, they are dynamic and made to give you fresh result on every check. Why you want to get FALSE results with it ?
LikeLike
Comment by Syed Jahanzaib / Pinochio~:) — February 6, 2015 @ 8:13 AM
what is the command to start the squid service in ubuntu 10.04??? and my all configuration is done but system shows “unrecognized bash command” like this please tell what is actual command to start and stop the service of squid???
/etc/init.d in the path squid not here but all other are here.
please solve me problem as soon as. I shall be thankful to you Sir
LikeLike
Comment by Ahamd Salman — May 12, 2015 @ 3:48 PM
Dear Mr.Zaib
I have installed the same on Ubuntu 12.04 with hardware config of Xeon-2.8Ghz quad core with dual cpu with 20 GB RAM and 1.5TB HDD, in which 4GB allocated to squid and 500GB Storage. The problem i face is after 1 or 2 day of working, it slows down or doesn’t [the browsing part]. We need to restart the lusca/squid in order to work again any solution for this
LikeLike
Comment by haroonnimroozy — June 6, 2015 @ 5:18 PM