Syed Jahanzaib – Personal Blog to Share Knowledge !

July 2, 2014

LUSCA Automated Install Scriptt

Filed under: Linux Related — Tags: , — Syed Jahanzaib / Pinochio~:) @ 12:22 PM

lusca_image


 

Following is an automated script to install LUSCA r14942 for UBUNTU with aggressive content caching support including some video web sites like YOUTUBE and few others as described in my other article @
https://aacable.wordpress.com/2014/04/21/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/

I will add more n more functions as soon as I get some free time, like configurable options via choice menu like cache size, mem, and other variables.


 

SCRIPT FUNCTIONS . . . 

This script will do the following

  • Update Ubuntu
  • Install some components required for Compilation of Lusca/Squid package
  • Backup squid.conf if already in /etc/squid.conf with squid.conf.old, stop any running squid instance
  • Download LUSCA r14942 source package to /temp folder and compile it
  • Download squid.conf and storeurl.pl from the internet and place them in /etc/squid.conf
  • Create cache directory like in /cache-1 and default cache size is 5 GB
  • add squid in /etc/rc.local so it may start auto upon system reboot

Note: You should modify all options in /etc/squid.conf after installation , like cache_dir, cache_mem and others as per your network and hardware specifications.


 

REQUIREMENTS . . .

1- Fresh Installation of UBUNTU OS and Configure Internet Access
2- root access to execute script
3- REMOVE ANY KIND OF PREVIOUSLY INSTALLED SQUID INSTALLATION IF ANY
4- Upload or create script in any folder of Ubuntu box,

or create new script with following commands

mkdir /temp
cd /temp
touch lusca_install.sh
chmod +x lusca_install.sh

nano lusca_install.sh

and paste the following code . . .


 


#!/bin/bash
# Version 1.0 / 2nd July, 2014
# LUSCA r14942 Automated Installation Script for Ubuntu flavor / jz
# Syed Jahanzaib / aacable @ hotmail.com  / https://aacable.wordpress.com

# Setting Variables . . . [JZ]
# You can change the URL if default url is not accessible in some cases. 
#URL=http://aacable.rdo.pt/files/linux_related/lusca
URL=http://wifismartzone.com/files/linux_related/lusca
SQUID_DIR="/etc/squid"
CACHE_DIR="/cache-1"
pid=`pidof squid`
osver=`cat /etc/issue |awk '{print $1}'`
squidlabel="LUSCA_HEAD-r14942"

# Colors Config  . . . [[ JZ . . . ]]
ESC_SEQ="\x1b["
COL_RESET=$ESC_SEQ"39;49;00m"
COL_RED=$ESC_SEQ"31;01m"
COL_GREEN=$ESC_SEQ"32;01m"

# OS checkup for UBUNTU
echo -e "$COL_GREEN Lusca r14942 Automated Installation Script ver 1.0 for Ubuntu . . .$COL_RESET"
echo -e "$COL_GREEN Checking OS version, as it must be Ubuntu in order to Continue . . .$COL_RESET"
if [[ $osver == Ubuntu ]]; then
echo
echo -e "$COL_GREEN Ubuntu is installed with following information fetched. $COL_RESET"
lsb_release -a
sleep 3
else
echo -e "$COL_RED Sorry, it seems your Linux Distribution is not UBUNTU . Exiting ...$COL_RESET"
exit 1
fi

# Make sure only root can run our script / Checking if user is root, otherwise exit with error [[Jz]]
echo
echo -e "$COL_GREEN Verifying if you are logged in with root privileges  . . .$COL_RESET" 1>&2
FILE="/tmp/out.$$"
GREP="/bin/grep"
if [ "$(id -u)" != "0" ]; then
echo
echo -e "$COL_RED This script must be run as root, switch to root now . . .$COL_RESET" 1>&2
exit 1
fi

# Clearing previous download if any in /tmp folder
echo
echo -e "$COL_GREEN Clearing previous downloads if any in /tmp folder to avoid duplication$COL_RESET"
sleep 3

rm -fr /tmp/squid.conf
rm -fr /tmp/storeurl.txt
rm -fr /tmp/storeurl.pl
rm -fr /tmp/LUSCA_HEAD-r14942*

# Checking IF $URL is accessible m if YES then continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]]
echo
echo -e "$COL_GREEN Checking if $URL is accessible in order to proceed further. . .!! $COL_RESET"
cd /tmp
wget -q $URL/squid.conf
{
if [ ! -f /tmp/squid.conf ]; then
echo
echo -e "$COL_RED ERROR: Unable to contact $URL, or possibly internet is not working or your IP is in black list at destination server  !! $COL_RESET"
echo -e "$COL_RED ERROR: Please check manual if $URL is accessible or not or if it have required files, JZ  !! $COL_RESET"
exit 0
fi
}
rm -fr /tmp/squid.conf
sleep 6
# Moving further . . .

clear
echo -e "$COL_GREEN You are logged in with root ID, Ok to proceed further . . .!! $COL_RESET"
echo

################################################################## [zaib]
echo
echo -e "$COL_GREEN Updating Ubuntu first . . . !! $COL_RESET"
apt-get update
echo
echo
echo -e "$COL_GREEN Installing required components . . . !! $COL_RESET"
sleep 3
apt-get install  -y gcc  build-essential   libstdc++6   unzip    bzip2   sharutils  ccze  libzip-dev  automake1.9  libfile-readbackwards-perl  dnsmasq

# Clearing OLD data files . . .
{
if [ -f $SQUID_DIR/squid.conf ]; then
echo
echo
echo -e "$COL_RED Previous SQUID configuration file found in $SQUID_DIR ! renaming it for backup purpose . . . $COL_RESET"
mv $SQUID_DIR/squid.conf $SQUID_DIR/squid.conf.old
else
echo
echo
echo -e "$COL_GREEN No Previous Squid configuration have been found in $SQUID_DIR. Proceeding further $COL_RESET"
fi
}

# Checking SQUID status if its already running - check by PID
if [ "$pid" == "" ]; then
echo
echo
echo -e "$COL_GREEN No SQUID instance found in memory , so it seems we are good to GO !!! $COL_RESET"
else
echo
echo -e "$COL_RED SQUID is already running, probably you have some previous copy of SQUID installation, Better to stop and remove all previous squid installation !! $COL_RESET"
echo
echo -e "$COL_RED KILLING PREVIOUS SQUID INSTANCE by killall -9 squid command  !! $COL_RESET"
killall -9 squid
sleep 3
fi

# Downloading Squid source package [zaib]
echo
echo
echo -e "$COL_GREEN Downloading SQUID source package in /tmp folder. . . !! $COL_RESET"
sleep 3

# Checking if /tmp folder is previously present or not . . .
{
if [ ! -d "/tmp" ]; then
echo
echo
echo -e "$COL_RED /tmp folder not found, Creating it so all downloads will be placed here  . . . $COL_RESET"
mkdir /tmp
else
echo
echo -e "$COL_GREEN /tmp folder is already present , so no need to create it, Proceeding further . . . $COL_RESET"
fi
}

cd /tmp

# Checking IF LUSCA_HEAD-r14942.tar.gz  installation file have been ALREADY downloaded in /tmp to avoid duplication! [[ JZ .. . .]]
{
if [ -f /tmp/LUSCA_HEAD-r14942.tar.gz ]; then
rm -fr /tmp/LUSCA_HEAD-r14942.tar.gz
fi
}

wget -c http://wifismartzone.com/files/linux_related/lusca/LUSCA_HEAD-r14942.tar.gz

# Checking IF LUSCA_HEAD-r14942 installation file have been downloaded properly. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]]
{
if [ ! -f /tmp/LUSCA_HEAD-r14942.tar.gz ]; then
echo
echo

echo -e "$COL_RED ERROR: SQUID source code package File could not be download or not found in /tmp/ !! $COL_RESET"
exit 0
fi
}
echo
echo

echo -e "$COL_GREEN Extracting Squid from tar archive. . . !! $COL_RESET"
sleep 3
tar zxvf LUSCA_HEAD-r14942.tar.gz
cd LUSCA_HEAD-r14942/
mkdir /etc/squid

echo -e "$COL_GREEN Executing $squidlabel Compiler [jz] . . . !! $COL_RESET"
echo
cd /tmp/LUSCA_HEAD-r14942
./configure --prefix=/usr --exec_prefix=/usr --bindir=/usr/sbin --sbindir=/usr/sbin --libexecdir=/usr/lib/squid --sysconfdir=/etc/squid --localstatedir=/var/spool/squid --datadir=/usr/share/squid --enable-async-io=24 --with-aufs-threads=24 --with-pthreads --enable-storeio=aufs --enable-linux-netfilter --enable-arp-acl --enable-epoll --enable-removal-policies=heap --with-aio --with-dl --enable-snmp --enable-delay-pools --enable-htcp --enable-cache-digests --disable-unlinkd --enable-large-cache-files --with-large-files --enable-err-languages=English --enable-default-err-language=English --enable-referer-log --with-maxfd=65536
echo
echo -e "$COL_GREEN Executing MAKE and MAKE INSTALL commands . . . !! $COL_RESET"
sleep 3
make
make install
echo
echo
echo -e "$COL_GREEN Creating SQUID LOGS folder and assiging permissions . . . !! $COL_RESET"
sleep 3

# Checking if log folder is previously present or not . . .
{
if [ -d "/var/log/squid" ]; then
echo
echo
echo -e "$COL_GREEN LOGS folder found. No need to create, proceeding Further . . . $COL_RESET"
else
echo
echo
echo -e "$COL_GREEN Creating LOG Folder in /var/log/squid and setting permissions accordingly (to user proxy) $COL_RESET"
mkdir /var/log/squid
fi
}
chown proxy:proxy /var/log/squid
## ** DOWNLOAD SQUID.CONF
echo
echo
echo -e "$COL_GREEN Downloading SQUID.CONF file from $URL and copy it to $SQUID_DIR. . . !! $COL_RESET"
sleep 3

# Checking IF SQUID.CONF File have been ALREADY downloaded in /tmp to avoid duplication! [[ JZ .. . .]]
{
if [ -f /tmp/squid.conf ]; then
rm -fr /tmp/squid.conf
fi
}

cd /tmp
wget $URL/squid.conf

# Checking IF SQUID.CONF file have been downloaded. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]]
{
if [ ! -f /tmp/squid.conf ]; then
echo
echo
echo -e "$COL_RED ERROR: SQUID.CONF File could not be download or not found in /tmp/ !! $COL_RESET"
exit 0
fi
}
cp -fr squid.conf $SQUID_DIR

## ** DOWNLOAD SQUID.CONF
echo
echo
echo -e "$COL_GREEN Downloading STOREURL.PL file from $URL and copy it to $SQUID_DIR. . . !! $COL_RESET"
sleep 3
cd /tmp

{
if [ -f /tmp/storeurl.txt ]; then
rm -fr /tmp/storeurl.txt
fi
}

wget $URL/storeurl.txt

{
if [ -f /tmp/storeurl.pl ]; then
rm -fr /tmp/storeurl.pl
fi
}

mv storeurl.txt storeurl.pl

# Checking IF STOREURL.PL file have been downloaded. if YEs continue further , otherwise EXIT the script with ERROR ! [[ JZ .. . .]]
{
if [ ! -f /tmp/storeurl.pl ]; then
echo
echo
echo -e "$COL_RED ERROR: STOREURL.PL File could not be download or not found in /tmp/ !! $COL_RESET"
exit 0
fi
}
cp -fr storeurl.pl $SQUID_DIR

echo
echo
echo -e "$COL_GREEN Setting EXECUTE permission for storeurl.pl . . . !! $COL_RESET"
chmod +x $SQUID_DIR/storeurl.pl

# Creating CACHE folders
echo
echo
echo -e "$COL_GREEN Creating CACHE directory in $CACHE_DIR , in this example,I used 5GB for cache for test ,Adjust it accordingly  . . . !! $COL_RESET"
sleep 3

# Checking if /cache-1 folder exist  . . .
{
if [ ! -d "$CACHE_DIR" ]; then
echo
echo
echo -e "$COL_GREEN Creating cache folder in $CACHE_DIR , Default size is 5GB, you should set it accordingly to your requirements  . . . $COL_RESET"
mkdir $CACHE_DIR
chown proxy:proxy $CACHE_DIR
chmod 777 -R $CACHE_DIR
squid -z
else
echo
echo -e "$COL_RED $CACHE_DIR folder already exists , Clearing it before proceeding. . . $COL_RESET"
rm -fr $CACHE_DIR/*
chown proxy:proxy $CACHE_DIR
echo -e "$COL_GREEN $CACHE_DIR Initializing Cache Directories as per the config  . . . $COL_RESET"
echo
squid -z
chmod 777 -R $CACHE_DIR
fi
}

echo
echo
echo -e "$COL_GREEN Adding squid in /etc/rc.local for auto startup . . . !! $COL_RESET"
sed -i '/exit/d' /etc/rc.local
sed -i '/[/usr\/sbin\/squid]/d' /etc/rc.local
echo /usr/sbin/squid >> /etc/rc.local
echo exit 0 >> /etc/rc.local
echo
echo -e "$COL_GREEN Starting SQUID (and adding 10 seconds Pause for proper initialization). . . !! $COL_RESET"
squid
sleep 5

# Checking SQUID status via PID [zaib]
#if [ "$pid" == "" ]; then
#echo
#echo -e "$COL_RED ERROR: UNABLE to start SQUID, try to run with -d1N syntax and see where its showing error !! $COL_RESET"
#else
ps aux |grep squid
echo
echo -e "$COL_GREEN $squidlabel is Running OK with PID number "$pid", no further action required, EXITING  . . .$COL_RESET"
echo
echo To view squid web access activity log, use command
echo -e "$COL_GREEN tail -f /var/log/squid/access.log $COL_RESET"
echo OR
echo -e "$COL_GREEN tail -f /var/log/squid/access.log |ccze $COL_RESET"
echo
echo -e "$COL_GREEN Regard's / Syed Jahanzaib . . . !! $COL_RESET"
echo


ALL DONE.

now execute the script by running

/temp/lusca_install.sh

It will start installation and will show you the progress with all the action its doing [in colored rows, RED color shows error, Green Color shows Ok/INFO].


 

TIP:

To start SQUID Server in Debug mode, to check any errors, use following

squid -d1n

if squid is successfully started , you can see its process via PS command

ps aux |grep squid

as showed in the image below …

squid-start-process

37 Comments »

  1. thank you… good post I will try now

    Like

    Comment by ven — July 2, 2014 @ 1:06 PM

  2. Dear sir;;

    How to block https traffic (facebook.com) by using squid with transparent proxy…

    Regards Assad

    Like

    Comment by Asad — July 2, 2014 @ 6:33 PM

  3. not working sir…

    Like

    Comment by tamsil — July 2, 2014 @ 6:54 PM

  4. nice share. This should be fresh ubuntu? I’ll try later when I had to reinstall.
    thanks

    Like

    Comment by papaeky — July 3, 2014 @ 12:42 AM

  5. is this Lusca proxy working with youtube https:// traffic recently youtube changed their traffic to https://

    Like

    Comment by ven — July 3, 2014 @ 1:28 PM

  6. Thanks ……….sir ALLAH aap ko jaza e khair dey………!
    We Check this Script…..

    Like

    Comment by Salman — July 6, 2014 @ 2:08 AM

  7. Sir…………….ubuntu 64 bit OS Architecture ” section ” & compiler for 64 bit
    not mention in this article……..ALLAH aap ko jaza e khair dey.

    Like

    Comment by Salman — July 7, 2014 @ 6:40 AM

  8. Dear Zaib thanks for your top POST
    I want to cache youtube https:// traffic
    if this possible with ssl?

    Like

    Comment by Haroon Rashidi — July 8, 2014 @ 3:04 AM

  9. Sir…Is this Script also work on ubuntu 64bit & 64bit OS Architecture?
    Because you don’t write in the script ( 64 bit compiler of squid )
    plz………explain ALLAH aap ko JAZA E KHAIR dey…..

    Like

    Comment by Salman — July 10, 2014 @ 6:22 AM

  10. Hi Syed, if i do LUSCHA with youtube for mikrotik. And want to do this script when computer restart. do i need to delete all the configs from LUSCHA and youtube? Thank you

    Like

    Comment by alex — July 10, 2014 @ 7:52 AM

  11. Hi Syed,

    I get this error

    ERROR: Unable to contact http://wifismartzone.com/files/linux_related/lusca, or possibly internet is not working or your IP is in black list at destination server !!
    ERROR: Please check manual if http://wifismartzone.com/files/linux_related/lusca is accessible or not or if it have required files, JZ !!

    Any possible way to resolve it.

    Thanks

    Like

    Comment by arnoldrodrigues — August 1, 2014 @ 7:16 PM

  12. Hi, I have installed the same on Ubuntu 12.04 with hardware config of I3-3.2Ghz with 8 GB RAM and 1TD HDD, in which 4GB allocated to squid and 500GB Storage. The problem i face is after 1 day of working, it slows down or doesn’t [the browsing part]. We need to restart the lusca/squid in order to work again any solution for this. The Bandwidth is around 40mb in peak time and average 20-25mbps .

    Like

    Comment by Sankar S — August 7, 2014 @ 10:52 AM

  13. hello mr syed.. i have a problem when the squid startup it take about 2 to 3 mb for upload and download without redirect any website to squid…can you help me…thanks you

    Like

    Comment by ismail — August 11, 2014 @ 2:14 PM

    • hmm there is no such issue in default squid config. You have to check at your router level that what data is being pulled/pushed from the squid. probably someone is downloading OR probably Linux is updating something. You should wait few hours , maybe it will end itself.

      Like

      Comment by Syed Jahanzaib / Pinochio~:) — August 14, 2014 @ 4:02 PM

  14. Hi Syed. First of all, many thanks for your blog!! it has a really helpful for my job!! I would like to ask you about Lusca is not caching youtube content anymore, i have tried on Ubuntu 12.04LTS and 14.04LTS with the same result: TCP_HIT: MISS, I have tried other tools/plugins as Squid Video & Music Booster but nothing. Any piece of advice?

    Like

    Comment by IgorLVG — August 15, 2014 @ 3:52 AM

  15. hi syed..its not working to caching youtube and facebook videos and applications..but working in other videos sites.. can i do anything to cache videos from youtube and facebook videos?

    Like

    Comment by ismail — August 18, 2014 @ 5:58 PM

  16. can i add android application to be cached ?

    Like

    Comment by ismail — August 18, 2014 @ 6:30 PM

  17. Excellent article, congratulations for the work
    which rules integration with mikrotik version 6.x with cache full?

    Marti Souza
    Brazil, Amazonas, Teffe

    Like

    Comment by Martinelli Souza — August 20, 2014 @ 12:42 AM

  18. thanks for your blogs sayd …and please tell me about tips that cache youtube and facebook sites
    cos
    all users tell me that face book more slowly!!!
    please tell me about that problem

    Like

    Comment by ahmed — August 29, 2014 @ 5:21 PM

  19. youtube no caching because is https. whats is the solution? thanks

    Like

    Comment by carlos pirela — September 7, 2014 @ 10:57 AM

  20. Dear zaib please tell us how to cache youtube https and another https ?????
    please help us
    thank you

    Like

    Comment by haroonnimroozy — September 8, 2014 @ 7:55 PM

  21. dear sir please help what is dscp this script ? for microtik router

    Like

    Comment by lalu — September 10, 2014 @ 11:50 PM

  22. i have problem : clientnatlookup nf getsockopt(so_original_dst) failed (92) protocol not available. how can solve?

    Like

    Comment by syarifudin — October 23, 2014 @ 1:28 PM

  23. Dear Sir, Please let me know that when installing ubuntu do we have to make partitions or it will make it automatically. if we do have to make then please do let me know how and we can make and which partitions. thanks

    Like

    Comment by Roger — October 25, 2014 @ 1:28 PM

  24. Thanks sir it is working on latest ubuntu 14.04 LTS server, but how can I restart the lusca service without restarting the whole machine? I’ve tried sudo service lusca restart and sudo service squid restart but its not working or the service is not recognized.

    Thanks in advance sir,

    Like

    Comment by Norman Villas — November 15, 2014 @ 12:17 PM

  25. Dear Sir, please tell me how to REMOVE ANY KIND OF PREVIOUSLY INSTALLED SQUID INSTALLATION COMPLETELY.
    Thanks b4
    dedmax

    Like

    Comment by dedmax — November 17, 2014 @ 7:17 PM

  26. alhamdulillah! it’s working! but i need help…
    my network is like this :
    ISP 1 (192.168.1.1) —-|
    MIKROTIK —————— LAN Client (192.168.5.0/24) GW 192.168.5.254
    ISP 2 (192.168.2.2) ——| |
    |
    Ubuntu server 12.04 LTS + LUSCA r14942 (192.168.4.2) GW 192.168.4.1

    eth01 – WAN (ISP1)
    eth02 – WAN (ISP2)
    eth03 – disabled
    eth04 – Ubuntu + LUSCA
    eth05 – LAN

    I can’t use your tutorial https://aacable.wordpress.com/2011/07/21/mikrotik-howto-redirect-http-traffic-to-squid-with-original-source-client-ip/
    if i’m using DSTNAT..it works fine…but i have to masquerade all interface i had. can you help me?
    oh by the way, i’m still using mikrotik 5.xx but theres an error on LUsca says
    unrecognized : ‘qos_flows’ and after i used

    tcp_outgoing_tos 0x30 “localnet” [ACL name for my local network or 192.168.5.0/24]
    zph_mode tos
    zph_local 0x30
    zph_parent 0
    zph_option 136

    another error says : unrecognized tcp_outgoing_tos

    pelase, can you help me?
    mail me at gonzhilaz@gmail.com
    Thank you!

    Like

    Comment by Bayu — November 21, 2014 @ 6:11 PM

  27. Hi!

    when running squid proxy mikrotik graphs are stopped. not changed. please give me solution

    Like

    Comment by PEDDAREDDY — December 4, 2014 @ 8:38 AM

  28. Thanks ……….sir ALLAH aap ko jaza e khair dey………!
    We Check this Script nyc work but sir 1kia lusca may speedtest.ptcl.net ya speedtest.net cache nahi hoti aksar form per frnds ne speedtest ki snaps di hoe hoti hain but mere pas speedtest cache nahi ho rahi…

    please tell me kia ye ho sekti hai is me……..???

    Like

    Comment by Muhammad Furqan Khan — February 5, 2015 @ 5:39 PM

    • You shouldn’t be caching speedtest.net like sites, they are dynamic and made to give you fresh result on every check. Why you want to get FALSE results with it ?

      Like

      Comment by Syed Jahanzaib / Pinochio~:) — February 6, 2015 @ 8:13 AM

  29. what is the command to start the squid service in ubuntu 10.04??? and my all configuration is done but system shows “unrecognized bash command” like this please tell what is actual command to start and stop the service of squid???
    /etc/init.d in the path squid not here but all other are here.

    please solve me problem as soon as. I shall be thankful to you Sir

    Like

    Comment by Ahamd Salman — May 12, 2015 @ 3:48 PM

  30. Dear Mr.Zaib
    I have installed the same on Ubuntu 12.04 with hardware config of Xeon-2.8Ghz quad core with dual cpu with 20 GB RAM and 1.5TB HDD, in which 4GB allocated to squid and 500GB Storage. The problem i face is after 1 or 2 day of working, it slows down or doesn’t [the browsing part]. We need to restart the lusca/squid in order to work again any solution for this

    Like

    Comment by haroonnimroozy — June 6, 2015 @ 5:18 PM


RSS feed for comments on this post. TrackBack URI

Leave a comment