Ultimate Security Proxy With Tor

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.

Submitted by Tb0hdan (Contact Author) (Forums) on Fri, 2008-10-24 12:21. :: Security

Ultimate Security Proxy With Tor

Nowadays, within the growing web 2.0 environment you may want to have some anonymity, and use other IP addresses than your own IP. Or, for some special purposes - a few IPs or more, frequently changed. So no one will be able to track you. A solution exists, and it is called Tor Project, or simply tor. There are a lot of articles and howtos giving you the idea of how it works, I'm not going to describe here onion routing and its principles, I'll rather tell you how practically pull out the maximum out of it.

So, let's start.

Major disadvantages of tor, with all its benefits like security (however, tor manufacturers advice not to rely on its strong anonymity, because there are some cases, when it is possible to track young h4x0r...) are:

1. Long enough session (60 seconds for one connection per one exit node, that means, that you will have one external IP for a minute)
2. Slow perfomance (request takes up to 20 seconds to complete, what makes surfing sites with lot of elements a disaster)
3. All the requests come through one node, and possibly route how your requests arrive can be calculated.

In order to fight these three we are going to use:

1. 8 tor processes, each using separate spool directory
2. 8 privoxy processes, each configured to talk to separate tor.
3. First squid, with malware domains blacklist, will have 8 round robin cache peers configured. (squid-in)
4. Havp, with squid-in as parent. (Anti-virus proxy, using clamav :) )
5. Second squid, that will use havp as parent (squid-out). Users will connect to this one.

Schema:

Tor Proxies

Okay, so you got it, how all it looks like. Now let's do all the required configuration.

For squid-in about 2Gb of disc cache will be enough.
For squid-out it is better to have about 12-15 Gb of disc cache to store clean elements.
So you'll need at least 20 Gb HDD.

Now about RAM.

Each tor process consumes ~ 6 Mb of Ram. 6x8 = 48 ~ 64 Mb of RAM for Tor processes.
Privoxy eats about 2 Mb each. 2x8 = 16 ~ 32 Mb of RAM.
So, 128 Mb of RAM will be enough for Tor and Privoxy processes.
Squid-in will have memory footprint approx 128 Mb. We assume it eats 256 Mb.
Same for Squid-out. We got 768 Mb of RAM total required. If it is gonna be a real system - we assume 1 Gb hardware RAM
installed.

CPU.

As you know, anti-virus scanning is very CPU demanding.
I've used a 2.0 GHz P4.
I've tested this system under concurrent load under 10 users, and it did now show it was exhausted.

Finally, we have:

CPU: 2.0 GHz
RAM: 1.0 Gb
HDD: 20 Gb

Used software:

Privoxy

privoxy --version

Privoxy version 3.0.8 (http://www.privoxy.org/)

Tor

tor --version

Oct 11 19:49:27.394 [notice] Tor v0.1.2.19. This is experimental software. Do not rely on it for strong anonymity.
Tor version 0.1.2.19.

Squid

squid -v

Squid Cache: Version 2.7.STABLE3
configure options: '--prefix=/usr' '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid' '--localstatedir=/var'
'--datadir=/usr/share/squid' '--enable-auth=basic,digest,ntlm' '--enable-removal-policies=lru,heap'
'--enable-digest-auth-helpers=password'
'--enable-basic-auth-helpers=SASL,PAM,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'
'--enable-external-acl-helpers=wbinfo_group,ip_user,session,unix_group'
'--enable-ntlm-auth-helpers=SMB,fakeauth' '--enable-ident-lookups' '--enable-useragent-log'
'--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log' '--enable-arp-acl' '--with-pthreads'
'--with-large-files' '--enable-htcp' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-maxfd=8192'
'--enable-snmp' '--enable-ssl' '--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'
'--enable-linux-netfilter' '--libdir=/usr/lib64' '--build=x86_64-pc-linux-gnu'
'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu' 'CC=x86_64-pc-linux-gnu-gcc'
'CFLAGS=-march=k8 -O2 -pipe -fomit-frame-pointer' 'LDFLAGS=-Wl,-O1'

Tor Configuration

/etc/tor/torrc

SocksPort 9050 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor

/etc/tor/torrc2

SocksPort 9150 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor2
ControlPort 9151

/etc/tor/torrc3

SocksPort 9250 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor3
ControlPort 9251

/etc/tor/torrc4

SocksPort 9350 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor4
ControlPort 9351

/etc/tor/torrc5

SocksPort 9450 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor5
ControlPort 9451

/etc/tor/torrc6

SocksPort 9550 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor6
ControlPort 9551

/etc/tor/torrc7

SocksPort 9650 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor7
ControlPort 9651

/etc/tor/torrc8

SocksPort 9750 # what port to open for local application connections
SocksBindAddress 127.0.0.1 # accept connections only from localhost
AllowUnverifiedNodes middle,rendezvous
Log notice syslog
RunAsDaemon 1
User tor
Group tor
DataDirectory /var/lib/tor8
ControlPort 9751

Configuration files for privoxy

/etc/privoxy/config

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8118
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9050 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config2

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy2
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8129
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9150 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config3

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy3
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8230
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9250 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config4

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy4
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8321
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9350 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config5

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy5
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8421
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9450 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config6

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy6
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8522
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9550 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config7

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy7
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8623
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9650 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

/etc/privoxy/config8

trust-info-url  http://www.example.com/why_we_block.html
trust-info-url  http://www.example.com/what_we_allow.html
confdir /etc/privoxy
logdir /var/log/privoxy8
actionsfile standard.action  # Internal purpose, recommended
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
logfile logfile
listen-address  127.0.0.1:8724
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
buffer-limit 4096
forward-socks4a   /               127.0.0.1:9750 .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0

Startup scripts

Privoxy: /etc/rc.d/privoxy (Arch linux)

#!/bin/bash
# source application-specific settings
[ -f /etc/conf.d/privoxy ] && . /etc/conf.d/privoxy
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/sbin/privoxy`
start() {
  stat_busy "Starting Privoxy, squid, havp"
  chown privoxy:privoxy /var/log/privoxy
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy/$i
      chmod 660 /var/log/privoxy/$i
  done
  chown privoxy:privoxy /var/log/privoxy2
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy2/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy2/$i
      chmod 660 /var/log/privoxy2/$i
  done
  chown privoxy:privoxy /var/log/privoxy3
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy3/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy3/$i
      chmod 660 /var/log/privoxy3/$i
  done
chown privoxy:privoxy /var/log/privoxy4
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy4/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy4/$i
      chmod 660 /var/log/privoxy4/$i
  done
   chown privoxy:privoxy /var/log/privoxy5
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy5/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy5/$i
      chmod 660 /var/log/privoxy5/$i
  done
  
    chown privoxy:privoxy /var/log/privoxy6
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy6/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy6/$i
      chmod 660 /var/log/privoxy6/$i
  done
  chown privoxy:privoxy /var/log/privoxy7
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy7/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy7/$i
      chmod 660 /var/log/privoxy7/$i
  done
    chown privoxy:privoxy /var/log/privoxy8
  # create missing logfiles
  for i in logfile jarfile; do 
      touch /var/log/privoxy8/$i
      chgrp $PRIVOXY_GROUP /var/log/privoxy8/$i
      chmod 660 /var/log/privoxy8/$i
  done
#[ -z "$PID" ] && /usr/sbin/privoxy $PRIVOXY_ARGS 2>/dev/null
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy.pid /etc/privoxy/config
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy2.pid /etc/privoxy/config2
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy3.pid /etc/privoxy/config3
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy4.pid /etc/privoxy/config4
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy5.pid /etc/privoxy/config5
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy6.pid /etc/privoxy/config6
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy7.pid /etc/privoxy/config7
  /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy8.pid /etc/privoxy/config8
  havp -c /etc/havp/havp.config  
  
  squid -f /etc/squid/squid-in.conf -DYC
  squid -f /etc/squid/squid-out.conf -DYC
  
  if [ $? -gt 0 ]; then
    stat_fail
  else
    add_daemon privoxy
    stat_done
  fi
}
stop() {
  stat_busy "Stopping Privoxy, Squid, Havp"
  #[ ! -z "$PID" ]  && kill $PID &> /dev/null
  killall privoxy
  killall squid
  killall havp
  
  if [ $? -gt 0 ]; then
    stat_fail
  else
    rm_daemon privoxy
    stat_done
  fi
}
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    $0 stop
    sleep 1
    $0 start
    ;;
  *)
    echo "usage: $0 {start|stop|restart}"  
esac
exit 0

Tor startup script: /etc/rc.d/tor (Arch linux)

#!/bin/bash
. /etc/rc.conf
. /etc/rc.d/functions
#PID=`pidof -o %PPID /usr/bin/tor`
tor='/usr/bin/tor'
case "$1" in
        start)
                stat_busy "Starting Tor Daemon"
                #[ -z "$PID" ] && /usr/bin/tor &>/dev/null
                ${tor} -f /etc/tor/torrc
                ${tor} -f /etc/tor/torrc2
                ${tor} -f /etc/tor/torrc3
                ${tor} -f /etc/tor/torrc4
                ${tor} -f /etc/tor/torrc5
                ${tor} -f /etc/tor/torrc6
                ${tor} -f /etc/tor/torrc7
                ${tor} -f /etc/tor/torrc8
    
                if [ $? -gt 0 ]; then
                        stat_fail
                else
                        add_daemon tor
                        stat_done
                fi
                ;;
        stop)
                stat_busy "Stopping Tor Daemon"
                #[ ! -z "$PID" ] && kill $PID &> /dev/null
                killall tor
                if [ $? -gt 0 ]; then
                        stat_fail
                else
                        rm_daemon tor
                        stat_done
                fi
                ;;
		restart)
                $0 stop
                sleep 3
                $0 start
                ;;
        *)
                echo "usage: $0 {start|stop|restart}"
esac
exit 0
# vim: ft=sh ts=2 sw=2

Squid-in config

/etc/squid/squid-in.conf

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl malware_domains url_regex '/etc/squid/Malware-domains.txt'
http_access deny malware_domains
http_access allow localhost
http_access deny all
icp_access deny all
http_port 3400
icp_port 0
hierarchy_stoplist cgi-bin ?
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
cache_peer localhost parent 8118 0 round-robin no-query
cache_peer localhost2 parent 8129 0 round-robin no-query
cache_peer localhost3 parent 8230 0 round-robin no-query
cache_peer localhost4 parent 8321 0 round-robin no-query
cache_peer localhost5 parent 8421 0 round-robin no-query
cache_peer localhost6 parent 8522 0 round-robin no-query
cache_peer localhost7 parent 8623 0 round-robin no-query
cache_peer localhost8 parent 8724 0 round-robin no-query
never_direct allow all
always_direct deny all
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
forwarded_for off
coredump_dir /var/cache/squid-in
cache_dir ufs /var/cache/squid-in 100 16 256
pid_filename /var/run/squid-in.pid
access_log /var/log/squid/access.squid-in.log
cache_store_log /var/log/squid/store.squid-in.log
cache_log /var/log/squid/cache.squid-in.log

Squid-out config

/etc/squid/squid-out.conf

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
icp_access deny all
http_port 3128
icp_port 0
hierarchy_stoplist cgi-bin ?
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
cache_peer localhost parent 3410 0 round-robin no-query
never_direct allow all
always_direct deny all
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
forwarded_for off
coredump_dir /var/cache/squid-out
cache_dir ufs /var/cache/squid-out 100 16 256
pid_filename /var/run/squid-out.pid
access_log /var/log/squid/access.squid-out.log
cache_store_log /var/log/squid/store.squid-out.log
cache_log /var/log/squid/cache.squid-out.log

Havp antivirus proxy config

/etc/havp/havp.config

USER havp
GROUP havp
DAEMON true
PIDFILE /tmp/havp.pid
SERVERNUMBER 4
MAXSERVERS 256
SCANTEMPFILE /tmp/havp-XXXXXX
TEMPDIR /tmp
DBRELOAD 60 
PARENTPROXY localhost
PARENTPORT 3400
FORWARDED_IP false
X_FORWARDED_FOR false
PORT 3410
BIND_ADDRESS 127.0.0.1
ENABLECLAMLIB true
ENABLECLAMD false
ENABLEFPROT false
ENABLEAVG false
ENABLEAVESERVER false
ENABLESOPHIE false
ENABLETROPHIE false
ENABLENOD32 false
ENABLEAVAST false
ENABLEARCAVIR false
ENABLEDRWEB false

Domain List updater script (can be launched from cron)

#!/usr/bin/python
# -*- coding: utf-8 -*-
# Author: Bohdan Turkynewych, AKA Gh0st, 2006-2008, tb0hdan[at]gmail[dot]com
# Distributed under the terms of GPLv2
# Parts of code from neutron jabber bot project, http://code.google.com/p/neutron/
from urllib2 import Request as urllib2_Request, urlopen as urllib2_urlopen
from sys import exit as sys_exit
from re import search as re_search
from time import ctime as time_ctime, time as time_time
from re import compile as re_compile
def t_conv(timestamp):
        reply = ''
        seconds = timestamp % 60
        minutes = (timestamp / 60) % 60
        hours = (timestamp / 3600) % 60
        days = timestamp / 216000
        if days: reply += str(days) + ' day(s) '
        if hours: reply += str(hours) + ' hour(s) '
        if minutes: reply += str(minutes) + ' minute(s) '
        reply += str(seconds) + ' second(s)'
        return reply
    
def uptime():
        global BOOTUP_TIMESTAMP
        if BOOTUP_TIMESTAMP:
                idletime = int(time_time() - BOOTUP_TIMESTAMP)
                reply = t_conv(idletime)
        else:
                reply = 'Unknown'
        return reply
strip_tags = re_compile(r'<[^]+>')
def decode(text):
    data = text.replace('<br>','\n').replace(' ', ' ').replace('<', '<').replace('>', '>').replace('"', '"').replace('<br>','\n').replace('
<li>','\r\n')
    return strip_tags.sub('', data)
def get_data(message):
    reply=[]
    try:
        for line in message.split('\n'):
            if re_search('wbr',line):
                #
                od = re_search('',line)
                d0main = line[od.end():]
                d0main = d0main[:re_search('',d0main).start()].strip()
                #
                if d0main == '-':
                        od = re_search('-',line)
                        bookurl = line[od.end():]
                        bookurl = bookurl[:re_search('',bookurl).start()].strip()
                        bookurl = decode(bookurl)
                else:
                    bookurl = decode(d0main)
                d0main = ''
                stripper = '[/|:]'
                if re_search(stripper,bookurl):
                      #d0main = bookurl
                     od = re_search(stripper,bookurl)
                     d0main = bookurl[od.end():]
                     d0main = bookurl[:re_search(stripper,bookurl).start()].strip()
                else:
                     d0main = bookurl
                if not d0main in reply:
                    reply.append(d0main)
        return reply
    except:
         raise
def update_list(filename):
    req2 = urllib2_Request('http://www.malwaredomainlist.com/mdl.php?sort=Domain&ascordesc=DESC&search=&colsearch=All&quantity=All')
    req2.add_header = ('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.9) Gecko/20071030 SeaMonkey/1.1.6')
    print '[+] Update process started at ' + time_ctime()
    try:
        r = urllib2_urlopen(req2)
        target = r.read()
        urls = get_data(target)
    except:
        raise
        print '[-] Error occured while fetching / parsing data.'
        sys_exit(1)
    print '[+] Got ' + str(len(urls)) + ' domains/ips.'
    try:
        file = open(filename,'w')
        for line in urls:
            file.write(line + '\n')
        file.close()
    except:
        print '[-] Error occured while flushing data to file'
        sys_exit(5)
    elapsed = uptime()
    print '[+] Update process finished at ' + time_ctime() + '. Took ' + elapsed
BOOTUP_TIMESTAMP = time_time()
update_list('/etc/squid/Malware-domains.txt')

Have fun!

P.S. Future optimisation requires 8 havp instances as well due to slowup.

P.S.S You may exclude havp and squid-out proxy from your system, and use only squid-in. That will be much, much faster.

add comment |

view as pdf |

print |

Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.

Hosts configuration for squid round-robin

Submitted by Bohdan Turkynewych (not registered) on Thu, 2009-04-23 14:50.

Sorry about that, I've forgot to add /etc/hosts file in the article...

That is how i fooled squid.

Here is the file:

127.0.0.1 localhost

127.0.0.1 localhost2

127.0.0.1 localhost3

127.0.0.1 localhost4

127.0.0.1 localhost5

127.0.0.1 localhost6

127.0.0.1 localhost7

127.0.0.1 localhost8

Subscribe to HowtoForge Newsletter and stay informed about our latest HOWTOs and projects.


(To unsubscribe from our newsletter, visit this link.)

Options For This Howto

Navigation

User login

Who's online

HowtoForge Forums

News

Recent comments

Newsletter

Syndicate

Ultimate Security Proxy With Tor