this translation is deprecated - please compare the translation with the 'Updated version of Original Article' below for updates. Sorry to all that I haven't had the time to translate the newest copies...

Updated version of Original Article

Postfix + Amavis/ClamAV + DSPAM, Mailfilter Gateway

Contents:

1. Introduction

Scenario

All email will be relayed through the mailfilter gateway. The mailfilter gateway will scan viruses and classify spam.

Individual emails containing a virus will be dropped before being delivered to the back-end server. Clean emails will proceed to the DSPAM filtering engine, which classifies the email as spam or non-spam (ham). After classification has been completed, all emails (spam and ham) will be sent to the back-end server, which will then deliver the email to the end user's mailbox using the local MTA (exchange,sendmail,insert_fun_mta_here). The filtering of spam/ham email can then be done using header/subject filters within the email client, or by the MTA/procmail/etc...

The path emails take while traveling through the mailfilter gateway, before being sent to the end-user mailbox, is approximately depicted below.

Virus-Free mail > Postfix
SMTP
Server		 > smtp-amavis
localhost:10024		 > dspam
localhost:10025		 > Postfix queue > SMTP Backend Server
Virus-Infected mail > Postfix
SMTP
Server		 > smtp-amavis
localhost:10024		 > localhost:10026 > Drop mail to quarantine folder > Send notification to virus-admin
(and to sender if necessary)

Preparations

Confirm back-end server can send and accept email. The MTA you decide to use on the back-end server can be anything you like... the filter gateway should be able to drop in front of any existing mail system (qmail, postfix, exim, exchange, etc)

Install postfix on a separate server from the back-end MTA server and confirm that it is functional

Note: Some postfix tutorials can be found at http://www.postfix.org/docs.html.

Warning: This example configuration was installed within the Gentoo GNU/Linux Operating System. There is a big possibility that the location/content of the configuration files below will not be suitable for other GNU/Linux Operating Systems without some modifying.

2. ClamAV

Installation

Code listing 2.1

# emerge clamav

Configuration

The configuration of ClamAV is brief and fairly simple

Code listing 2.2: /etc/clamav.conf

LogFile /tmp/clamd.log
LogFileMaxSize 2M
LogTime
LogSyslog
PidFile /var/run/clamd.pid
DatabaseDirectory /var/lib/clamav
LocalSocket /var/run/clamav/clamd
MaxThreads 10
ReadTimeout 300
MaxDirectoryRecursion 15
User clamav
AllowSupplementaryGroups
ScanOLE2
ScanArchive
ArchiveMaxFileSize 10M
ArchiveMaxRecursion 5
ArchiveMaxFiles 1000
ArchiveMaxCompressionRatio 200

It may be necessary to update the freshclam config before you can download new virus definitions

Code listing 2.3: /etc/freshclam.conf

 
# For example, we may need to define a proxy server
             
HTTPProxyServer proxy.example.com
HTTPProxyPort 3128

Setting up /etc/conf.d/clamd

Code listing 2.4: /etc/conf.d/clamd

# Config file for /etc/init.d/clamd
 
START_CLAMD=yes
CLAMD_OPTS=""
CLAMD_LOG=""
 
START_FRESHCLAM=yes
FRESHCLAM_OPTS="-d -c 2"
FRESHCLAM_LOG="/var/log/clam-update.log"

Start the clamd daemon and add it to the default runlevel so it starts when the mailfilter is rebooted.

Code listing 2.5

# mkdir /var/run/clamav
# chown clamav:clamav /var/run/clamav
# /etc/init.d/clamd start
# rc-update add clamd default

3. Amavisd-New

Installation

Code listing 3.1

# emerge amavisd-new

Note:
Emerge automatically installs dependancies that are needed.

If installing from source, several perl modules must also be installed:
Archive::Tar (Archive-Tar-x.xx)
Archive::Zip (Archive-Zip-x.xx)
Compress::Zlib (Compress-Zlib-x.xx)
Convert::TNEF (Convert-TNEF-x.xx)
Convert::UUlib (Convert-UUlib-x.xxx)
MIME::Base64 (MIME-Base64-x.xx)
MIME::Parser (MIME-Tools-x.xxxx)
Mail::Internet (MailTools-1.58+)
Net::Server (Net-Server-x.xx)
Net::SMTP (libnet-1.16+)
Digest::MD5 (Digest-MD5-x.xx)
IO::Stringy (IO-stringy-x.xxx)
Time::HiRes (Time-HiRes-1.49+)
Unix::Syslog (Unix-Syslog-x.xxx)

Several compression programs are needed, like unzip, unarj (as seen in /etc/amavisd.conf)

Configuration

We could read the file /etc/amavisd.conf, and configure in accordance to the requirements, but you must have at least something close to this example config.

Code listing 3.2: /etc/amavisd.conf

$MYHOME = '/var/amavis'
$mydomain = 'example.com'; (updated with your domain)

# daemon user must be the same user that started the clamav daemon
$daemon_user = 'clamav';
$daemon_group = 'clamav';

# emails that were processed and do not contain a virus will be forwarded to smtp:127.0.0.1:10025. 

# This smtp connection will be equipped with the content filter DSPAM
$forward_method = 'smtp:127.0.0.1:10025';

# Notification emails/alerts will not be processed by DSPAM and will be forwarded to smtp connector without a content filter

$notify_method = 'smtp:127.0.0.1:10026';

# The spamassin functions in amavisd-new will be disabled, since the process of the spam identification will taken over by DSpam.
# Update this configuration variable near line 146

@bypass_spam_checks_acl  = qw( . );


# Update the virus admin defined near line 459
$virus_admin = "admin\@example.com";

# Update virus trashcan dir near line 519
$QUARANTINEDIR = '/var/virusmails';


# Update clamd configuration near lines 1211-1214 
# Confirm that /var/run/clamav/clamd is correct and matches with
# LocalSocket /var/run/clamav/clamd in /etc/clamav.conf

['Clam Antivirus-clamd',
    \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd"],
    qr/\bOK$/, qr/\bFOUND$/,
    qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],

Directories within our configuration must be set with the proper permissions

Code listing 3.3

# mkdir /var/amavis
# chown clamav:clamav /var/amavis

# chmod 750 /var/amavis
# mkdir /var/virusmails
# chown clamav:clamav /var/virusmails

Start the amavisd-new daemon and add it to the default runlevel so it starts when the mailfilter gateway is rebooted...

Code listing 3.4

# /etc/init.d/amavisd start
# rc-update add amavisd default

4. DSPAM

Installation

As of this writing/translation, a final ebuild for DPSPAM is under development, so we must fetch and compile the source from http://www.nuclearelephant.com/projects/dspam/

Code listing 4.1

# cd /root
# wget http://www.nuclearelephant.com/projects/dspam/sources/dspam-2.10.6.tar.gz

# tar zxvf dspam-2.10.6.tar.gz
# cd dspam-2.10.6

Create user dspam

Code listing 4.2

# adduser -d /var/dspam -s /bin/false dspam

View the file dspam-2.10.6/README, then configure the source (requires mysql)

Code listing 4.3

# ./configure \
   --with-userdir=/var/dspam \
#   --with-dspam-home=/var/dspam \
# --with-userdir changed to --with-dspam-home in 3.x
   --with-userdir-owner=dspam \
#   --with-dspam-home-owner=dspam \
   --with-dspam-home--group=postdrop \
   --with-dspam-owner=dspam \
   --with-dspam-group=postdrop \
   --with-dspam-mode=4510 \
   --with-local-delivery-agent=/usr/sbin/sendmail \
#   --with-delivery-agent=/usr/sbin/sendmail \
# --with-local-delivery-agent changed to --with-delivery-agent in 3.x
   --with-storage-driver=mysql_drv \
   --with-mysql-includes=/usr/include/mysql \
   --with-mysql-libraries=/usr/lib/mysql \
   --enable-alternative-bayesian \
   --disable-trusted-user-security \
   --enable-large-scale \
   --enable-virtual-users \
   --enable-long-usernames \
   --enable-spam-delivery 
#   --enable-spam-delivery
# removed from 3.x

Create and import DSPAM MySQL databases

Code listing 4.4

# mysql -u root -p -e "create database dspam"

# mysql -u root -p -e "grant all on dspam.* to dspam@localhost identified by 'DSPAMPASS'"
# mysql -u dspam -p dspam < tools.mysql_drv/mysql_objects.sql.speed.optimized
# mysql -u dspam -p dspam < tools.mysql_drv/virtual_users.sql
# cp tools.mysql_drv/purge.sql /var/dspam

Code listing 4.5: /var/dspam/mysql.data

/var/run/mysqld/mysqld.sock #(or possibly /tmp/mysqld.sock)
 
dspam
DSPAMPASS
dspam

Code listing 4.6

# chown dspam:postdrop /var/dspam/mysql.data
# chmod 440 /var/dspam/mysql.data

Add the DSPAM maintenance purge.sql query to run from cron daily at midnight. This program will remove tokens that have expired so the database does not continue to grow.

Code listing 4.7: /etc/crontab

0 0 * * * (/usr/bin/mysql -u dspam -p'DSPAMPASS' dspam  < /var/dspam/purge.sql)1>&2>/dev/null

5. Postfix

Not much will need to be changed from the default postfix configuration...

Code listing 5.1: /etc/postfix/master.cf

# amavisd-new + clamav : virus checker
# dspam                : spam checker

smtp                 inet      n      -      n      -      -      smtpd
  -o content_filter=smtp-amavis:[127.0.0.1]:10024
 
smtp-amavis          unix      -      -      n      -      2      lmtp
  -o smtp_send_xforward_command=yes
 
127.0.0.1:10025      inet      n      -      n      -      -      smtpd
  -o cleanup_service_name=pre-cleanup
  -o content_filter=dspam:dummy
  -o local_recipient_maps=
  -o relay_recipient_maps=
  -o smtpd_restriction_classes=
  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o strict_rfc821_envelopes=yes
  -o smtpd_error_sleep_time=0
  -o smtpd_soft_error_limit=1001
  -o smtpd_hard_error_limit=1000
                                                                                                                    
127.0.0.1:10026      inet      n      -      n      -      -      smtpd
  -o local_recipient_maps=
  -o relay_recipient_maps=
  -o smtpd_restriction_classes=
  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o strict_rfc821_envelopes=yes
  -o smtpd_error_sleep_time=0
  -o smtpd_soft_error_limit=1001
  -o smtpd_hard_error_limit=1000
                                                                                                                    
dspam                unix      -      n      n      -      -      pipe
  flags=Rhq user=dspam argv=/usr/local/bin/dspam -i -f ${sender} -- %u --user ${recipient}
#dspam		     unix	-	n	n	-	10	pipe
#flags=Rhq user=dspam argv=/usr/local/bin/dspam --mode=teft --deliver=innocent,spam --feature=chained,noise -i -f ${sender} -- %u --user ${recipient}

# The above two lines are for dspam 3.x, otherwise it will call you an insensitive clod ...
# and possibly make you cry like a little girl with a skinned knee... I know I did
                                                                                                                    
cleanup              unix      n      -      n      -      0      cleanup
  -o header_checks=
  -o mime_header_checks=
  -o nested_header_checks=
  -o body_checks=
                                                                                                                    
pre-cleanup          unix      n      -      n      -      0      cleanup
  -o canonical_maps=
  -o sender_canonical_maps=
  -o recipient_canonical_maps=
  -o masquerade_domains=
  -o virtual_alias_maps=
  -o always_bcc=
  -o sender_bcc_maps=
  -o recipient_bcc_maps=
                                                                                                                   
local                unix      -      n      n      -      -      local
  -o content_filter=
  -o myhostname=localhost
  -o local_recipient_maps=
  -o relay_recipient_maps=
  -o mynetworks=127.0.0.0/8
  -o mynetworks_style=host
  -o smtpd_restriction_classes=
  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject

dspam-add            unix      -      n      n      -      -      pipe
   flags=Rhq user=dspam argv=/usr/local/bin/dspam --user ${user}@${nexthop} --addspam
#   flags=Rhq user=dspam argv=/usr/local/bin/dspam --mode=teft --user ${user}@${nexthop} --class=spam --source=error
# --class=spam --source=error is for dspam 3.x
dspam-del             unix      -      n      n      -      -      pipe 
   flags=Rhq user=dspam argv=/usr/local/bin/dspam --user ${user}@${nexthop} --falsepositive  
#   flags=Rhq user=dspam argv=/usr/local/bin/dspam --mode=teft --user ${user}@${nexthop} --class=innocent --source=error
# --class=innocent --source=error is for dspam 3.x

Code listing 5.2: /etc/postfix/main.cf

relay_domains = $mydestination example.com

transport_maps = regexp:/etc/postfix/transport_regexp, 
                 hash:/etc/postfix/transport.domain


# IMPORTANT!		 
dspam_destination_recipient_limit = 1
dspam-add_destination_recipient_limit = 1
dspam-fp_destination_recipient_limit = 1

For the process of training DSPAM on spam or false-positive emails (ham marked as spam), All email that is not properly identified as spam should be sent/forwarded to user@spam.example.com. False-positives should be forwarded to user@ham.example.com. Make sure to add DNS entries for spam.example.com and ham.example.com, both pointing to the IP of the mailfilter gateway

Code listing 5.3: /etc/postfix/transport_regexp

/^.*@spam.(.*)$/     dspam-add:${1}
/^.*@ham.(.*)$/          dspam-del:${1}

Send email to smtp back-end server

Code listing 5.4: /etc/postfix/transport.domain

example.com smtp:xyz.xyz.xyz.xyz

Code listing 5.5

# postmap /etc/postfix/transport.domain
# newaliases
# postfix reload

6. Finishing

Example configuration of a per-user procmailrc

Code listing 6.1: ~user/.procmailrc

VERBOSE=off
MAILDIR=$HOME/mail
 
:0 H
* ^X-DSPAM-Result: Spam
SPAM

Setting Primary MX record domain

Code listing 6.2: db-example.com.

; primary MX record for domain example.com should be directed to mailgateway.example.com
example.com.	IN	MX  10  mailgateway.example.com.

; Create an A record for mailgateway.example.com
mailgateway     IN 	A       mailgateway.ip.add.here
spam		IN	CNAME   mailgateway
ham		IN	CNAME   mailgateway

; Create an A record for mail.example.com (back-end server, where per-user mailboxes are kept)
mail            IN	A       mail.ip.add.here

Settings for the back-end MTA server

Point outgoing mail directly at the mailfilter gateway (relayhost).

Training DSpam

Each email that you categorize as spam you will send to user@spam.example.com
If it was false-positive (email not spam, but marked as spam by DPAM) you will send the email to user@ham.example.com
Little by little, DPSAM will learn based on what you categorize as spam/ham.

Each user is free to categorize their own spam, which will not affect other user's spam/ham training of DSPAM

Note: Speaking from personal experience, the accuracy of DSPAM was good enough after only training 100 emails

Important: Email that you sent to user@spam.example.com should only be email that you categorize as spam after it slipped through the DSPAM filter into your inbox. Any spam caught by the filter will have the added header: 'X-DSPAM-Result: Spam', so you will need to set your email client to separate this spam to another folder, such as 'email marked as spam' and then check through that folder once or twice a week for false positives when you begin to first use DSPAM - then send those false positives to user@ham.example.com

To speed up the training process, we could make use of email that has already been classified as spam/non-spam and import them to dspam.
That could then serve as a global classification database for users.
At the DSPAM website they offer some email that has been pre-classified, which they refer to as the SA-Training-Corpus.

Code listing 6.3


# wget http://www.nuclearelephant.com/projects/dspam/SA-Corpus.tar.gz
# tar zxvf SA-Corpus.tar.gz
# cd SA-Corpus

Unfortunately the train.pl script within SA-Corpus is for DSPAM 3.0.x beta If you use 2.x, use the train.pl

Download modified train.pl

Code listing 6.4

# ./train.pl user@example.com #(or global for all users)

7. Rewrite SPAM Subject (Optional)

Not all mail clients can filter email based on specific headers. Unfortunately, dspam did not provide a feature for this.
Using a little trick, we were able to rewrite the spam subject

The result is that each email categorised as spam will be tagged with the prefix [**SPAM**], so a user can easily filter spam based on subject.

8. References
Updated 05/12/2004
Cecep Mahbub
Author

Summary: This article gives a general outline (not all details) of the process of installing a gateway mail filter server dedicated to scanning/tagging/dropping viruses and unwanted mail.
Copyright 2004. Cecep Mahbub. Translated by Kyler Klein