DSPAM v2.7 <jonathan@nuclearelephant.com>
Copyright (c) 2003 Network Dweebs Corporation
http://www.nuclearelephant.com/projects/dspam/

LICENSE

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.

TABLE OF CONTENTS

General DSPAM Information

  1.0 About DSPAM
  1.1 Installation
  1.2 Testing
  1.3 Troubleshooting
  1.4 DSPAM Tools
  1.5 Driver Specific Tools

Advancced DSPAM functionality

  2.0 Linking with libdspam
  2.1 Configuring groups
  2.2 Participating in the global stats project

Miscellaneous

  3.0 Bugs, Ports, and the like 
  3.1 Known Bugs
  3.2 Adding the dspam logo button to your website
  3.3 CVS Access

1.0 ABOUT DSPAM

DSPAM is an open-source, freely available anti-spam solution designed to combat
unsolicited commercial email using deobfuscation techniques combined
with an advanced implementation of Baye's theorem of combined probabilities.
DSPAM is the first and only implementation of Baye's theorem to incorporate
token chains (or technically put, chained binary polynomial hashing).  The
result of DSPAM's efforts is an administratively maintenance free system 
capable of learning each user's email behaviors with very few false positives. 
DSPAM is among one of the more popular and successful attempts at truly 
accurate spam filtering, and is rapidly gaining a large support forum.  
Contributions to the project are welcome via the dspam-dev mailing list. 

DSPAM can be implemented in one of two ways:

1. The DSPAM mailer-agent provides server-side spam filtering, quarantine
box, and a mechanism for forwarding spams into the system to be automatically
analyzed.  Advanced features, such as opt-in/opt-out filtering, are supported.

2. Developers may link their projects to the dspam core engine (libdspam) in
accordance with the GPL license agreement.  This enables developers to
incorporate libdspam as a "drop-in" for instant spam filtering within their
applications - such as mail clients, other anti-spam tools, and so on.

Many of the foundational principles incorporated into this agent were 
contributed by Paul Graham's white paper on combatting SPAM, which can be 
found at http://paulgraham.com/spam.html.  Many new approaches have been layered
on top of the original core, thanks to the efforts of contributors to the DSPAM
project.

DSPAM is designed to run with a separate dictionary for each user, in
order to provide the best accuracy and learning potential.  A composite
dictionary can be created to seed a new user's mailbox, however over time
each user will develop their own statistics based on the behavior of their
email.  

The DSPAM Solution is split up into the following pieces:

DSPAM CORE ENGINE

The DSPAM core engine, also known as libdspam, provides all major spam
filtering functions.  The engine is linked to other dspam components (or
shells) to provide functionality.  The core engine is capable of being linked
in with any other application as a "drop-in" to provide spam filtering to
mail clients, other anti-spam tools, and other such type projects that
would benefit from its use.  Both static and shared versions are built by
libtool into .libs. 

libdspam provides a storage driver abstraction layer, enabling developers to 
easily change how information is stored on the system (for example Berkeley 
DB, MySQL, proprietary, etc.) with enough flexibility to write a storage
driver utilizing stone tablets and chisels.   

DSPAM AGENT

The DSPAM agent is a shell for libdspam providing server-side spam filtering.
The agent masquerades as the mail server's local delivery agent where 
it processes the email and then either delivers it using the real local 
delivery agent, or quarantines it.  The agent is where all learning and 
statistical calculation take place.  The agent also services SPAM processing: 
emails forwarded by the user that are SPAM.  This is critical to the 
learning operations of the DSPAM agent.  DSPAM can also be configured to
tag messages as spam, rather than quarantine them.

The MTA (sendmail, exim, qmail, etc) calls DSPAM with parameters identifying 
the destination user (DSPAM recognizes --user [user]").  DSPAM performs its 
internal calculations and will then call the real local delivery agent 
(mail.local, procmail, etc.) with any non-DSPAM-specific parameters passed to 
it, or move the message into the user's quarantine box.

When an email is delivered to the end-user, a DSPAM signature containing
a key to reversal information stored on the server is appended.  If the email 
is the forwarded into DSPAM as a false positive or missed spam, DSPAM will use 
this information to relearn the message.  All signatures are kept on the 
server so as not to flood emails.  As a result, the tools provided should be 
run to maintain each user's dictionary and signature databases.

CGI CLIENT

The CGI client is a very basic tool enabling a mail user to view their
quarantine box, reverse the occasional false positive, and delete
SPAMs permanently.  The CGI client works in conjunction with the DSPAM agent. 
It is possible to eliminate the quarantine box in lieu of an alternative
solution, such as client-filtering/forwarding.

TOOLS

Some basic tools which have been provided to manage dictionaries, automate 
corpus feeding, and create seeded [composite] dictionaries.  Some driver-
specific tools also exist for maintenance of any proprietary files or objects 
a particular driver may need to perform. 

1.1 INSTALLATION

UPGRADING

   ------------------------------------------------------
   IMPORTANT UPGRADE STEPS FOR USERS UPGRADING FROM < 2.7
   ------------------------------------------------------

   VERSION 2.7 IMPLEMENTED ENHANCED SECURITY

   Please read the section TRUSTED USERS SECURITY in this document for 
   instructions on setting up your trusted user file and your overrides file.  
   You MUST perform this step BEFORE upgrading to version 2.7 otherwise dspam
   will run in 'untrusted user' mode and function incorrectly.

   Version 2.7 also changed the rules for username identification:

  - DSPAM no longer recognizes -d to identify the user, but instead --user
    must be used.  This is to make things simpler in that:

    --user will never be passed onto the local delivery agent.
    -d, when specified, will always be bassed onto the local delivery agent.

    This means if you're upgrading from 2.6, you'll need to change your
    MTA configuration from:

    /path/to/dspam -d [user variable]
  
    to
  
    /path/to/dspam --user [user variable] -d %u

   Supplying %u instead of your normal user variable tells dspam to specify 
   the current user when calling the LDA.  This is important when the email
   is being delivered to multiple local users.
 
   In summary, we've decided to give DSPAM its own user flag to avoid confusion
   with your local delivery agent's user flag.  The %u flag has been created
   to insure you don't have to specify your user variable twice (some MTA's,
   such as sendmail) have a problem with this.  Technically, $u may be used,
   but %u is a safe alias that prevents your MTA from substituting the
   parameter with a list of users (bad).

   Once you have implemented the changes above, you may upgrade by building 
   the distribution and installing over the old one.  See the next section 
   (FRESH INSTALLATION) for more information.

   ------------------------------------------------------
   IMPORTANT UPGRADE STEPS FOR USERS UPGRADING FROM < 2.6
   ------------------------------------------------------

   If you are running a DSPAM version prior to 2.6, you will need to convert
   your databases to the new 64-bit database format before using this version 
   of DSPAM.  PREVIOUS VERSIONS OF 2.6-BETA ARE NOT COMPATIBLE with this
   version's databases.  If you are running with one such beta, you will need
   to convert from a backup of the 2.5-Release databases, or start with
   a fresh set of databases.

   To convert from a 2.5 database, you will need the dspam_convert tool
   available in the v2.6 distribution (no longer supplied with later
   distributions).  Once you have downloaded and built dspam_convert,
   the following operations may be used.  Be sure to keep a backup of your 
   2.5 databases until 2.6 goes production.

   cp -pr /etc/mail/dspam /tmp/dspam_crc32
   ./configure --with-userdir=/tmp/dspam_crc32
   make
   ./tools/dspam_convert
     (ignore conflicts, make sure all databases were converted successfully)
   shut down sendmail
   cp /tmp/dspam_crc32/*.dict /etc/mail/dspam
   rm -f /etc/mail/dspam/*.sig
     (the signature files contain only temporary information)
   chown root:mail /etc/mail/dspam/*.dict
   chmod 660 /etc/mail/dspam/*.dict
   make clean
   ./configure [options]
   make
   make install

   Once you have converted your databases (if necessary), all you should 
   have to do, unless otherwise stated in the CHANGE log or the website, is:

   ./configure [options]
   make
   make install

   Please see the section 'driver specific tools' for instruction on 
   building any driver-specific tools

FRESH INSTALLATION

First you will need to download a few prerequisite tools:

   Depending on which storage driver you want to use, you will need:
 
   libdb4_drv: Berkeley DB-4. 
   libdb3_drv: Berkeley DB-3.
   mysql_drv:  MySQL client libraries (and a server to connect to) 
   ora_drv:    Oracle Call Interface (and a server to connect to)

   The default is libdb4_drv.

   You can download Berkeley DB from http://www.sleepycat.com.  
   You can download MySQL from http://www.mysql.com.
   You can obtain more information about Oracle at http://www.oracle.com.

   Be sure the necessary libraries are available to root, the MTA user, and 
   the CGI user. The easiest way to do this is to copy them to /usr/lib or 
   /lib.

1. CONFIGURATION

   ./configure [options]

   The most widely used options for DSPAM include:

   --with-local-delivery-agent=PROG
   Specify an alternative local delivery agent, other than the one specific
   to your operating system.  If you are building on an unsupported platform,
   you will need to specify this.  You may use quotes if you wish to include
   additional commandline flags.  DSPAM will automatically relay the 
   commandline parameters it was initially given, with the exception of any
   DSPAM-specific parameters (such as --user, --corpus, etc.)

   Currently, DSPAM has a default local delivery agent selected for Linux, 
   FreeBSD, and Solaris platforms.

   NOTE: When specifying a series of arguments, you will need to use quotes
   around PROG.  You may also use the $u identifier to specify that you
   with DSPAM to place the destination user's ID in the corresponding space
   in the arguments list. For example: 

       --with-local-delivery-agent="/path/to/lda -d \$u" 

   Where $u will be replaced by the destination user prior to calling the LDA.
   This could potentially cause problems, however, if your MTA requires the
   user argument list to come last, which is why DSPAM, by default, will allow
   you to set this in the MTA configuration.

   NOTE: be sure to escape the $ in $u. Only do this when specifying $u on the 
   commandline.  This will prevent $u from being overwritten with the shell's 
   environment variable 'u'.

   --with-storage-driver=DRIVER
   Specify an alternative storage driver.  A storage driver is a driver
   written specifically for DSPAM to store tokens, signature data, and
   perform other proprietary operations.  The default driver is libdb4_drv,
   which incorporates Berkeley DB-4.  As of the time this README was written, 
   the following drivers have been provided:

     libdb4_drv: Berkeley DB4 Library
     libdb3_drv: Berkeley DB3 Library
     mysql_drv:  MySQL Drivers
     ora_drv:    Oracle Drivers (BETA)

   You may also need to use some of the driver-specific configure flags
   (discussed later).

   --with-quarantine-agent=PROG
   By default, DSPAM automatically quarantines spams in its internal
   user quarantine box.  If you wish to override this default behavior,
   however, you may do so by specifying your own quarantine agent.  The same
   notes from the --with-local-delivery-agent option apply here. 

   --enable-signature-attachments
   Instead of storing the DSPAM signatures on the server (which could take
   up considerable disk space), this option will cause DSPAM to rewrite
   each message to include a dspam.dat attachment, which contains all of
   the tokens used to calculate the original message.  When the spam or
   false positive is processed back into the system, this signature will
   be read.  May increase bandwidth on an average between 2k-32k per
   message, depending on the original message's size.

   NOTE: This option doesn't work correctly with mail clients that quote an
   embedded, forwarded message (such as some or all versions of elm) and 
   should only be used on networks where all clients can properly understand 
   an embedded multipart message (Outlook, Ximian Evolution, Etcetera), and 
   forward the attachment as an attachment instead of quoted text.  In othe  
   words, this breaks a lot of stuff if you're not on a standardized GUI-based 
   client network.  Server-side signatures is still the most reliable method 
   and works for all known clients.

   --disable-trusted-user-security
   Administrators who wish to disable trusted user security may do so by 
   using this configure flag.  This will cause DSPAM to treat each user as
   if they were "trusted" which could allow them to potentially execute
   arbitrary commands on the server via DSPAM.  Because of this, administrators
   should only use this option on either a closed server, or configure their 
   DSPAM binary to be executable only by users who can be trusted.  This 
   option SHOULD NOT be used as a solution to your MTA dropping privileges
   prior to calling DSPAM.  Instead, see the TRUSTED SECURITY section of this
   document.

   --with-signature-life=DAYS
   Specifies the length (in days) a signature should remain stored on the
   server.  The default is 14 days.  This value should accurately represent
   the maximum amount of time a user would need to identify and forward
   a missed spam, or mark a false positive.  Consider vacations.

   --with-userdir=DIR
   Specify an alternative storage directory for dspam user information.  The
   default is /etc/mail/dspam.

   --disable-chained-tokens
   Disables chained tokens.  Please read the white paper on chained tokens
   before you disable them, as using them makes DSPAM significantly more
   accurate.  Only use this option if disk space is a serious issue.

   --enable-debug
   Turns on debugging output to /tmp/dspam.debug and /tmp/dspam.messages

   --enable-verbose-debug
   Turns on extremely verbose debugging output to /tmp/dspam.debug and
   /tmp/dspam.messages.  Never use this flag in production!

   --prefix=DIR
   Specify an alternative root prefix for installation.  The default is
   /usr/local

   --enable-large-scale
   Switch for large-scale implementation.  Currently only affects the filesystem
   layout for dspam.  User data will be stored as $USERDIR/u/s/user instead
   of $USERDIR/user

   --enable-alternative-bayesian
   Switches from the standard Bayesian calculation mode to Brian Burton's
   alternative Bayesian algorithm.  The differences are:
     - 27 Samples are used instead of 15
     - Tokens appearing more than once may take up to 2 slots in the
       calculation.  This is ideal when there is very limited data

   --enable-chi-square
   Switches from the standard Bayesian calculation mode to Robinson's 
   Chi-Square algorithm.  The differences are:
     - 27 Sapmles are used instead of 15
     - The entire combination algorithm is different.  See:
       http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html
       for more information.

   NOTE: Chi-Square is a very controversial algorithm, and may provide
         significantly better, OR SIGNIFICANTLY WORSE results.  In our 
         fundamental internal tests, Bayesian algorithms scored much
         more accurately than Chi-Square.  Use at your own risk.
       
   --disable-bias
   When bias is disabled, dspam no longer biases the statistics in favor of
   innocent mail, but measures both spam and innocent tokens equally in the 
   calculation equally.  This may provide more effective spam filtering,
   but could potentially weaken false positive protection.

   --enable-spam-delivery
   When messages are marked as spam, they are tagged using X-DSPAM headers.
   Using this configure flag will cause DSPAM to deliver spams to the user's
   mailbox (where they should be filtered using the client's own mechanism)
   instead of quarantining them.  This presents a potential learning curve
   for end-users as they will still need to forward in missed spams and false
   positives (instead of just moving them to the correct mailbox).

   --enable-homedir-dotfiles
   When enabled, instead of checking for $USERDIR/$USER[.nodspam|.dspam],
   DSPAM will check for a .nodspam|.dspam file in the user's home directory.
   These two dotfiles are used for opt-out or opt-in filtering.
                                                                                
   --enable-opt-in
   Causes DSPAM to filter mail only for users with a .dspam dotfile.  The 
   default is opt-out, which requires a .nodspam file to exist to bypass 
   filtering.

   DRIVER SPECIFIC:

   libdb4_drv:
     --with-db4-includes=DIR
     Specify a path to the Berkeley db4 includes

     --with-db4-libraries=DIR
     Specify a path to the Berkeley db4 libraries

   libdb3_drv:
     --with-db3-includes=DIR
     Specify a path to the Berkeley db3 includes

     --with-db3-libraries=DIR
     Specify a path to the Berkeley db3 libraries
     (Currently links to -ldb3, to you may need to symlink libdb-3.3.so to
      libdb3.so if it doesn't exist)

   mysql_drv:
     --with-mysql-includes=DIR
     Specify a path to the MySQL includes

     --with-mysql-libraries=DIR
     Specify a path to the MySQL libraries
     (Currently links to -lmysqlclient, also -lcrypto on some systems)

     --with-virtual-users
     Tells DSPAM to create virtual user ids.  Use this if your users don't
     actually exist on the system (e.g. in /etc/passwd if using a password file)

     NOTE: Please see the file tools.mysql_drv/README for more information
     about configuring the mysql_drv storage driver.

   ora_drv:
     --with-oracle-home=DIR
     Specify the Oracle Home (or client home)

     --with-virtual-users
     Tells DSPAM to create virtual user ids.  Use this if your users don't
     actually exist on the system (e.g. in /etc/passwd if using a password file)                                                                                
     NOTE: Please see the file tools.ora_drv/README for more information
     about configuring the ora_drv storage driver.

2. BUILDING AND INSTALLING

   After you have run configure with the correct options, build and install
   DSPAM by performing:

   make && make install

   If you are a developer wanting to link to the core engine of dspam,
   libdspam will be built during this process.  Please see the
   example.c file for examples of how to link to and use libdspam.  Static
   and dynamic libraries are built in the .libs directory.

3. PERMISSIONS

   After install, the USERDIR will have been created for you automatically
   (the default is /etc/mail/dspam).  Insure the permissions of the directory
   are writable by both your MTA and CGI users.    

   You may need to add root, your MTA user, and your CGI user to the
   directory's [mail] group in /etc/group.  The MTA user is usually 'daemon' 
   or 'smmsp' although on FreeBSD the default is 'mailnull'.  This is very 
   important, as your MTA user needs to be able to lock and work with files.

   IMPORTANT!!!

   FreeBSD's mail.local changes its effective uid, and so in order to use it
   dspam must be installed as setuid root to work on the commandline properly.
   This is done automatically on install.

   DSPAM CGI

   The CGI will need to function in the same group as the dspam agent.  The
   best way to do this is to create a separate virtualhost specifically for
   the CGI and assign it to run in the MTA group.  If you are using
   procmail, DSPAM may also need to be setuid root in order to deliver
   false positives through the CGI.  Be sure not to open up the permissions
   of the dspam agent where another user or cgi could call dspam with
   setuid or setgid permissions as this is insecure.

   TRUSTED USERS SECURITY

   DSPAM has tighter security for untrusted users on the system, to prevent
   them from being able to spoof other users or specify their own passthru
   arguments to potentially hijack the local delivery agent.  This method
   of security has been implemented due to the fact that some implementations
   (such as those using procmail) may require the DSPAM agent to be setuid or
   setgid.

   The trusted.users file should be created in $USERDIR (defaulted to
   /etc/mail/dspam).  This file should contain a list of trusted users who 
   should be allowed to set the dspam user, passthru parameters, and other 
   information that would be potentially dangerous for a malicious user to 
   be able to set.  The file should contain one username per line, and will
   generally the usernames of the MTA and CGI users.  Example:

   root
   smmsp
   daemon
   cgi
   mailnull

   Where cgi represents the special CGI user you configure Apache to
   run your dspam.cgi as.

   Be sure to examine /tmp/dspam.debug to insure that you don't get any
   untrusted user warnings when submitting spam or a false positive, as both
   of these actions frequently call dspam from a different user than
   standard mail delivery.

     NOTE ON CGI USERS:  It is far more secure to create a separate virtual
     host for the DSPAM CGI running as a different user than any other
     scripts on the system.  This avoids giving trusted user privileges to
     another CGI.  If you do this, be sure to add the CGI user to the trusted
     users list.

   If you are using an MTA that changes its userid before calling DSPAM to
   match the destination user, you should NOT add each user to the trusted
   users file, but instead configure a preset commandline.  DSPAM will see
   that the user is not trusted and automatically set their DSPAM user id
   and optionally the passthru local delivery agent arguments.
   
   To override an untrusted user's passthru local delivery agent arguments
   (arguments which could be used to hijack the local delivery agent to gain
   privileged access to the system) you will need to set up a file called 
   untrusted.mailer_args in the same directory ($USERDIR).  The first line 
   should contain the path to the local delivery agent followed by a list of 
   all the LDA arguments to pass through (including a user identity flag if 
   necessary).  This file's information will override any passthru commandline 
   parameters specified by the user.  For example:

   /bin/mail -d $u

   The variable $u informs DSPAM that you would like the destination username
   to be used in the position $u is specified, so when DSPAM calls your LDA
   for user 'bob', it will call it with:

   /bin/mail -d bob

   NOTE: In the event that ALL of the following are true:

    - Your MTA performs a setuid() to the destination user prior to calling
      DSPAM

    - There are additional _dynamically assigned_ parameters that must be
      passed to DSPAM which cannot be specified in configuration

    - The local delivery agent has no potentially dangerous commandline
      options, or you are placing a wrapper around the local delivery
      agent

    Then you may want to remove the untrusted.mailer_args file all together.
    If the file cannot be found, dspam will permit the user to specify their 
    own passthru arguments to the preconfigured LDA (with some basic sanity 
    checking) which COULD POTENTIALLY BE INSECURE if improperly set up..  It
    is strongly recommended you use this file to override the user.

    DSPAM warns you (over log record) when unable to open
    untrusted.mailer_args file.

    If you don't want to see this warning then make untrusted.mailer_args
    file exists but empty.

4. MAIL SERVER CONFIGURATION

   Modify your mail server configuration to use /usr/local/bin/dspam as the
   default local delivery agent.  

   SENDMAIL EXAMPLES
   If you are using sendmail, modify the Mlocal tag, as shown below:

Mlocal,		P=/usr/local/bin/dspam, F=lsDFMAw5:/|@qfSmn9, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL,
		T=DNS/RFC822/X-Unix,
		A=dspam --user $u -d %u

   You may also need to add this to DSPAM_ARGS in dspam.cgi:

   $CONFIG{'DSPAM_ARGS'} = "--user $ENV{'REMOTE_USER'} -d \$u";

   If you are using procmail with sendmail, you can preserve the existing
   procmail flags with the exception of LMTP.  For example:

Mlocal,		P=/usr/local/bin/dspam, F=lsDFMAw5:/|@qSPfhn9, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL,
		T=DNS/RFC822/X-Unix,
		A=dspam -t -Y -a $h --user $u -d %u

   The --user flag identifies the user to DSPAM, and does not actually get
   passed through to the LDA.  Be sure to change "-d $u" to "-d %u" to let
   DSPAM supply the username to the LDA.

   NOTE: be sure to escape the $ in $u ONLY when specifying it on the 
   commandline.  This will prevent $u from being overwritten with the shell's 
   environment variable 'u'.
                                                                                
   Finally, if you are using sendmail restricted shell (smrsh), you will
   need to either disable it, or add /usr/local/bin/dspam to the allow list. 
   Some people have experienced problems using smrsh with DSPAM.

   It is required that dspam be called with the --user flag followed by the user
   name of the local recipient.  

   EXIM
   To integrate DSPAM with exim, you'll need to create a new director in the
   exim configuration.  First, add the following code to the directors :

spamscan:
  no_verify
  condition = "${if and { {!eq {$received_protocol}{spam-scanned}} } {1}{0}}"
  driver = localuser
  transport = spamcheck

  Then add the following code to the transports:

spamcheck:
  driver = pipe
  command = /usr/local/bin/dspam --user $local_part -d %u 
  user = mail
  group = mail
  return_path_add = false
  log_output = true
  return_fail_output = true

   Finally, you will need to configure your local delivery agent into dspam.
   This may end up calling exim again, if you are using maildir
   format for example.  An example of calling back to exim for delivery might
   look like:

   ./configure --with-local-delivery-agent="/usr/sbin/exim -oMr spam-scanned"

   Of course, any parameters you pass to DSPAM will also get passed onto
   the local delivery agent, so you can safely locate the parameters in
   your exim configuration incase you want to change them later:

   command = /usr/local/bin/dspam --user $local_part -d $local_part -oMr spam-scanned

   then configure with just:

   ./configure --with-local-delivery-agent=/usr/sbin/exim

   This was tested under Debian Sarge with Exim 3.36-6.

   COURIER

   Step 1: Configuring Courier to use DSPAM

   To set Courier to use DSPAM as its local delivery agent, you'll want to
   change the DEFAULTDELIVERY parameter.  In the file /etc/courier/courierd,
   find the line for DEFAULTDELIVERY and change it to:

   DEFAULTDELIVERY="| /usr/local/bin/dspam --user $USER"

   Step 2: Setting DSPAM to use Maildrop as LDA
   Maildrop is Courier's native equivalent of procmail.  To configure DSPAM
   to use it, use the following configure flags:

   --with-local-delivery-agent="/usr/bin/maildrop -d \$u"

   You'll also need to configure untrusted.mailer_args as Courier drops to the
   uid of the destination user and changes to that users homedir before calling
   the LDA.  Accomplish this by using the following parameters in your
   untrusted.mailer_args file:

   /usr/bin/maildrop -d $u

   Step 3: Configuring DSPAM aliases for missed spam/false positives

   The aliases for missed spam/false positives should be added to the file
   /etc/courier/aliases/system.  After adding new aliases, run 'makealiases'
   and restart Courier.

   OTHER MAIL SERVERS
   Consult your mail server's documentation on how to replace the local 
   delivery agent.  It may be necessary to recompile the software.  Worst 
   case scenario, you may have to symlink dspam to the filename of the local 
   delivery agent, and rename the local delivery agent.  If you do this, be 
   sure the change is reflected in your configuration of dspam using
   the --with-local-delivery-agent= flag.  We have reports of dspam working 
   with postfix, and will post documentation for configuring it as soon as 
   its available.

5. ALIASES

   For each user, you will need to create an email address the user can
   send spam to, so that DSPAM can analyze and learn.  The easiest way to
   do this is to create a new alias.  For example:

   spam-bob:      "|/usr/local/bin/dspam --user bob --addspam"

   You will end up having one alias per mail user on the system.  Be sure the
   aliases are unique and each username matches the name after the --user flag.
   A tool has been provided called dspam_genaliases.  This tool will read the
   /etc/passwd file and write out a dspam aliases file that can be included
   in your master aliases table.  

   To report spams, the user should be instructed to forward each spam to
   spam-user@yourhost

   If you will be using the --enable-spam-delivery mechanism, you will also
   need an alias to forward false positives into.  The following example should
   suffice:

   fp-bob:	"|/usr/local/bin/dspam --user bob --falsepositive"

   It doesn't really matter what you name these aliases, so long as the flags
   being passed to dspam are correct for each user.  It might be a good idea
   to create an alias custom to your network, so that spammers don't forward
   spam into it.  For example, fp-yourcompany-bob or something.  

6. DRIVER SPECIFIC TOOLS

   Depending on the storage driver you are using, there may be tools specific
   to that driver that will be built.
   
THE CLEANUP AND PURGE TOOLS

   CLEANUP

   You should configure dspam_clean to run under cron nightly.

   This clean tool will read all signature databases and purge signatures that
   are older than 14 days (configurable).  Without this tool, old
   signatures will continue to pile up.  A cron like the one below should
   suffice.  Be sure the user running cleanup has full read/write permissions
   on the DSPAM data files.

   0 0 * * * /usr/local/bin/dspam_clean

   PURGE

   Depending on which storage driver you choose, there may be a driver-specific
   purge tool designed to keep user dictionaries clean, small, and optimized by 
   deleting unimportant tokens and rewriting the database from scratch every 
   time it is run.  Unimportant tokens are tokens that have fewer than the 
   minimum number of hits to be assigned a true probability.  The minimum is 
   presently Spam Hits + 2 * Innocent Hits < 5.  This tool can be run as often 
   as you like, but if implemented on a large-scale, should only be run when 
   the system is fairly acquiesce as it locks the user it is processing, 
   creating potential delays in mail delivery.  Depending on how much disk 
   space you wish to conserve with this tool, you may run it every day, week, 
   or two weeks.  To configure this tool in cron, use an entry similar to below:

   0 0 5,10,15,20,25,30 * * * /usr/local/bin/libdb4_purge

   The above entry will run the libdb4-specific purge tool on the 5th, 10th, 
   15th, and so on.  It is not absolutely necessary to run the purge tool 
   (especially if you wish to collect all available data over a long term 
   period), but is healthy in maintaining a smaller set of user dictionaries 
   containing only interesting tokens.  Without it, your user's dictionaries 
   could easily grow in size over time. 

   Obviously if you are using a SQL-based driver, you will not need to compress
   files, but may want to run some basic SQL commands to delete unused tokens, 
   etc.

THE CGI CLIENT

   The CGI client (dspam.cgi) can be run from any executable location on
   a web server, and detects its user's identity from the REMOTE_USER
   environment variable.  This means you'll need to use HTTP password
   authentication to access the CGI (Any type of authentication will work,
   so long as Apache supports the module).  You'll want the usernames to match
   the actual username on the system.  A copy of the shadow password file
   will suffice for authentication. 

   The accompanying files in the cgi/ folder should be copied into the same
   location as dspam.cgi, as they are needed by the tool to generate output.
   Be sure to copy the templates and graphics into the cgi-bin as well.

   NOTE: Some authentication mechanisms are case insensitive and will 
   authenticate the user regardless of the case they type it in.  DSPAM, 
   on the other hand, is case sensitive and the case of the username used
   will need to match the case on the system.  If you suffer from this
   authentication problem, and are certain all of your users' usernames are
   in lowercase, you can add the following line of code to the CGI right
   after the call to &ReadParse...

   $ENV{'REMOTE_USER'} = lc($ENV{'REMOTE_USER'});

   You will also want to set the LARGE_SCALE variable in dspam.cgi if you are
   compiling with --enable-large-scale.  This will adjust the CGI to use the
   correct filesystem paths for dspam's large-scale filesystem implementation.

   Please see the section on permissions for more information about setting up
   the CGI's group and agent permissions.

1.2 TESTING

  Most software packages are supplied with a test suite to determine if the
  software is functioning properly.  Since dspam's correct function relies 
  primarily on having the correct permissions and mail server configuration,
  a test script fails to provide the level of testing required for such a
  package.  The following exercise has been provided to test dspam's correct
  functioning on your system.  This exercise does not test the CGI, but only
  the core dspam agent.
  
  Before running the test, you should have completed section 1.1's instructions
  for compiling and installing dspam as well as configured your mail server
  to support dspam.

  1. Create a new user account on your system.  It is important that this be a 
  new account to prevent any unrelated email from being delivered during 
  testing.  Be sure to configure a spam alias for the test account.

  2. Send a short (10 words or less) email to the account, and pick it up 
  using your favorite mail client.  

  3. Run dspam_stats [username] on the server.  You should see a value of 1 
  for "TI" or "Total Innocent" as shown below:

  dspam-test            0 TS       1 TI       0 TM       0 FP

  If you receive an error such as "unable to open /etc/mail/dspam/... for
  reading", then the dspam agent is not configured correctly.  The problem
  could lie in either your mail server configuration or one or more of the
  permissions on the directory or agent.  Check your configuration and
  permissions, and repeat this step until the correct results are experienced.

  4. Run dspam_dump [username] to get a complete list of tokens and their 
  statistics.  Each token should have an I: (innocent) hit count of 1. The 
  tokens will be represented as 64-bit values, for example:

3126549390380922317              S:    0  I:    1  LH: Mon Aug  4 11:40:12 2003
13884833415944681423             S:    0  I:    1  LH: Mon Aug  4 11:40:12 2003
14519792632472852948             S:    0  I:    1  LH: Mon Aug  4 11:40:12 2003
8851970219880318167              S:    0  I:    1  LH: Mon Aug  4 11:40:12 2003

  5. Forward the test message to the spam alias you've created for the test 
  account.  Provide enough time for the message to have processed.

  6. Run dspam_stats [username] on the server again.  Now, the value for TI 
  should be zero and the value for TM (total misses) should be 1 as shown
  below:

dspam-test            0 TS       0 TI       1 TM       0 FP

  If this is not the case, check the group permissions of the dspam agent as
  well as the permissions your MTA uses when piping to aliases.
  
  7. Run dspam_dump [username] again.  make sure that _EVERY_ token now has an 
  I: of zero and a S: of 1:

3126549390380922317              S:    1  I:    0  LH: Mon Aug  4 11:44:29 2003
13884833415944681423             S:    1  I:    0  LH: Mon Aug  4 11:44:29 2003
14519792632472852948             S:    1  I:    0  LH: Mon Aug  4 11:44:29 2003
8851970219880318167              S:    1  I:    0  LH: Mon Aug  4 11:44:29 2003

  If you have some tokens that do not have an S: of 1 or an I: of 0, the dspam
  signature was not found on the email, and this could be due to a lot of
  things.

1.3 TROUBLESHOOTING

    Problem: I get an error similar to 'cannot find -ldb-4.1'
   Solution: Your compiler can't locate your db libraries.  Try installing
             them into /usr/lib, or add them to your (and your MTA's)
             LD_LIBRARY_PATH.  You may also use --with-db4-includes and
             --with-db4-libraries as configure flags.  If you are using libdb3,
             use the db3-specific configure parameters.
 
    Problem: Dictionary isn't updating
   Solution: Check the file permissions of both the .dict and the .mbox files.
             These files will need to be writable by the dspam agent as well
             as the CGI user.

    Problem: No files are being created in the user directory
   Solution: Check the directory permissions of the directory.  The user 
             directory must be writable by the user the dspam agent is running
             as as well as the CGI user.

    Problem: False positives are never being delivered
   Solution: Your CGI most likely doesn't have the privileges required by
             the LDA to deliver the messages.  Make sure the CGI user is in
             the correct group.  Also consider setting the dspam agent to
             setuid or setgid with the correct permissions.

1.4 DSPAM TOOLS

  A few miscellaneous tools have been provided to make DSPAM management 
  a bit easier.  These tools include:

  dspam_corpus - Used to feed an existing corpus of mail (in mailbox format)
    into the dspam system.  
    Syntax: dspam_corpus [username] [filename] [--addspam]
    where username is the username of the user to apply the corpus to,
    filename represents the filename of the mailbox, and the optional -a flag
    to specify if this corpus is known spam (to add as spam into the user's
    dictionary).  
 
  dspam_dump - Dumps a DSPAM dictionary. This can be used to view the 
    entire contents of a user's dictionary, or used in combination 
    with grep to view a subset of data.  Syntax: dspam_dump [username] 
    where username is the DSPAM user's username.

  dspam_clean - Used in conjunction with the SERVER_SIDE_SIGNATURE mode
    of operation where signatures are stored locally in a database.  The
    clean tool should be run nightly and is responsible for deleting old
    signatures (>14 days by default) from the datafiles.

  dspam_purge - Processes every user dictionary on the system and keeps each
    dictionary clean of unimportant tokens over a long term period.  This tool 
    deletes non-qualifying tokens that have not been hit after a certain 
    duration.  The defaults specified in config.h are:

    - Delete any tokens with a (SH)+(2)(IH) value of less than 5 after 
    not being hit for 60 days
    - Delete any tokens with a (2)(IH) value of less than 5 and zero
    SH after not being hit for 30 days
    - Delete any tokens with an IH of 1, SH of zero after not being
    hit for 15 days.

  dspam_stats - Displays the spam statistics for one or all users on the system.
    Syntax: dspam_stats [username].  If no username is provided, all users 
    will be displayed.  Displays TS (Total Spams), TI (Total Innocent), TM
    (Total Spam Misses) and FP (Total False Positives).  Spam misses are
    spams that were forwarded in by the user.  To calculate the total number
    of spams caught by DSPAM, subtract TM from TS. 

  dspam_genaliases - Reads the /etc/passwd file and outputs a dspam aliases
    table which can be included in the master aliases table.

  dspam_merge - Merges multiple users' dictionaries together into one user's
    dictionary (does not affect the merge users).  This can be used to create
    a seeded dictionary for a new user, or to copy a single user's dictionary
    to a new file.

  dspam_ngstats - A global statistics tool for those who wish to participate
    in global statistics tracking.  This tool reports your system's spam
    totals to a master server on a daily or weekly basis, where they are
    merged with the statistics of other participating systems to provide
    global statistics of DSPAM's efficiency and how many spams DSPAM has
    put a stop to.  See section 2.2 for more information.

2.0 LINKING WITH LIBDSPAM

  Developers are able to link to the DSPAM core engine (libdspam) to provide 
  "drop-in" spam-filtering for their applications.  Examples of the libdspam
   API can be found in the example.c file included with this distribution.

  To link to libdspam, follow the instructions in section 2.1 for compiling
  and installing dspam.  When dspam is compiled, the libdspam static and
  shared libraries are also built.  This library contains all the functions 
  necessary to use dspam's filtering in your application. 

  Your application will also need to link to Berkeley db4.  If you would
  prefer to use your own proprietary storage mechanisms, modify the 
  localdb.c code and recompile.  

  To build with the dspam API, you will also need the following header
  files: 

  libdspam.h         - library API
  libdspam_objects.h - structures and macros used
  buffer.h           - used internally
  decode.h           - used internally
  lht.h              - used internally
  nodetree.h         - used internally

  If you are interested in linking libdspam with your project and have 
  questions or concerns, please contact the dspam-dev mailing list.

2.1 CONFIGURING GROUPS

  Group dictionaries enable a group of users with similar email behavior to
  share the same dictionary while still maintaining a private quarantine box.
  Groups can have both positive and negative effects on accuracy.  If a group
  consists of users with similar, predictable email behavior, the users in
  the group can benefit from a larger dictionary of spam and faster filtering 
  ramp (especially for newcomers in the group).  If a group consists of users 
  with different email behavior, however, the users in the group will experience
  poor spam filtering and a higher number of false positives.

  With this in mind, to create groups, you'll want to create a file with the
  filename 'group' located in the DSPAM user directory.  The default is
  /etc/mail/dspam/group.  The format of the file should look like this:

  group1:user1,user2,user3
  group2:user4,user5,user6

  DSPAM will read this file upon startup and determine if the user fits into
  any particular group.  If so, the dictionary used will then become the
  group dictionary rather than the individual user's dictionary.  Each user
  will continue to have their own quarantine box and signature file for
  privacy.
  
  Use the dspam_stats tool to keep an eye on the effectiveness of each group.
  If a group experiences poor performance, find the users whose email behavior
  is inconsistent with that of the group and remove them from the group.

2.2 PARTICIPATING IN THE GLOBAL STATS PROJECT
                                                                                
  A small tool has been provided for administrators wishing to participate in
  the dspam global stats project.  The dspam_ngstats tool may be run nightly,
  weekly, or at any time interval you choose, and will report the global stats
  of your entire installation to the dspam stats collector, where they will
  be merged with the stats from other dspam implementations.  This will provide
  the community with one global statistic for the entire dspam project.  If you
  wish to participate, simply add dspam_ngstats to your crontab.  No information  about your users or implementation is broadcasted, only your system's IP
  address (for tracking purposes) is recorded along with your global stats.
                                                                                
                                                                                
  NOTE: If your server does not have a static public ip address, you will need
  to configure a unique id inside dspam_ngstats.c.  Set the definition of
  NGSTATS_UID from "" to "my_unique_id".  Your unique id may be comprised of
  underscores, periods, and alphanumeric characters.  Any invalid id's will
  cause the stats for that machine to be ignored.
                                                                                
3.0 BUGS, PORTS, AND THE LIKE

  Please report any questions, bugs, suggestions, and the like to the 
  dspam-users mailing list.  See the project website for details.

  If you port DSPAM to another platform, or would like to submit changes to
  the distribution, please email a diff along with any other pertinent 
  information to the dspam-dev mailing list.

  If you like DSPAM and want to buy the author pizza (or a ferrari),
  paypals may be sent to jonathan@nuclearelephant.com.

  Thanks =)

3.1 KNOWN BUGS

  - DSPAM presently does _not_ handle a mass forward of emails, but only one
    forward at a time.  Be sure to tell your users not to select multiple
    messages and forward them...this results in a single message being sent
    into DSPAM instead.  Users should individually forward each spam.  

3.2 ADDING THE DSPAM LOGO BUTTON TO YOUR WEBSITE

  A small button has been included for those who would like to advertise dspam
  on their web page.  To use, copy the graphic (dspam-button.gif) into your
  web page's directory and use the following code wherever you'd like the
  button displayed:

  <A HREF="http://www.networkdweebs.com/software/dspam/">
  <IMG BORDER=0 SRC="dspam-button.gif"></A>

3.3 CVS ACCESS

  The DSPAM source tree can be downloaded via read-only cvs access using the
  following commands:

  cvs -d :pserver:cvs@cvs.nuclearelephant.com:/usr/local/cvsroot login
  cvs -d :pserver:cvs@cvs.nuclearelephant.com:/usr/local/cvsroot checkout dspam 

