November 2, 2004
amavisd-new-2.2.0 release notes
SECURITY:
- modified MIME entity traversal to include MIME container parts
(e.g. multipart/*) as pseudo parts. Such pseudo-parts do not carry
any body contents but do show up (with their MIME content type only)
in the tree structure as seen by banned names checks. This makes it
possible to specify more complex banned rules based on the placement
of leaf nodes within or outside of mime multipart containers.
This also re-enables the possibility to check such MIME wrappers for
banned MIME Content-Types (most notably for the message/partial and
message/external-body), which was lost with the change of internal
representation of mail parts in version 2.0 (amavisd-new-20040701);
- preserve original zip archive for virus scanners to see, if the archive
contains any zero-length members (Archive::Zip module chooses not to
extract members with declared zero size, even if the size does not
correspond to the actual size); based on a patch by Dirk Datzert;
- add tests to mime_traverse to verify that the files produced by MIME::Parser
really do exist and are readable; (and sent a patch adding I/O checks
for MIME::Tools to its maintainer; please use MIME::Tools 5.414 or later
from CPAN to avoid possibility of full /tmp partition to cause infected
or spam mail to pass through);
- changed recommendation in INSTALL to choose the latest version of
MIME-Tools from CPAN - the 5.415 at the time of this writing;
- do_unrar: recognize encrypted entire archives (not just their individual
members), and flag mail as undecipherable;
- recognize file(1) report/^MS Windows\b.*\bDLL\b/ as short type 'dll';
add 'dll' to example patterns in amavisd.conf and amavisd.conf-sample;
add name extension '.cpl' to the list of basic banned names;
INCOMPATIBILITY:
- incompatible change: the default value for $recipient_delimiter is
now undef and no longer '+'; adding address extensions must now be
explicitly enabled;
- minor change in the default X-Virus-Scanned: header field, see below;
OTHER CHANGES SINCE 2.1.2:
- rewritten LDAP modules, by Michael Hall;
- improvements in handling of double errors (errors that occur while
handling consequences of some previous error); be permissive on failures
in DESTROY methods; prevent some cases for child process not being able
to sign off from the nanny database;
- enforce $max_requests more strictly, dropping SMTP session after task
count is exceeded by one; this is in response to the new smtp session
caching behaviour in Postfix, which is now much more persistent in keeping
the session open on a busy mailer; although dropping session at the server
side is discouraged by RFC 2821, this change was recommended on the
postfix-users mailing list;
- added a site-wide mail size limit $smtpd_message_size_limit, and a
per-recipient mail size limit lookups @message_size_limit_maps along with
SQL and LDAP fields, making it possible to reject mail based on its size.
The list of lookup tables maps recipients to mail size in bytes, e.g.:
@message_size_limit_maps = ({'user1@example.com' => 20_000_000,
'user2@example.com' => 15*1024*1024,
'user3@example.com' => 0, # uses global limit
'.example.com' => 10*1024*1024 });
$smtpd_message_size_limit = 20*1024*1024; # global limit if nonzero
A value of 0 or undef disables the check and is a default. A per-recipient
limit is bound on the high side by the $smtpd_message_size_limit, and on the
low side by 64kB, which is a minimal allowed size limit imposed by RFC 2821.
This limiting really belongs to MTA and is only partially supported here
(no admin notifications, no quarantine, no final_*_destiny configurability).
It is mostly provided here to be able to place some sanity limit on runaway
or malicious clients, or if someone insists on using amavisd-new in a
pre-queue filtering setup; suggested by Tuomo Soini;
- internal: add new object $msginfo->mail_text_fn to hold the file name
of the original mail, decouple the location of file email.txt from the
temporary directory, which was implied until now. This is presently needed
for the Courier interface. Add optional attribute 'mail_file' to the
AM.PDP protocol, see README.protocol;
- in banned parts descriptor strings which are used in banned name checking,
provide a 'T=empty' short type for empty mail parts, including for empty
MIME parts (instead of omitting T=... altogether). This can be used in
banned rules to test for empty parts, generally or restricted to empty
MIME parts; suggested by Ricardo Stella and Stephane Lentz;
- a banned lookup result (which is interpreted as boolean for most purposes:
zero or empty for false, anything else for true) may give a result 'DISCARD'
(which is true as well), which will disable DSN if it turns out the mail
was blocked by such banned rule (akin to viruses_that_fake_sender and
spam_dsn_cutoff_level). Here is an example rule in $banned_namepath_re:
[ qr'(?# BLOCK EMPTY MIME PART APPLICATION/OCTET-STREAM )
^ (.*\t)? M=application/octet-stream \t(.*\t)* T=empty (\t.*)? $'xmi
=> 'DISCARD' ],
Suggested by Ricardo Stella and Stephane Lentz;
- fix 'Insecure dependency in unlink' in sub files_to_scan which could
happen when some decoder left non-regular files in the directory;
reported by Andrzej Kukula;
- bug fix: only insert LDAP and SQL lookups objects into lists of maps at the
first task of a child process, later tasks should not insert duplicates;
- new subroutine do_ar and new configuration variable $ar to handle
standard Unix archives, including Debian binary packages;
- recognize a Unix archive and give it a short type .a;
- recognize a Unix relocatable binary and give it a short type .o;
- do not penalize SMTP status "554 5.1.1 Error (DATA): no valid recipients",
the situation arises regularly when pipelining is in effect and all
recipients happen to be rejected;
- protect spaces and newlines when logging broken Message-ID and
Resent-Message-ID header fields (macros %m and %r) to facilitate log
parsing; protect newlines in logged Subject header field (macro %j);
parsing difficulty pointed out by Chris Lee;
The present solution uses =XX (hex) encoding and is a quick and dirty
fix. A cleaner solution to avoid double sanitation of special characters
is needed, but would involve a deeper reorganization;
- updated example list of "banned extensions - long" in amavisd.conf
and amavisd.conf-sample;
- change the default lock file name from "$MYHOME/amavisd-$$.lock" to
"$MYHOME/amavisd.lock", to avoid inventing a new name at every restart
and leaving old files behind; pointed out by Dale Walsh and Martin Orr;
- updated av entry for nvcc (Norman Virus Control v5 / Linux) to include
statuses 10,11, and 2,14 to the status lists according to documentation;
password protected or corrupted archive (status 11) was not recognized
as non-infected status; thanks to Michael Ramke of Norman Data Defense
Systems GmbH;
- updated DrWebD entry to recognize and ignore flag DERR_SKIPPED;
- support Mail::ClamAV 0.12 and 0.13 or later, which is incompatible with
0.11 due to change of constant names in the underlying ClamAV library;
- added 'check-jpeg' example entry to the @av_scanners list and provide the
associated module JpegTester.pm; it offers a fully-fledged check for jpeg
comment field buffer overflow attempts; should serve mainly as an example
for adding similar quick responses to new threats;
- added 'check-jpeg-simple' example entry to the @av_scanners list (only in
amavisd.conf-sample); it offers a quick check (and not very exact one:
checks only the first 32kB) for jpeg comment field buffer overflow attempts;
should serve mainly as an example for adding similar quick responses
to new threats;
- relax too restrictive sanity check on temporary directory name
when accepting requests from a helper program or via AM.PDP protocol
(e.g. with sendmail milter setup); reported by Babu Kanagala;
- relax allowed set of characters when receiving XFORWARD attribute values,
it turns out that characters like '=' and '+' are allowed;
- when using "bsmtp:" delivery method suppress X-Envelope-From and
X-Envelope-To header fields, as the addresses are already available
in the envelope;
- when using the "bsmtp:" quarantine method the *_quarantine_to was
completely ignored, which made it impossible to turn off quarantining
selectively for certain users by specifying an empty or undef value.
Since 2.2.0 an empty *_quarantine_to turns off quarantine for a recipient
regardless of the quarantine method. A nonempty string in *_quarantine_to
(the exact value is still ignored) must now be used even with "bsmtp:"
to enable quarantining. Inconsistency discovered by Sean Doherty;
- suppress leading $QUARANTINEDIR string from the value of macro %q,
thus hiding the absolute file path from notifications;
- add configuration variable $local_client_bind_address (and equivalent
policy bank key), to allow for explicitly binding local socket address
to a specific interface in SMTP client; suggested by Wouter de Jong;
- keep whatever (if any) file results from gunzip and family (do_uncompress)
even if the decompressor's exit status is nonzero; reason: gzip returns
status 2 when decompressing file with trailing garbage; reported by
Tobias Reckhard;
- collect declared original file name from gzipped (and friends) files
if reported by file(1), making them available to banned name checking;
- avoid unnecessarily checking white/black lists if spam checks will not
be used (e.g. infected mail);
- use qquote_rfc2821_local to properly quote e-mail addresses in the
most visible log entries;
- if there is more than a single (or less then one) occurrence of %k in the
SQL SELECT clause template, multiplicate the set of query keys accordingly,
making possible more complicated custom SELECT clauses;
- don't forget to load amavis policy delegation protocol support code
if AM.PDP is explicitly requested in $protocol, even if not listening
on a Unix socket;
- add 'queue_id' attribute to the AM.PDP protocol; equivalent to a change
in the Courier support code by Martin Orr;
- include the declared (faked) sender address in the virus recipient
notification template, in addition to the originator IP address;
- add macro %Q and method Amavis::In::Message::queue_id, holding a
MTA queue ID if available (in Courier and milter/AM.PDP setup);
by Michael Musikhin (through Martin Orr);
- add macro %y to show elapsed processing time; suggested by Ed Walker;
- sanitize newlines and spaces (and some other characters) when moving
syntactically invalid Message-ID and Resent-Message-ID to macros %m and %r
for logging purposes; resulting wrapped main log entry reported by CRivera;
- bring up syslog priority to LOG_NOTICE when debug or debug_oneshot
is in effect;
- make a product name, version ID and version date available as separate
variables to avoid the need to parse $myversion for the purpose of
customizing e.g. the setting of $X_HEADER_LINE; based on suggestion
by Dale Walsh; the re-introduction of a date ($myversion_date) also
suggested by Stephane Lentz.
Added variables: $myproduct_name, $myversion_id, $myversion_id_numeric,
and $myversion_date, which serve to assemble the $myversion. Modified the
default templates of $smtpd_greeting_banner and $smtpd_quit_banner to
take advantage of the new variables. Changed $X_HEADER_LINE default
from "by amavisd-new at $mydomain" to "$myproduct_name at $mydomain" and
added an example of a $X_HEADER_LINE setting with version number included
to the amavisd.conf-sample;
- added SQL fields 'virus_admin' and 'spam_admin' to lookup lists
@virus_admin_maps and @spam_admin_maps; patch by Robert Collier;
- add a log message 'SPAM-KILL, ...' (at log level 3) for not-passed mail,
to complement the 'SPAM-TAG' log message for passed mail;
- add Mail::SpamAssassin::Plugin::SPF to a list of modules that SA fails
to load at init time;
- prevent sending the same SMTP response more than once, if the first
attempt failed due to disconnected SMTP session;
- fix a double-@ formatting buglet in the log message
"adding address extension _spam to user@@domain", reported by Vincenzo;
- add kill('TERM',$$) to the arsenal of attempts to get rid of unwanted
forked process;
- includes rpm spec file, including the init script, contributed by
Marius Andreiana, based on previous work by Dag Wieers;
- document the localization template directory contents
(in file amavisd.conf-sample) when read_l10n_templates is used;
thanks to Joël Brogniart;
- includes file Macintosh.tar.gz, which contains auto-startup scripts
and installation instructions for Mac OS X, contributed by Dale Walsh
of the Dale Enterprise L.L.C.
---------------------------------------------------------------------------
September 6, 2004
amavisd-new-2.1.2 release notes
- fixed (hard)black- and white-listing on static lookup tables
which failed to match any sender; reported by Derck Floor;
- use $hdrfrom_notify_recip address in the From: field for recipient
notifications, instead of $hdrfrom_notify_admin; inconsistency
pointed out by Ekkehard Burkon;
- the 'neutral' sender notification template was joining the Subject
and the Message-ID header fields into one longer Subject when it was
reporting some nondelivery other than the 'invalid characters in header'.
Likewise the first body line of this same DSN was eaten up:
"This nondelivery report was generated by the amavisd-new program"
(the problem was introduced in amavisd-new-20030616 and never reported);
- in amavisd-agent, amavisd-nanny, amavisd: extend the signal and error
handling in code sections holding bdb locks from just ignoring the SIGINT,
to controlled catching and re-signaling several signals and error conditions;
problem reported by Tom Mulder;
- suppress duplicate names from the list of virus names in macro %V;
by Gregor Weiss;
- fine-adjusted log level of some log messages;
- discard leading and trailing whitespace from the macro %t (Received trace);
- extend the search for IP in the Received trace from 4 to a maximum of
6 entries;
- ignore private IPv6 addresses (RFC 3513: link-local, site-local, multicast)
when searching through Received trace for the origin of mail;
- place mail header field X-Envelope-From in front of the X-Envelope-To in
quarantined mail; also changed case of X-Quarantine-id into X-Quarantine-Id
for consistency with other header fields;
- provide new macro %e which evaluates to our best guess of the originator
IP address collected from the Received trace, complementing similar macros
%t, %a and %g; suggested by Gregor Weiss;
- add the result of macro %e to the default 0-level log entry;
- provide new macros %u and %U to evaluate to a timestamp of the message
reception similar to an existing macro %d (RFC 2822 local date-time);
the (%u) as Unix time (seconds since 1970-01-01T00:00Z as a decimal integer,
suggested by Gregor Weiss), and (%U) as ISO 8601 (EN 28601) UTC date-time;
- avoid some empty lines in default DSN templates and fix some inconsistencies
in their formatting;
- internal: collect existing common code for time formatting as new
subroutines iso8601_timestamp and iso8601_utc_timestamp; collect existing
common code to find IP address in the Received trace as a new subroutine
best_try_originator_ip;
- bump up the version number in $myversion - the 2.1.1 still presented
itself as 2.1.0;
- add a note about a data structure difference between @score_sender_maps and
$per_recip_blacklist_sender_lookup_tables (amavisd.conf-sample, amavisd)
---------------------------------------------------------------------------
August 24, 2004
amavisd-new-2.1.1 (amavisd-new-20040824) release notes
- unconditionally initialize @banned_filename_maps to (\$banned_filename_re),
otherwise $banned_filename_re is ignored by default (unless amavisd.conf
explicitly assigns to @banned_filename_maps); a patch by Thomas Jarosch;
- fixed inappropriate log entry in SQL whitelisting:
wbl: (SQL) recip <xxxx@yyyy.com> whitelisted sender <...>,
unexpected wb field value: "1";
reported by Carlos Horowicz;
- added missing import of &ca to the amavisd-new-courier.patch; by Martin Orr;
- produce a warning when there is an unknown field in the policy bank
to be loaded;
- with delivery method 'bsmtp:' prepend a directory $QUARANTINEDIR to the
file path if not explicitly specified, to behave like the 'local:' delivery
method, making it possible to hide full path from the X-Quarantine-Id
and notifications; a patch by Thomas Jarosch;
- pre-load SA 3.0.0 module Mail::SpamAssassin::Plugin::Hashcash
to make it available in the chroot jail;
- pre-load modules Mail::SpamAssassin::SpamCopURI and URI::*
for SA older than 3.0.0;
- enhancement to amavisd-nanny: when terminating a process and SIGTERM
produces no result for some time, try SIGKILL; contributed by Philip Engdahl;
---------------------------------------------------------------------------
August 15, 2004
amavisd-new-2.1.0 (amavisd-new-20040815) release notes
The 2.1.0 release is mostly a maintenance release over 2.0, with
only a handful of smaller features added. Based on a manual code audit
the number of smaller internal code changes is rather extensive,
some changes dealing with long-standing known deficiencies, minor bugs,
documentation problems and typos. Only a few fixes are for new bugs
introduced in 2.0.
The files amavisd.conf, amavisd.conf-default and amavisd.conf-sample
have been extensively reworked, with the hope to suit better the
new installations, while possibly causing some head-scratching for
existing installations when looking at a diff. The file amavisd.conf
is the one that should serve as a sound base for the initial config file,
while keeping an eye on the list of all variables and their defaults
in amavisd.conf-default. The amavisd.conf-sample is being phased out
of active maintenance, and should serve mostly as a set of examples and
the source of documentation until better documentation is available.
Two nice features are available:
- the use of BerkeleyDB and libdb is now optional;
controlled by variables $enable_db and $enable_global_cache;
- a new program 'amavisd-nanny', with the accompanying instrumentation
in amavisd, displays the general health of all amavisd child processes,
reports crashed ones and attempts to kill long overdue processes;
It is still experimental and minimalistic, problem reporting is currently
only to stdout.
Other changes:
- incompatible change since 2.0: the use of BerkeleyDB is now off by default;
The use of BerkeleyDB and libdb is made optional, controlled by
variables $enable_db and $enable_global_cache, both false by default.
$enable_db:
enables the use of BerkeleyDB/libdb
(for SNMP counters database and nanny, and optionally for cache);
$enable_global_cache:
enables the use of libdb-based cache when $enable_db is also true;
If either the $enable_global_cache or $enable_db are false,
cache of mail body MD5 digests is kept in child-local memory
as in pre-2.0 versions, and is therefore local and short-lived,
with lower expected cache hit rate;
- incompatible change: DSPAM 3.0 or better is needed (if $dspam is enabled),
no longer works with 2.x;
- incompatible change: changed name of the (hardly ever needed) configuration
variable auth_supported_out to auth_required_out, to better reflect its
semantics (should be true if MTA server to which amavisd is sending
notifications and forwarding mail requires authentication (AUTH smtp
command));
- a new small program included: amavisd-nanny is a program to show the status
and keep an eye on the health of amavisd child processes (experimental);
- fixed a bug in lookup_acl where a "." did not act as a catchall;
thanks to JP;
- fixed a problem in SQL lookup which could return undef even when
not all the matched records had NULL in the field;
- fixed compatibility with old 'amavis' helper program
('delivery_care_of' defaulted to 'client', instead of depending on
the presence of ldaargs), reported by Charlie Schluting and Christer;
- fixed long standing problem in do_ascii, which could return without calling
Convert::UUlib::CleanUp, occasionally spilling state into subsequent
mail checks within the same process;
- fixed macros %D, %O and %N when log_recip_templ is being expanded;
a patch by Ed Walker;
- fixed recognition of separators in a nested call during macro expansion;
- pre-load missing modules Net::LDAP, Net::LDAP::Schema, Net::LDAP::Search,
and Net::LDAP::RootDSE; suggested by Paul Jacobson;
- fix locale-related bug in rfc2822 date generation, where we were restoring
the saved LC_TIME value to LC_CTYPE (!);
a patch by Henrique de Moraes Holschuh / Debian support team;
- protect from signals while bdb cursor holds a lock;
- new subroutine inherit_header_edits() and slight code restructuring
makes possible for spam_scan() and other code before the final delivery
to start submitting common header edits into $msginfo->header_edits,
avoiding the need for passing them through global variables;
- now a loglevel-based automatic syslog priority assignment can no longer
lower a message syslog priority below the syslog priority specified
in the $SYSLOG_LEVEL, it can only increase it; the violation of the
least-surprise principle pointed out by Andy Dills;
- a small optimization in logging: a new subroutine ll allows to save time
in preparing complex log entries when we know their log level exceeds
the current log level and won't be logged;
- in default macro templates $log_templ and $log_recip_templ:
* placed 'spam' condition before 'bad header' for consistency
with program behavior;
* added reporting of tag/tag2/kill levels in $log_recip_templ
(experimental: macro names may change in future versions);
* rewritten templates using negation (i.e. [?x|1] ) to avoid unsightly
selector nesting;
- MIME decoding is now allowed to exceed the decoding quota, avoiding
the problem when a small quota settings might not allow even a plain
mail through;
- override LC_TIME to "C" on every log message, to work around issues
with Unix::Syslog, which would log stuff with the date stamp localized,
which syslog would dislike and add its own, and the resulting mess
is not recognized by amavis log processors;
a patch by Henrique de Moraes Holschuh / Debian support;
- changed dspam command line options to work with dspam 3.0 (no longer
with 2.x), a patch by Ron Ohmer, Nabil Sefrioui, and Reech;
- dspam header fields are now inserted into passed mail if all recipients
are local;
- supply better defaults for $hdrfrom_notify_sender, $hdrfrom_notify_recip,
$hdrfrom_notify_admin and $hdrfrom_notify_spamadmin, similar to defaults
from amavisd-new-20030616-p10;
- when parsing output from the 'AVG grisoft' virus scanner,
don't include CR in virus name; reported by Vernon A. Fort;
- new file 'amavisd-new-qmqpqq.patch' provided by Martin Solciansky,
(similar to fixes by Christopher Odenbach) making it work again
with the current version of amavisd-new;
- use lstat instead of stat, and test for soft links wherever appropriate;
- remember inode and device number when creating temporary directory
and temporary file, and test for change before removing them;
- enhanced security: certain tainted values are allowed to enter deeper
into program, untainting them only where and when really necessary;
- avoid a taint problem in Mail::ClamAV;
- added AV entry for CAI eTrust Antivirus; by Stephane Lentz
(requires a suid shell wrapper around inocmd32);
- added status 9 to the set of infected statuses for the
drweb command line scanner (DrWeb Antivirus);
- use our subroutine q_encode to Q-encode header fields from the
notification templates, instead of MIME::Words::encode_mimeword
(a similar fix in 2.0 applied only to encoding of modified headers
in passing mail);
- add attribute 'x-spam-type=original' to the Content-Type header field
(the SpamAssassin's code to recognize an original email) when defanging
spam, facilitating reporting spam via SA to other spam fighting tools;
a patch by Brian May;
- add a note to amavisd.conf that $sa_auto_whitelist has no effect on SA
since 3.0.0 - SA has now a configuration file option 'use_auto_whitelist';
- turn off timer in post_process_request_hook() to avoid periodically
recreating child processes on an idle machine;
- added @mynetworks_maps and enhanced lookup_ip_acl() to take a list
of lookup tables: currently members can be an array ref (as before),
or a hash ref (new) or a plain constant (new);
- generalized @debug_sender_acl into @debug_sender_maps along the lines of
other lookup tables, and make it part of policy banks;
- added @warnvirusrecip_maps, @warnbannedrecip_maps, @warnbadhrecip_maps;
- added @spam_subject_tag_maps and @spam_subject_tag2_maps,
to allow per-recipient spam tags string; suggested by Ed Walker;
Note, there is an inconsistency in names of legacy variables and the
new @*_maps, in an attempt to rectify an unfortunate choice of name
for seldomly used variable $sa_spam_subject_tag1:
@spam_subject_tag_maps = (\$sa_spam_subject_tag1); # exotic
@spam_subject_tag2_maps = (\$sa_spam_subject_tag); # in common use
- added configuration variable $auth_reauthenticate_forwarded, which
directs amavisd to apply its own credentials ($amavis_auth_user and
$amavis_auth_pass) to unauthenticated forwarded (passed) mail, besides
using them for submitting original messages (notifications and quarantine).
This is similar to how mailing list managers are allowed to work (rfc2554).
Note that the Perl module Net::SMTP in its current version is unable to
specify the 'submitter' in its 'MAIL FROM' command, this should be rectified;
---------------------------------------------------------------------------
July 1, 2004
amavisd-new-20040701 / amavisd-new-2.0 release notes
MAJOR NEW FEATURES (since amavisd-new-20030616-p10):
- security improvements: no shell required in chroot jail, checks performed
to see if dropping privileges was successful, can drop privileges before
reading config file;
- the 'amavisd reload' command reimplemented, it now works even
if running chroot-ed;
- new feature: policy banks hold sets of configuration variables that may
be switched with another predefined set based on incoming port number or
original SMTP client IP, avoiding the need to run more than one instance
of amavisd daemon;
- new feature: @score_sender_maps is a soft variant of black- and whitelisting;
- extended semantics of SQL field wblist.wb for soft black/white-listing;
- redesigned mail structure representation allows better control
over 'banned' names and types;
- MIME defanging can wrap the entire original mail in a MIME container;
- more flexible control on lookups: configurable list of lookup tables
observes the specified order of tables and permits arbitrary number
of tables of any available type;
- level-0 logging either by-message or by-recipient;
- syslog priorities are now dynamically derived from the log level;
- constantly updates a small database of SNMP-style counters, providing
real-time measurements for status monitoring and statistics reporting,
reducing the need for analyzing a log file; includes a sample/demo
program 'amavisd-agent';
- new 'policy delegation protocol' between helper program and the daemon
can pass more information to the daemon and allows the daemon to
pass more instructions back to MTA (useful for sendmail milter setup);
- persistent cache of recent virus and spam checks, common to all child
processes, improves the cache hit rate;
- support for IPv6 address formats;
- provisional/experimental support for DSPAM spam checker;
- support for ClamAV virus scanner via Perl module Mail::ClamAV;
- cleaned amavisd.conf :
amavisd.conf configuration file with the essentials
amavisd.conf-default lists all configuration variables with their defaults
amavisd.conf-sample traditional-style commented amavisd.conf with examples
----
INCOMPATIBLE CHANGES since amavisd-new-20030616 (any patch level)
- requires Perl module BerkeleyDB with libdb version 3.1 or later
(tested with db 4.1); This requirement will be made optional
at a later date (in amavisd-new-2.1.0);
- a directory at $db_home (default /var/amavis/db) must be manually created
to store cache and snmp DB files. It should be writable by user running
amavisd. The db files within are removed and re-created at each restart
to avoid having to deal with db recovery (but need not be, as far as the
program logic is concerned);
- sending signal HUP in order to restart amavisd no longer works (previously
it only worked in non-chrooted environment and relied on guessing amavisd
absolute path); please use 'amavisd reload', or 'amavisd stop' and restart;
If the HUP method is really still needed, please replace the line
commandline => [], # disable
by:
commandline => ['/usr/local/sbin/amavisd','-c',$config_file],
in file amavisd, adjusting the path if necessary.
- due to changed names of temporary files, the old 'DrWebD' av entry needs
to be replaced with the current one;
- changed $final_virus_destiny default from D_BOUNCE to D_DISCARD
- changed $final_spam_destiny default from D_REJECT to D_BOUNCE
- changed defaults for variables $virus_quarantine_to, $banned_quarantine_to,
$bad_header_quarantine_to and $spam_quarantine_to from undef (no quarantine)
to values 'virus-quarantine', 'banned-quarantine', 'bad-header-quarantine'
and 'spam-quarantine' respectively. Set them to undef or '' (empty string)
to disable quarantine;
- add address extension at tag2 level, not at kill level as before;
suggested by Jacob Elder and others;
- because of the reorganization of lookup tables, a new way
of quickly disabling virus or spam checks in amavisd.conf is used.
Instead of:
# @bypass_virus_checks_acl= qw( . ); # uncomment to DISABLE anti-virus code
# @bypass_spam_checks_acl = qw( . ); # uncomment to DISABLE anti-spam code
the new recipe is:
# @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code
# @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code
- @virus_admin_maps and @spam_admin_maps now take as lookup keys recipient
addresses, not sender address as before. The new semantics was often
requested, the old semantics was not useful because modern viruses and
spam fake sender address, so the choice was made to incompatibly change
semantics and use the same config variable names, rather than introduce
new names and leave behind useless variables;
An implication is that with multiple-recipient mail it is now possible
to have more than one admin notification generated when recipients have
different admin addresses associated. Still, each distinct admin address
receives only one admin notification;
- removed old compatibility variable $mailto. Use $virus_admin and $spam_admin
variables instead, as suggested in amavisd-new-20021116 release notes,
or the more recent inventions @virus_admin_maps and @spam_admin_maps,
or their equivalents in policy banks;
- removed support for old configuration variable $mailfrom. Use variables
$mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip,
and $mailfrom_notify_spamadmin as introduced in amavisd-new-20020630
for the purpose;
- removed ancient variable @local_domains, use @local_domains_acl instead,
or the more general @local_domains_maps
- removed old compatibility code which allowed for a couple of traditional
variables to treat value "no" as false. The use of "no" for false has been
deprecated since amavisd-new-20021116.
- revoke an old compatibility measure where a missing (undef) tag2 level
would fall back to the kill level value;
- lookup_hash incompatible change (but hardly anyone will notice):
a key presence in the Perl hash but with undefined hash value
used to be interpreted as true, but now it is treated as undef,
causing search to continue with remaining lookup tables (if any).
This is more general and more in spirit with other lookup mechanisms;
- changed a default to initialize SAVI-Perl every time a child process is
started, no longer at master process startup time only. This is to avoid
the need to restart amavisd every time the Sophos IDE database is changed.
One can revert to the previous behavior by uncommenting a call to
Amavis::AV::sophos_savi_init in subroutine fetch_modules_extra;
SECURITY
- no longer invoke shell to call gzip for compressing quarantined files
or to call the sendmail command for submitting messages; the most important
consequence is that a shell is no longer needed in a chroot environment
and should preferably be removed;
- not to forget what was introduced in -p10: inserted a security check
for a missing Net::Server patch, and abort if vulnerable;
- new command line options '-u user' and '-g group' are available.
These are pretty much equivalent to doing a su(1) to the specified user
first (in which case the use of these options is redundant).
By doing 'su' or by specifying a command-line option '-u username'
one can prevent a potential security risk on misconfigured sites where
amavisd.conf is writable by UID running amavisd (e.g. not owned by root).
If a (non-root) username or UID is specified, privileges are now dropped
_before_ opening and evaluating a configuration file. The consequence
is that the configuration variables $daemon_user and $daemon_group
(in amavisd.conf) can not have an after-effect (a warning is issued
if different).
If -u is not specified, the behaviour is as before, i.e. the config file
is read and evaluated under the current UID (as root unless 'su' was done),
and the values of $daemon_user and $daemon_group from the config file
are passed to Net::Server, which changes UID during its startup after
chroot-ing (if requested).
If chroot is desired, the -u must not be used: the root privilege is
required to do chroot, and the config file must be read _before_ doing
chroot. A case of Catch-22. Be doubly careful of who can modify the
configuration file.
Another consequence of specifying -u is that any external files (e.g.
templates, lookup hashes) as possibly read from amavisd.conf, are now
accessed as unprivileged user and no longer as root. The same goes for
opening the log file when not logging via syslog.
MAJOR NEW FEATURES
- policy banks hold sets of configuration variables controlling most of
per-message settings, including: static lookup tables, IP interface
access rules, forwarding address, log level, templates, administrator
addresses, spam trigger levels, quarantine rules, lists of anti-virus
scanner entries (or just a subset), banned names rules, defang settings,
etc. The whole set of these settings may be replaced with another
predefined set based on incoming port number, making it possible for
one amavisd daemon to cope with more diverse needs of served user
communities which could so far only be implemented by running more than
one instance of the amavisd daemon, each with its own configuration file;
This mechanism brings new potentials for the future: in principle policy
banks could be swapped not only based on port number or SMTP client
IP address, but on any characteristics pertaining to a mail message as
a whole (not specific to each of its recipients), or to characteristics
of a connection from a mailer (e.g. the interface address or protocol);
Until a better mechanism is available, a policy bank named 'MYNETS' has
special semantics: this policy bank gets loaded whenever MTA supplies a
SMTP client IP address (Postfix XFORWARD extension or a new AM.PDP protocol)
and that address matches the @mynetworks list.
A hash %$interface_policy is a current mechanism of assigning a
policy bank to an incoming TCP port number (port must be in the list
@$inet_socket_port). Whenever the connection from MTA is received,
first a built-in policy bank with an empty name - the $policy_bank{''}
gets loaded, which bringings in all the global/legacy settings.
Then it is overlaid by whatever configuration settings are in the bank
named in the $interface_policy{$port} if any, and finally the bank 'MYNETS'
is overlaid if it exists and the SMTP client IP address is known
(by XFORWARD SMTP extension command from MTA) and it matches @mynetworks.
See amavisd.conf-sample for examples.
When a new policy bank is overlaid over an existing set of configuration
variables, the variables not present in the new policy bank retain their
value.
The built-in policy bank (with empty name) is predefined, and includes
references to most other variables (the dynamic config variables),
which are accessed only indirectly through the currently installed
policy bank. Overlaying a policy bank with another policy bank may
bring in references to entirely different variables, possibly unnamed.
Configuration variables are referenced from a built-in policy bank (which
is implemented as a perl hash, i.e. an associative array) by keys of the
same name, e.g. { log_level => \$log_level, inet_acl => \@inet_acl, ...}.
For scalars one level of indirection is allowed, e.g.
a policy bank { log_level => \$log_level }; $log_level=2;
is equivalent to { log_level => $log_level } or { log_level => 2 },
but in the first example with an indirect reference, the $log_level
may be assigned to even _after_ the policy bank has already been formed.
One word of caution: the syntax of entries within a policy bank hash
is slightly different from assignments to configuration variables.
This is because entries within policy bank are not asssignments, but
key=>value pairs as in any Perl hash. And these pairs are delimited by
commas, unlike statements, which are delimited by semicolons.
Value is separated from its key by '=>' (or by a comma), whereas the
assignment operator is '='. Keys of a policy bank are without leading $
or @ or %, unlike variable names. Values of a hash can only be scalars
(e.g. strings or refs to arrays).
Compare:
- value of a policy bank is a reference to a Perl hash, e.g.:
{ log_level => 3,
forward_method => 'smtp:[127.0.0.1]:10025',
spam_admin_maps => ["spamalert\@$mydomain"],
}
- normal assignments look like:
$log_level = 3;
$forward_method = 'smtp:[127.0.0.1]:10025';
@spam_admin_maps = ("spamalert\@$mydomain");
And a final note: Perl can detect and report typing mistakes in variable
names, but mistyped key is just some unused hash entry lurking in a hash,
never used and never reported as mistyped/useless.
- @score_sender_maps is a soft variant of black- and white-listing,
contributing positive or negative score points based on sender
e-mail address. Btw, the 'score' in the name '@score_sender_maps' is meant
as a verb, recipient is scoring a sender (= to grade, to determine the
merit of);
Whitelisting is becoming deprecated because it is often and easily abused
(but blacklisting can still be useful);
# ENVELOPE SENDER SOFT-WHITELISTING / SOFT-BLACKLISTING
# Instead of hard black- or whitelisting, a softer approach is to add
# score points (penalties) to the SA score for mail from certain senders.
# Positive points lean towards blacklisting, negative towards whitelisting.
# This is much like adding SA rules or using its white/blacklisting, except
# that here only envelope sender addresses are considered (not addresses
# in a mail header), and that score points can be assigned per-recipient
# (or globally), and the assigned penalties are customarily much lower
# than the default SA white/blacklisting score.
#
# The table structure is similar to $per_recip_blacklist_sender_lookup_tables
# i.e. the first level key is recipient, pointing to by-sender lookup tables.
# The essential difference is that scores from _all_ matching by-recipient
# lookups (not just the first that matches) are summed to give the final
# score boost. That means that both the site and domain administrators,
# as well as the recipient can have a say on the final score.
- the MIME defanging (defang: to make harmless or less powerful)
wraps the entire original mail in a MIME container of type
'Content-type: multipart/mixed', where the first part is a text/plain
with a short explanation, and the second part is a complete original
mail, enclosed in a 'Content-type: message/rfc822' MIME part.
Defanging is only done when enabled (selectively by malware type)
and the malware is allowed to pass (*_lovers or *_destiny=D_PASS)
Conventional mail header fields are retained, and header fields
Resent-{From,Date,Message-ID} are added. A header field X-Amavis-Modified
is inserted to indicate that the mail body has been modified.
It is an experimental feature (disabled by default, except for mail bombs),
and not very efficient for large mail.
MIME defanging is unconditionally done for mail bombs, i.e. when the
X-Amavis-Hold header field is added. The text in the first MIME part
describes the reason, e.g.:
WARNING, possible mail bomb, NOT CHECKED FOR VIRUSES:
Exceeded storage quota 29089500 bytes by do_unzip; ...
When MIME defanging is enabled for passed spam, the first MIME part
contains the full SpamAssassin report.
The MIME defanging feature is not available in the sendmail milter setup.
- new parameter: @mynetworks
It is an IP access list which determines if the original SMTP client
IP address belongs to our internal networks. It is much like the Postfix
parameter 'mynetworks' in semantics and similar in syntax, and its value
should normally match its Postfix counterpart. It only affects the value
of a macro %l (=sender-is-local), and the loading of policy 'MYNETS'
(if present). Note that '-o smtp_send_xforward_command=yes' (or its lmtp
counterpart) must be enabled in the Postfix service that feeds amavisd,
otherwise client IP address is not available to amavisd-new and new
features based on @mynetworks do not work (the %l macro works as before
and bases its decision on sender e-mail address matching local_domains);
LOGGING
- syslog priorities are now dynamically derived from the log level of each
log message (the first argument of sub do_log). The priority as specified
in the $SYSLOG_LEVEL configuration variable is ignored in 2.0 (no longer
in 2.1), but the 'facility' is not ignored. This makes it possible
to influence the log verbosity by syslog.conf settings. Here is an
example of a useful syslogd.conf setting (some levels may be left out;
assumes the $log_level is high enough, e.g. 2, to produce any low-priority
log messages at all):
mail.err /var/log/messages
mail.notice /var/log/amavisd.log
mail.info /var/log/amavisd-info.log
mail.debug /var/log/amavisd-debug.log
(On linux do not forget to prefix filenames in syslog.conf with
a '-' to disable sync, which has much impact on syslog performance!)
At the moment the mapping of message log levels to syslog priorities
is hardwired:
level <= -3: LOG_CRIT
level <= -2: LOG_ERR
level <= -1: LOG_WARNING
level <= 0: LOG_NOTICE
level <= 1: LOG_INFO (in version 2.0)
level <= 2: LOG_INFO (since 2.1)
else: LOG_DEBUG
Not to be confused with the $log_level configuration variable setting,
which still works as before, suppressing generation of all log messages
with log levels above $log_level.
For efficiency reasons one should not specify unnecessarily high $log_level
and then discard low syslog priority messages in the syslogd.
- added $log_recip_templ variable, which is similar to $log_templ,
but gets called for every recipient (the $log_templ is evaluated only once
per message). Normally one or the other log template should be disabled
by assigning undef or an empty string to the corresponding variable;
- the default $log_templ no longer shows quarantine ID if quarantining is
disabled;
- added a macro %. (a dot), which might be useful in the $log_recip_templ;
Its value is empty when $log_templ is expanded, and is a recipient counter
(starting by 1) when $log_recip_templ template is expanded. Based on
this macro one can provide a single template for both the $log_templ
and the $log_recip_templ if needed, or perhaps let the log entry for
the first recipient be more verbose that the rest;
- added a macro %T which expands to a list of triggered SA tests, but only
when $log_templ and $log_recip_templ are expanded. In notifications
the %T is still a list of To: addresses. An overlaid semantics, but we
are running out of letters and a macro expander rewrite would be needed;
- new macros %k, %1, %2, %O, please see README.customize
LOOKUPS
- new configuration variables make it more flexible to specify arbitrary
list of lookup tables. Legacy configuration variables are still available
and are referenced from the default values of @*_maps lists. If these lists
are redefined, legacy variables are not used.
The new variables (lists) are:
@local_domains_maps
@bypass_virus_checks_maps @bypass_spam_checks_maps
@bypass_banned_checks_maps @bypass_header_checks_maps
@virus_lovers_maps @spam_lovers_maps
@banned_files_lovers_maps @bad_header_lovers_maps
@virus_admin_maps @spam_admin_maps
@virus_quarantine_to_maps
@banned_quarantine_to_maps @bad_header_quarantine_to_maps
@spam_quarantine_to_maps @spam_quarantine_bysender_to_maps
@banned_filename_maps
@spam_tag_level_maps @spam_tag2_level_maps @spam_kill_level_maps
@spam_dsn_cutoff_level_maps
@spam_modifies_subj_maps
@whitelist_sender_maps @blacklist_sender_maps @score_sender_maps
@addr_extension_virus_maps @addr_extension_spam_maps
@addr_extension_banned_maps @addr_extension_bad_header_maps
@keep_decoded_original_maps
@map_full_type_to_short_type_maps
@viruses_that_fake_sender_maps
- for more informative logging of lookup operations, a new object type
Amavis::Lookup::Label can be inserted to lists of lookup tables
for the purpose of labeling the main purpose of the list;
- all lookup* subroutines can now return matching key when called in a
list context;
- lookups can now return a list of all matching entries (not just the first
match); used for the new soft- white/blacklists (@score_sender_maps);
- sub lookup() now allows for one level of list elements dereferencing,
which makes possible the construction of the argument list and later
still be able to modify its members (e.g. creation of regexp lookup table
objects in the configuration file); It facilitates transition from old
hard-wired lists of lookup tables to new @*_maps list variables which
permits specifying an arbitrary number of lookup tables and to specify
their search order;
- simplify and unify calls to lookup() by collecting arguments (references
to lookup tables) in lists, e.g. @local_domains_maps, @virus_lovers_maps,
@virus_admin_maps. These array variables default to lists of legacy
variables, which are never directly used by the program. Either the
individual legacy variables may be assigned to, or the entire list
replaced, in which case the legacy variables no longer have any effect.
- lookup_acl: respect $localpart_is_case_sensitive setting;
- lookup_hash and lookup_sql: rewritten lookup_hash and factored out the
common code from lookup_hash and lookup_sql to make_query_keys();
- lookup_hash bug fix: avoid splitting address literal as if it were
a domain name; (a bug with key '.' not being tried for address literals
fixed thanks to Uwe S. Fuerst);
SQL, LDAP LOOKUPS
- extended semantics of SQL field wblist.wb, which can hold a score value
boost, which is interpreted as soft black/white-listing (the same semantics
as the value in @score_sender_maps);
- recognize SQL server error 2013/"Lost connection to" and treat it
the same as 2006/"MySQL server has gone away"; by Max Kalika;
- full domain stripping: @.sub.example.com @.example.com @.com @.
- lookup_hash and lookup_sql: limit the list of subdomain
search keys to 10 levels as a sanity measure; e.g. for address
user@14.13.12.11.10.9.8.7.6.5.4.3.2.com the subdomains keys search list
starts at .9.8.7.6.5.4.3.2.com; (domain names are limited by standard
to 127 levels);
- prepare SELECT statements on demand;
- recognize all-zero and all-null boolean fields as false;
- recognize new (optional) fields in the table 'policy':
spam_dsn_cutoff_level
virus_quarantine_to, banned_quarantine_to, bad_header_quarantine_to
addr_extension_virus, addr_extension_spam,
addr_extension_banned, addr_extension_bad_header
(the addition of virus_quarantine_to was suggested by Harald Kapper);
- consider the SQL user id a string (no longer required to be numeric);
thanks to Max Kalika / Gentoo support;
- LDAP white/black list support by Jacques Supcik (similar to contribution
from Scott Dier and Eric Dorland, which I forgot about, sorry);
- added amavisSpamQuarantineTo to the LDAP schema;
new version of LDAP schema; by Jacques Supcik:
In the previous schema, the tag levels have been defined as integer.
This is too restrictive, and have now been changed to strings
(there is no float type in LDAP);
- added LDAP attributes for completeness: amavisBadHeaderLover,
amavisBypassBannedChecks, amavisBypassHeaderChecks, amavisVirusQuarantineTo,
amavisBannedQuarantineTo, amavisBadHeaderQuarantineTo; by Jérôme Schell;
DECODING / DEARCHIVING
- provide optional ability to retain complete email message in its
un-decoded form (alongside its decoded parts) for a virus scanner
to see (enabled if $keep_decoded_original_re matched string 'MAIL');
suggested by Tomasz Papszun; (partly backported to amavisd-new-20030616-p8);
- rewrite code that generates new file names (Amavis::Unpackers::Part->new),
and rewrite code dealing with banned names. Keep information about each
part organized as a tree, matching the descendence of each part, new
package/object Amavis::Unpackers::Part to collect such information;
- make Amavis::Unpackers an optional module: the interface code to external
decoder/unpackers/dearchivers does not get compiled and does not consume
virtual memory if $bypass_decode_parts is true; (previously it just
didn't get called, but was sitting in memory nevertheless);
- decode RPM archives by converting them to cpio, if rpm2cpio and cpio
are available;
- do_tnef: extract $tnef->message if it exists, not just $tnef->attachments;
- support extracting MS cabinet files (CAB) by calling cabextract, if enabled
and found. Beware: Lars Hecking warns that cabextract 0.6 is quite buggy
and the author has been notified.
VIRUS SCANNING
- support for ClamAV virus scanner via Perl module Mail::ClamAV,
based on code by Roberto Pereira da Rosa;
- don't call virus scanners if there are no files in the directory to be
scanned (e.g. mail with an empty body); some virus scanners don't like
to be given an empty directory (e.g. Symantec newer savsecls);
reported by Marco Bicca;
- rewritten/unified/generalized subroutines ask_daemon and sophos_savi
based on the new subroutine ask_av;
- scan parts directory for file names exactly once regardless of the number
of virus scanners and their arguments (containing '{}/*' or not);
- supply full original mail to virus scanners in case of MIME parse errors
(in addition to any possibly decoded parts);
- when collecting file names to be virus scanned, prepare a hash which maps
base file names to Amavis::Unpackers::Part objects, and makes it available
to virus scanner interface routines, which may benefit from the additional
information about the file to be scanned.
In particular, the new interface to Mail::ClamAV now turns on the option
CL_MAIL, and the interface to SAVI turns on the option MIME, when entire
mail is passed to AV scanner for checking. This enables ClamAV and SAVI
to attempt MIME decoding the file by itself.
TODO: The same option (--mime) would need to be specified when calling
'clamscan' and supplying a non-decoded mail for checking; pointed out
by Riccardo Ghiglianovich and Michael Boelen;
OTHER EXTERNALLY VISIBLE CHANGES
- The 'amavisd reload' command is now implemented differently:
old: signals SIGHUP to a running amavisd process and exits immediately;
the running amavisd process (under control of Net::Server) when it
receives a SIGHUP starts its own copy with same arguments and exits;
new: signals SIGTERM to a running amavisd process, waits for it to finish,
then continues (same as 'amavisd start') to become a new daemon;
The new method works even when chrooted, and is more reliable when the
existing process is slow to terminate, as it actively waits for the previous
daemon to finish before proceeding to promote itself to become a new daemon.
- a simple demo program 'amavisd-agent' is provided, allowing
for continuous inspection of SNMP counters; a path to the
/var/amavis/db is hardwired in the program, modify it if necessary.
- server-side support for optional Postfix SMTP/LMTP command XFORWARD:
information about the original SMTP client IP, its DNS name, HELO name
and protocol used is now made available to the amavisd program for
logging and other purposes. The same information can also be obtained
from the 'Amavis policy delegation protocol (AM.PDP)' if the helper
program supports it (useful for sendmail milter setup);
- client-side support for optional Postfix SMTP command XFORWARD:
if MTA announces in its SMTP EHLO response that it supports XFORWARD,
amavisd will provide additional information about the original SMTP client
if the information is available (either from XFORWARD on the receiving
side, or by the 'Amavis policy delegation protocol';
- server side support for the new amavis helper protocol which allows
for header modifications, removal of recipient addresses (e.g. non- spam
lovers) or rewrite of recipient addresses (e.g. adding address extensions),
and specification of full SMTP response;
- modified search logic for matching mail parts against $banned_filename_re;
The old search order for names did not result in what one might
expect when pattern list elements with a value of false were used
in $banned_filename_re. Namely, all three components were searched
independently (Content-Type, declared name, and file(1) type)
and a logical or was used. Because searches for each mail part were
independent, it was not possible to specify for example that anything
within a zip would be allowed. If any of these searches returned true,
mail was blocked.
To make this useful, a complete rewrite of mail unpacking was needed
and all information be made available in one place after the unpacking
is over, so that checking for banned names can be done all at once.
The search order is now much the same as used in rsync and its server,
see man rsync, section 'EXCLUDE PATTERNS'. The new comments in
amavisd.conf-sample explain the new situation.
- replaced $relayhost_is_client by a more flexible specification:
To make it possible for several hosts to share one content checking daemon,
the IP address and/or the port number in $forward_method and $notify_method
may be specified as an asterisk. An asterisk in the colon-separated
second field (host) will be replaced by the SMTP client peer address
(i.e. the MTA host). An asterisk in the third field (tcp port) will be
replaced by the incoming SMTP/LMTP session port number plus one. This
obsoletes the previously used less flexible configuration parameter
$relayhost_is_client. An example:
$forward_method = 'smtp:*:*'; $notify_method = 'smtp:[127.0.0.1]:10025';
The same functionality can also by achieved by using a bigger hammer,
the policy banks. These may completely replace the global settings
for $forward_method and $notify_method, based on incoming port number;
- turn address extension variables (the so called "plus addressing")
into recipient-based lookup tables, including the SQL lookups. For example:
@addr_extension_virus_maps = ('virus'); # defaults to empty
@addr_extension_spam_maps = ('spam'); # defaults to empty
@addr_extension_banned_maps = ('banned'); # defaults to empty
@addr_extension_bad_header_maps = ('badh'); # defaults to empty
or perhaps:
@addr_extension_virus_maps = (
{'sub.example.com'=>'infected', '.example.com'=>'malware'}, 'virus' );
suggested the Gentoo modification, Jacques Supcik, and others;
- log and report hits and tag/tag2/kill levels rounded to three decimal
places (trailing zeroes trimmed), no longer rounded to one decimal place;
- added @spam_dsn_cutoff_level_maps, making it possible to specify
different DSN cutoff levels for different recipient domains or users.
In multi-recipient messages where recipients can specify different values,
the maximum value is used for deciding whether DSN should be suppressed;
suggested by Ales Casar;
- configuration variable $gets_addr_in_quoted_form is no longer used;
knowledge about address form (quoted or not) is now implicit in the
receiving protocol;
- if tag level turns out to be undef, it will not be shown in X-Spam-*
header fields, and will be interpreted as having a value lower than any
spam score when deciding whether to insert X-Spam-* header fields or not;
- added macros %a and %g:
* %a original SMTP session client IP address (empty if unknown)
* %g original SMTP session client DNS name (empty if unknown)
(like macros %I and %M that were once proposed by Dibo in his 2002-07 patch)
This information may be available from Postfix when XFORWARD protocol
extension to SMTP is enabled, and can be made available by helper program
(e.g from sendmail milter) when the new AM.CL protocol is used;
- added macro %p, expanding to a current policy bank name (or empty
if a built-in policy bank is still in place);
- added macro %r, expanding to the contents of the first Resent-Message-ID
header field, or empty if no such field exists. Include reporting the
Resent-Message-ID in the log and in the sender notification;
suggested by Oliver Gorwits;
- new configuration variable $addr_extension_bad_header for completeness;
- added $bad_header_quarantine_to, @bad_header_quarantine_to_maps,
$bad_header_quarantine_method, and $warnbadhrecip for completeness;
suggested by Robin Lynn Frank;
- MIME::Parser errors now contribute to bad-header checks, so that the
header checking is now conceptually extended to MIME sub headers
(Postfix similarly considers MIME subheaders part of mail headers);
MIME::Parser 6.1xx or later is recommended.
- allow $*_quarantine_method to be undef as a quick way of disabling
some quarantine (it also can be disabled as previously, by using
method 'local:' and following its rules);
- persistent cache of recent virus and spam checks, global to all child
processes, can improve the cache hit rate. Uses BerkeleyDB database (hash
and queue) and its interlocking mechanisms (Berkeley DB Concurrent Data
Store) for the purpose. The V3.1 or better is required, V4 is recommended.
- include version information in the 'Usage' text;
- rewritten lookup_ip_acl() and added ip_to_vec() to allow for IPv6
address syntax as specified in rfc3513 to be used in IP lookups;
- @inet_acl now defaults to ('127.0.0.1', '::1'), i.e. it adds the
IPv6 loopback address to the list;
- new configuration variable $sa_spam_level_char (defaults to '*')
allows specifying another character for X-Spam-Level bar. Empty or
undefined value disables inserting the X-Spam-Level header field;
- added configuration variable $sa_spam_report_header to enable/disable
inserting the X-Spam-Report header; patch by Craig Sanders;
- added $banned_quarantine_to configuration setting to make possible the
quarantining of banned mail to a different place from viruses;
- don't insert virus-, banned- and bad headers- related headers for passed
mail to recipients with corresponding bypass_*_checks, making them believe
the mail was not spam-checked (as they are not expecting such headers
anyway); This was already done in version amavisd-new-20030616-p6
but only for spam-related headers;
- for choosing address extensions use the same criteria as for adding
header fields, e.g. pretend to not know the result of a certain test
(virus, spam, ...) when recipient chooses to bypass such test,
even if the result of the test is known;
- added variable $sa_spam_subject_tag1 (undef by default).
If $sa_spam_subject_tag will not be inserted (at tag2 level), and
$sa_spam_subject_tag1 is nonempty, this string (e.g. '***possible SPAM*** ')
will be inserted into the Subject header field for spam levels above
tag level; suggested by Immo Goltz;
- added separate configuration variables $banned_files_quarantine_method
and $bad_header_quarantine_method. Quarantining of banned files and
bad headers were previously controlled by $virus_quarantine_method;
- rewritten read_hash, it is now possible to specify key value (right-hand
side) for each key. If value is not specified, a '1' is assumed as before;
- SMTP server support for rfc2554 authentications (PLAIN and LOGIN only)
allows client authentication to be relayed to the MTA when message is
forwarded. Might be useful if amavisd-new is used in a Postfix SMTP proxy
setup, but is not needed for other setups. Disabled by default,
see variables $auth_required_inp and @auth_mech_avail.
- SMTP client support for rfc2554 authentications (any authentication method
as supported by Net::SMTP and Authen::SASL Perl modules). Authentication
of forwarded mail (PLAIN and LOGIN only) is carried over from the incoming
mail, authentication to be used when submitting notifications is controlled
by configuration variables $amavis_auth_user and $amavis_auth_pass.
Disabled by default, see variable $auth_supported_out.
- provisional/experimental support for DSPAM spam checker (pre 3.0):
if configuration variable $dspam is nonempty and represents a path to
a 'dspam' program, a message is passed to dspam and its inserted headers
of the form X-DSPAM-* are axtracted and then made available for
SpamAssassin rules to check and score if desired.
All messages are currently presented to dspam as the same user, affecting
how its database is being built. False negatives and false positives
(based on SA assessment) are fed back into DSPAM as a simple form
of auto-learning. Works reasonably well, but do not expect miracles.
See subroutine spam_scan.
Here is how DSPAM can be installed to be able to be used by amavisd-new:
dspam 2.x:
./configure --enable-alternative-bayesian \
--with-userdir=/var/amavis/dspam --enable-signature-headers \
--without-local-delivery-agent --without-quarantine-agent
dspam 3.0.0:
./configure \
--with-dspam-home=/var/amavis/dspam --enable-signature-headers \
--without-delivery-agent --without-quarantine-agent
make install
chmod u-s,a+rx /usr/local/bin/dspam
chown vscan:vscan /var/amavis/dspam
and the following can be inserted into the SA config file (local.cf)
to make it recognize and incorporate DSPAM's assessment:
header DSPAM_SPAM X-DSPAM-Result =~ /^Spam$/
describe DSPAM_SPAM DSPAM claims it is spam
score DSPAM_SPAM 0.5
header DSPAM_HAM X-DSPAM-Result =~ /^Innocent$/
describe DSPAM_HAM DSPAM claims it is ham
score DSPAM_HAM -0.1
Eventually DSPAM support should be removed from amavisd-new, as soon as
SA will be able to call it on its own.
INTERNAL CHANGES
- reformatted the whole program, reducing indentation from 4 to 2
and replacing TABs with spaces (with some dubious help from perltidy,
plus lots of manual adjustments);
- completely rewritten code to handle both the old and the new amavis helper
protocol, as well as Postfix 'TCP client/server table lookup protocol'
as specified in the Postfix documentation: tcp_table(5);
(process_policy_request, prepare_policy_query, check_amcl_policy)
- type_short may now be a list of short types. Typical use is to classify
a MS executable as both an 'exe' and as 'exe-ms', which makes more specific
banned rules possible without unnecessary complication in regexps;
- parts now carry attributes, which can be inspected for banned checks;
current attributes are U for undecodable, and C for (en)crypted;
- opened another can of Perl worms (taint bugs): turn on Perl pragma
"use re 'taint'" in all modules, and selectively turn it off where needed.
It replaces cumbersome manual preservation of taintedness when regexp
saved ranges are used without intention to untaint. Because of Perl bugs,
strategically placed local($1,$2,...) are needed, otherwise previous
taint flag in $1, $2, ... can be brought on to new variables, which can
all of a sudden become tainted out of nowhere;
- catch and report throws (die) in pre_loop_hook() to properly report
problems during initialization;
- introduced subroutine exit_status_str and unify reporting of
subprocess status;
- enhanced sub best_try_originator to ignore IP addresses from private,
local and dedicated IANA networks (rfc3330) and look for the first
public address in the 'Received' path;
- examine first four (chronologically) Received header fields (instead of
first two) when looking for an originator IP address, and ignore those
with private IP addresses;
- moved code dealing with body digest cache to a new package
Amavis::Cache to facilitate transition to shared or persistent cache;
- new explicit cache expiration time controls (time to live in seconds):
$virus_check_negative_ttl $virus_check_positive_ttl
$spam_check_negative_ttl $spam_check_positive_ttl
$cache_entry_ttl
- discard cache db ($db_home) and recreate it during restart;
- more informative changes of child process $0, which may show in the ps(1)
output;
- store tempdir of a current message to the Amavis::In::Message object;
- gather some statistics about idle time;
- reorder and adjust mapping from file(1) results to file type classes;
- optimization: instead of invoking file(1) utility program for each
part to be analyzed, now call it once for each round of currently
available parts, giving it the list of all available parts as arguments;
- shorten the names of parts from part-..... to p..., to be able to stash
more files names into a command line, e.g. when calling file(1) or
external virus checker which can not deal with directories;
- use regexp lookup table mechanism (table $map_full_type_to_short_type)
to match long types (output of file(1) utility) to short types (.exe,
.jpg, .doc, ...). The default table can be replaced by a customized
table in amavisd.conf;
- replace $(?!\n) with \z in regular expressions throughout;
replace most of the remaining $ with \z in regular expressions.
The regular expression primitive \z is available since perl 5.005.
- TODO: disregard $MAXFILES during initial MIME unpacking;
reported by Stephane Lentz and Robert LeBlanc;
Some un-edited notes on the new banned rules mechanism:
(I wrapped your log lines, and replaced \\ by \ for clarity:
| Feb 24 19:07:29 hauptpostamt amavis[29847]: (29847-04-5) p.path BANNED:
|
| "P=p002,M=application/octet-stream,T=zip,N=document.zip |
| P=p003,T=exe,T=exe-ms,N=document.htm .scr",
part p003 is of type (file(1)) MS executable, with suggested
name "document.htm .scr" (lots of spaces in the name)
its parent resides on temp file p002 (i.e. p003 was extracted from it),
which is of type (T) zip archive, with suggested (MIME) name
(N) "document.zip", and has a MIME type (M) "application/octet-stream".
such a component p003 lying within such p002 is considered banned
by the following regexp rule (one rule within the $banned_namepath_re list):
| matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* \.
| (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"
which says that any component at any level must not have a name (N)
matching a pattern:
any number of characters,
a dot,
any number of non-dot and non-slash characters
a dot,
and ending with: exe or vbs ...
(basically: double extension ending with listed extensions)
The complications such as using [^\t\n]* instead of .* are there
to keep regexp contained within fields and ancestors/descendents.
There is one detail to remember when comparing logged p.path log entries
and the actual matching rules:
- for the sake of readability the logged entry has \n (newlines) converted
into ' | '. The \n is a separator between components in the tree
from the root (the mail itself, hidden) to the the leaf component
which can not be further expanded (i.e. not an archive)
- for the sake of readability the logged entry has \t (a tab) converted
into comma, separating information fields such as P=... M=.. T=.. M=..
So the above logged string:
P=p002,M=appl...,T=zip,N=document.zip | P=p003,T=exe,T=exe-ms,N=document.htm.scr
is actually a single string:
P=p002\tM=appl...\tT=zip\tN=document.zip\nP=p003\tT=exe\tT=exe-ms\tN=document.htm.scr
and a Perl regexp is applied directly to it.
The raw string is rather unsightly, but the \n and \t were chosen
to minimize clash with valid characters within file names.
If a \n or \t is present in a name of the components, such character
is converted into a space to avoid clashing with separators.
| Feb 24 19:11:58 hauptpostamt amavis[31505]: (31505-01-5) p.path BANNED:
| "P=p002,M=application/octet-stream,T=zip,N=paypal.zip |
| P=p003,T=exe,T=exe-ms,N=paypal.scr",
a MS executable named "paypal.scr" within a zip archive "paypal.zip"
| matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \.
| (exe|vbs|pif|scr|bat|com) (\t.*)? $)"
block component at any level with a name (N) terminating
by dot followed by any of the listed extensions.
| Feb 24 19:18:25 hauptpostamt amavis[32159]: (32159-01-2) p.path BANNED:
| "P=p002,M=application/octet-stream,T=zip,N=text.zip |
| P=p003,T=exe,T=exe-ms,N=text.txt .exe",
a MS executable named "text.txt .exe" (with lots of spaces in the name)
within a zip archive named "text.zip"
| matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n
| (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"
blocked by the double-extension rule.
| Feb 24 19:30:15 hauptpostamt amavis[1690]: (01690-02-8) p.path BANNED:
| "P=p002,M=application/octet-stream,T=zip,N=jokes.zip |
| P=p003,T=exe,T=exe-ms,N=jokes.doc .exe",
| matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n
| (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)"
same thing
---------------------------
---------------------------------------------------------------------------
June 29, 2004
Patch: amavisd-new-20030616-p10
- insert a security check to test for a missing Net::Server patch, and abort
if vulnerable;
- provide and use our own subroutine q_encode to do the Q-encoding when
editing an existing invalid header field with non-encoded 8-bit characters,
e.g. when inserting ***SPAM*** or ***UNCHECKED*** into Subject.
The MIME::Words::encode_mimeword() does not encode spaces and does not limit
encoded words to 75 characters, which violates the RFC 2047 and breaks mail
readers; reported by Sebastian Hagedorn and Gregor Hoffleit;
- fixed a bug in inserting the tag_level header field, which was missing
if sender was blacklisted and tag_level was greater or equal to 0;
thanks to Joerg Thaler;
- amavis-milter.c
* log envelope sender address at the same log level (DBG_INFO) as
recipient addresses;
* remove the log message "(mlfi_eom) header already present", it was
inappropriate, the call to smfi_chgheader succeeds even if no such
header was already present;
* relax permissions on created directory and temporary file to allow
group read access (needed if virus scanner runs under a different
user id but within the same group); by Adam C. Migus;
* add queue id (when available) to most log messages;
- do not preserve evidence just because a message gets an X-Amavis-Hold
header field;
- when logging directly to a file and started as root, change UID of a
log file to $daemon_user to avoid restart problems; based on a patch
by Carsten Hoeger and Gregor Weiss;
- added configuration variable $first_infected_stops_scan
to stop anti-virus scanning when the first scanner detects a virus;
the default is false, all scanners in a group are called (as usual);
- in the AV entry for clamscan added the option '--tempdir=$TEMPBASE';
- in the AV entry for 'Norman Virus Control v5 / Linux'
changed the command name 'nvccmd' into 'nvcc'; correction by
Michael Ramke of the Norman Data Defense Systems GmbH;
- insert debug reports into sub ip_addr_to_name to help recognize DNS problems;
- revoke deleting an existing 'X-Scanned-By' header field, which was
introduced in -p8; removing it gets in a way when more than one content
filter is being chained;
- load a missing SpamAssassin v3.0 module, needed when running in chroot;
- contributed (for now in the form of an optional patch): support for
the incoming qmqpqq protocol over a TCP socket - to be used with qmail.
Apply the provided patch 'amavisd-new-qmqpqq.patch', it updates files
amavisd.conf and amavisd in the current directory; by Martin Solciansky;
- documentation updates;
---------------------------------------------------------------------------
April 2, 2004
Patch: amavisd-new-20030616-p9
- avoid choking on undefined $banned_filename_re; thanks to Ales Casar,
Sebastian Hagedorn and E. Falk;
- if Subject mail header field got $undecipherable_subject_tag inserted, it
would also receive the spam tag $sa_spam_subject_tag unconditionally;
fixed, thanks to Francis Stevens;
- updates to the @av_scanners list in amavisd.conf:
* DrWebD now works with a new DrWeb daemon 4.31,
thanks to information provided by Krzysztof Cegielski, DrWeb Polska;
* updated BitDefender bdc, thanks to Alfredo Milani Comparetti;
* clamscan: use documented option --no-summary instead of the
undocumented/old(?) option --disable-summary; by Georgy Salnikov;
* updated Kaspersky aveclient and F-Secure fsav, thanks to Tomi Hakala;
* recognize that KasperskyLab antivirus in demo mode turns on the bit 0x10
in status; avp and avpdc now use the same two sets of status codes;
avp: statuses 3 and 6 moved to infected (to match avpdc);
statuses 2 and 5 considered infected; all suggested by Georgy Salnikov;
- when original undecoded mail is to be kept for virus scanners (requested
by patterns /^MAIL$/ or /^MAIL-UNDECIPHERABLE$/ in $banned_filename_re),
the preserved file is now named parts/part-00000 instead of parts/email.txt
to preserve the size of the name, upon which some virus scanners depend
(e.g. DrWebD);
A note to Courier users: due to the way a file name in Courier is created
and passed to amavisd-new, it is currently not possible to use the triggers
/^MAIL$/ and /^MAIL-UNDECIPHERABLE$/ in $keep_decoded_original_re, or a
failure to create a hard link occurs; Thanks to Bowie Bailey for helping to
troubleshoot the problem; (this limitation is lifted in amavisd-new-2.2.0)
- remove option -c when calling gzip, bzip2, compress, lzop and unfreeze
(as has long been done in the development version); the option -c is not
needed when no file argument is present, and some implementations of gzip
and compress may choke on it;
- make loading of Perl module Carp::Heavy optional, versions of Carp that
came with Perl 5.005 did not have it; reported by Jefferson Pizzolatti;
- load module Mail::SpamAssassin::BayesStore::DBM, required by SA 3.0
running in chroot jail;
- look for program 'gcpio' ( $cpio = ['gcpio','cpio'] ), as on OpenBSD the
plain cpio does not support required options (the --no-absolute-filenames
is essential), but GNU cpio does; thanks to Manfred Gloiber;
- relax parsing of file(1) output to allow tab as well as space to follow
a file name; the file(1) on Solaris uses tab instead of space;
suggested by Glen Harris;
- make LHA understand self-extracting archives (SFX); patch by Georgy Salnikov:
Although lha unpacks any non-SFX archive independently if its extension,
it unpacks the SFX lha archives only if they have the extension .exe.
Now the file is symlinked to $part.exe for checking by lha: this will now
work for SFX, and will still work for non-SFX. Also, do_lha will return 0
if $exec and the part cannot be de-archived, the same as in do_unrar.
The executable file formats are checked for being zip/rar/lha SFX archives.
However, if the SFX archive is not a zip archive, do_unrar always returns
success, so that if the archive is also not a rar, it will be never checked
by lha. Now do_unrar returns 0 if $exec and the corresponding part can not
be un-RARed;
- do_executable and do_unarj: added checking for SFX arj; by Georgy Salnikov:
(commented out the call to do_unarj in do_executable until more experience
is gained on how well unarj survives certain mail contents; Mark);
- do_unarj: let arj/unarj work on file named part*.arj;
- when calling IO::File::open() use '+>' instead of 'w+' to avoid
Perl taint bug ($mode turns tainted) (bug still present in 5.8.2)
triggered by expression in IO::Handle::_open_mode_string();
- attack the Perl 5.8.0/5.8.1/5.8.3 taint bug (once variables $1,$2,etc
get tainted they start spreading taintedness to other variables):
* insert local($1,$2,$3,...) in blocks of code which call external
modules which trigger the bug (Mail::SpamAssasin, MIME::Parser, ...)
* insert local($1,$2,$3,...) in blocks of code which depend on these
variables to be clean, and which demonstrated through bug reports
and experience with various version of Perl that these variables
were not always taint-clean;
The last taint incident triggered by SA 3.0.0 (svn) reported by Luc de Louw;
- recognize status EX_NOUSER when forwarding via pipe to sendmail (old setup);
previously it was treated as a temporary failure; patch by The Mindflayer;
- turn error message 'error reading from client socket' back into
'client broke the connection without a QUIT' for consistency with P7;
- when %virus_admin lookup table is used, prefer $msginfo->sender_source
(unmangled sender domain) over $msginfo->sender; suggested by Pawel
Golaszewski;
- nicety: in virus recipient notifications now supply the To: header field
with the true recipient address instead of "undisclosed-recipients:;"
in case of single-recipient mail; suggested by several people;
- amavisd would insert differently capitalized header fields (either
X-AMaViS-Alert or X-Amavis-Alert), depending on the reason being reported;
now use the X-Amavis-Alert throughout; reported by Carsten Hoeger;
- fetch Perl auto-loaded modules auto::POSIX::setgid and auto::POSIX::setuid
if they exist; they are needed by older versions of Perl when running in
chroot. A manual change was needed until now (documented in README.chroot),
which should no longer be needed; a FreeBSD problem report 64636.
- helper program amavis-milter.c (used in the sendmail milter setup):
* amavis-milter.c only included the client IP and client host name on the
first mail transaction of a multiple-transaction SMTP session, but not
in subsequent transactions; pointed out by Stephane Lentz. The solution
was once already provided back in July 2002 by Radoslav Dibarbora - Dibo,
and forgotten; credits where credits are due:
http://marc.theaimsgroup.com/?l=amavis-tech&m=102569946910555
* make a clear distinction between message data and connection data,
which required code reshuffling and revealed previous unlean solutions;
* add error checking and reporting to mkdir/rmdir/open/unlink/write
system calls; previously an error could pass by unnoticed or just
caused a tempfail without an explanation;
* change final milter status ACCEPT into CONTINUE to allow further
milters in the milter chain to examine the mail;
* code cleanup;
* regenerated helper-progs/configure with Autoconf 2.57 to make it
capable of detecting mkdtemp and still be able to find sendmail
libmilter files; thanks to Sebastian Hagedorn for a problem report,
for testing, and for revealing a bug in the pre-released version;
* use different syslog priority for different internal message
log levels; by Sebastian Hagedorn;
* adjust log levels of messages, set default verbosity 1 (DBG_WARN);
- documentation updates, including the updated comments in amavisd.conf
regarding whitelisted senders, to reflect the change indicated in the
amavisd-new-20030616-p4 release notes;
---------------------------------------------------------------------------
March 9, 2004
Patch: amavisd-new-20030616-p8
- be compatible with SpamAssassin version 2.70 and 3.0 as well as 2.6x;
SA changed its API, replacing Mail::SpamAssassin::NoMailAudit
with $spamassassin_obj->parse and belonging objects;
- as a stop-gap solution to the W32/Bagle-{F,...} detection problem
(password-protected zip archives), three new measures are available:
* ability to present the full non-decoded original message to virus
scanners was partly back-ported from the development version.
Enabled by adding qr'^MAIL$' or qr'^MAIL-UNDECIPHERABLE$' to the
list in $keep_decoded_original_re, as illustrated in amavisd.conf;
similar to a patch by Ted Cabeen. The following keys are used for a
lookup into $keep_decoded_original_re:
'MAIL'
always provide a full original message to virus scanners
(besides its successfully decoded components);
'MAIL-UNDECIPHERABLE'
same as for 'MAIL', but only if it contains undecipherable components
such as password protected archive members, unsupported compression
methods or encrypted parts (e.g. with PGP). Don't put too much trust
into this, as some more exotic file formats may not be understood
and not flagged as undecipherable;
* a key 'UNDECIPHERABLE' is matched against $banned_filename_re when mail
contains any undecipherable components, and if lookup returns true,
mail will be banned. For example:
$banned_filename_re = new_RE(
qr'^UNDECIPHERABLE$', # contains any undecipherable components
... );
* a string can be prepended to Subject (for local recipients only)
if mail could not be decoded or checked entirely, e.g. due to
password-protected archives or non-decodable mail bombs:
$undecipherable_subject_tag = '***UNCHECKED*** '; # undef disables it
NOTE: this solution is a quick-fix response to popular demand.
Although the same or similar functionality will probably remain
in future versions, the syntax and exact semantics may be refined.
- bring do_unarj in line with the rest of de-archivers: provide the
same degree of mail bomb protection that was available in do_unrar;
retain original archive for inspection by a virus scanner if it contains
any members that can not be extracted (e.g. password protected members
or unsupported compression schemes). (The .arj size checking deficiency
was mentioned in the AMaViS Security Announcement (ASA) 2004-01-19);
- don't call a virus scanner if there are no files (no mail parts) to scan;
it caused problems with certain scanners like aveclient, which expect
a list of file names as arguments and complain if the list is empty;
reported by Daniel Luttermann;
- a much needed feature: can specify $sa_dsn_cutoff_level in amavisd.conf
to suppress sending a DSN (delivery status notification) when spam level
is above this value, effectively turning D_BOUNCE into D_DISCARD for
this message; undef disables the feature and is a default. A good
first approximation value is 10, or with some risk go down to 8.
The parameter has no effect if DSNs are already disabled (e.g. when
$final_spam_destiny is D_DISCARD or D_REJECT);
- do_unrar: double check the archive size (against summary line as well);
- derive the value of macro %l from sender_source instead of from a declared
sender address, so as to not rise false alarms when sender address is known
to be faked; patch by Joerg Friedrich;
- reduce the number of retries to connect to a daemonized virus scanner
from 3 to 2, so that a fallback to backup scanners occurs sooner;
- updated $banned_filename_re example in amavisd.conf, modify it to will;
- updates to the @av_scanners list in amavisd.conf:
* KasperskyLab AVP - aveclient: *IMPORTANT* updated entry for the exit
status only reflects the result of the last file scanned, we must use
regular expression to detect viruses; fix provided by Andreas Triller;
* Trend Micro FileScanner - vscan: added option -za, otherwise
some broken archive may sneak-in a virus; suggested by Stephane Lentz;
* Sophos sweep: added options -cab -tnef --no-reset-atime;
* Dr.Web command line scanner: updated options, recognize exit status
when using evaluation license;
* ClamAV clamd: change socket location to a more usual value, adjust to will;
- amavisd.conf: added new virus names to the $viruses_that_fake_sender_re
list, and uncommented the [qr/.*/=>1] line, so that by default any unknown
virus will be treated as a sender-faking virus; adjust to will;
- do not send bad header notifications in response to messages
from mailing lists;
- added header check for folded header field lines made up entirely of
whitespace (a 'header space gap' violation to rfc2822); a check is only
made when other header checks (bad character in header) are enabled;
- distinguish an empty string from undef in $mailfrom_to_quarantine,
making it possible to specify a null return path when quarantining
to a mailbox;
- helper-progs/amavis-milter.c (sendmail) enhancement based on work
by Stephane Lentz:
* the name of a temporary directory is derived from the sendmail queue ID,
making it easier to match sendmail and amavisd-new log entries;
the queue ID is also a part of the quarantine file name;
* a phantom 'Received:' header field is prepended on the temporary file
to preserve the information on the original SMTP client IP address,
host name and queue ID. This trace header field does not propagate to
recipients (is not inserted into the original message), but is available
in the quarantined messages and is visible to SpamAssassin.
Adding a milter macro {b} on ENVFROM is advised to preserve the MTA
timestamp in the log, although not mandatory (falls back to current time):
define(`confMILTER_MACROS_ENVFROM',
confMILTER_MACROS_ENVFROM``, {b}'')dnl
If the new amavis-milter.c gives you trouble, switch the soft link to the
previous version in the same directory. Keeping macro {b} does not hurt.
- catch possible errors in pre_loop_hook and report them properly;
- better check for I/O errors on SMTP input socket;
- add the following modules to the list of pre-fetched modules:
Carp::Heavy, IO::Handle, IO::Socket::UNIX, IO::Socket::INET.
The absence of Carp::Heavy could mute error report or a backtrace
when running chroot-ed; the rest are for completeness only;
- added a macro %z which expands to the original mail size (in bytes);
it could be useful in the $log_templ; thanks to Nick Leverton;
- make SpamAssassin timeout value configurable: variable $sa_timeout;
by Henrique M. Holschuh (Debian);
- 'Received:' header field cosmetics: use 'unknown' in case the HELO argument
was empty, and put-in a 'unix socket' if message was received from a helper
program; by Henrique M. Holschuh (Debian);
- remove more pre-existing X-Spam* header fields from other scanners:
X-Spam-Tests, X-Scanned-By; by Henrique M. Holschuh (Debian);
- helper-progs/amavis.c updates (old sendmail setup): report errors,
log to syslog, change default dir to something obvious for bug reporting;
by Henrique M. Holschuh (Debian);
- updated documentation;
---------------------------------------------------------------------------
January 5, 2004
Patch: amavisd-new-20030616-p7
- do_unzip and do_unrar: retain an archive if any of its components is
password protected or encrypted (plus unpack what can be unpacked,
as before). This gives virus checkers a chance to examine the original
unpacked archive as a whole (e.g. scanning for variants of W32/Mimail),
matching it in non-decoded form against virus patterns even if containing
password-protected components. As a consequence, some virus scanners
may now log their complaint when encountering such protected archives
which previously didn't reach them. Such log entries should be considered
informational only;
- add module Net::DNS::RR::AAAA to the list of Perl modules to be fetched
before chroot takes place; thanks to Per olof Ljungmark;
- preload Perl modules DBD::*, based on @lookup_sql_dsn;
required when using SQL lookups from a chroot jail;
- updated example dsn in @lookup_sql_dsn (file amavisd.conf) to use the
new DBD::mysql syntax, and to show how to force accessing SQL server
via inet socket which makes it easier to use from a chroot jail;
- disregard cached spam results for mail with small body.
The most pronounced undesirable effect was in mail with
an empty body where spam score was derived from header only;
- change the default value for local_domains_sql lookup for the catchall key
'@.' under conditions: when using SQL lookups and user record with key '@.'
is present in the database and a field 'local' is not present. Previously
it surprisingly defaulted to true, now it falls back to static lookup table
defaults, the same as if the record '@.' were not present in the table;
- fixed ugly text formatting in recipient notifications template;
reported by Florian Effenberger;
- fixed parsing of 'Received:' header field for some unusual cases;
- updated amavisd-new-courier.patch to apply cleanly against -P6 (now P7);
by Martin Orr;
- added av_scanners entry for 'AVG Anti-Virus', kindly provided by
Grisoft s.r.o. from Czech Republic, http://www.grisoft.com/
- make Dr.Web Daemon av entry work with evaluation or regular license;
thanks to Andrew I Baznikin;
- added av_scanners entry for new AVP client (aveclient) that is shipped
whith avp 5.x.x.x. Moved kavscanner to the @av_scanners_backup list
(it is presumably slow and less reliable than aveclient), updated other
AVP/Kaspersky entries; thanks to Nabil Sefrioui;
- added status codes 10 and 15 to the list of ok statuses for kavscanner
(10=Password-protected archives, 15=Corrupted files);
- add an example dummy virus scanner 'all-clean' to the @av_scanners_backup
list which always succeeds, always returning false (= status clean).
Uncomment it if desired to avoid mail requeue when all other scanners fail,
and to just pass the mail unchecked;
- disregard rfc2821 recommendation that 552 smtp response code
should be treated as 452. It is unnecessary in amavisd-new setups,
and it is wrong because 552 has other meanings assigned to it
besides "too many recipients";
- allow IP address in $forward_method and $notify_method to be bracketed,
which is needed for IPv6 addresses containing colons. Both formats
are now allowed: 'smtp:127.0.0.1:10025' and 'smtp:[127.0.0.1]:10025',
the later is now preferred;
- change IP address bracketing in log entries 'SEND via SMTP' and
'FWD via SMTP' from [127.0.0.1:10025] into [127.0.0.1]:10025 to match
the syntax in the connect log entry and to facilitate parsing of IPv6
addresses from the log;
- update received_line() to generate valid Received header field even for
IPv6 client addresses;
- modified helper program amavis.c to allow it to be run non-root and to
set temp file mode for group accessibility. Change its default log level
to DBG_WARN;
- fix an incorrect SELECT example in README.lookups; thanks to Nabil Sefrioui;
- documentation updates;
---------------------------------------------------------------------------
November 10, 2003
Patch: amavisd-new-20030616-p6
- change SQL lookup code to better handle SQL database server restarts.
After a SQL server restart amavisd-new would previously TEMPFAIL (4xx)
all messages until amavisd child process would run down, then resume
normal operation with the new child birth. Now the SQL server reconnect
is done when the next mail arrives, so only one mail with each amavisd
child process TEMPFAILs during SQL server restart, and normal operation
resumes faster;
- in flatten_and_tidy_dir and in do_ascii: sanitize protection of files and
directories which may otherwise be made inaccessible to virus scanners;
based on patch by Henrique de Moraes Holschuh (Debian amavisd-new support);
problem reported by Aspa and by Tomasz Papszun;
- fix a potential security problem: don't let rmdir_recursively and
rmdir_flat follow symbolic links; this might be exploited by attempting
to delete some foreign file using privileges of the amavisd process
(which should not be root);
- use cpio option -d when unpacking cpio archives;
- don't insert spam-related headers for passed mail to recipients with
bypass_spam_checks, making them believe the mail was not spam-checked
(as they are not expecting such headers anyway);
- text added to the banned mail sender notification template (at the end
of file amavisd), explaining to the sender what happened and how to avoid
the restriction; edit to will;
- amavis-milter.c program in the helper-progs subdirectory is now based
on the most recent version 1.1.2.3.2.40 from the AMaViS CVS (maintained
by Lars Hecking), but hacked a bit to make its options mostly compatible
with the previous version. Start it with option -h to get current usage text.
See helper-progs/README if using sendmail milter setup;
Please revert to the old one if the .40 gives you trouble;
both version are included in the helper-progs subdirectory.
- fix parsing of unrar info lines;
- consistency with other virus/banned logic: don't send recipient notification
(reporting banned name) if mail contains both a virus and a banned name,
but $warnvirusrecip is false; reported by Nathan G. Grennan and Urska Brinar;
- check for possible I/O errors when reading from SMTP socket,
and distinguish error condition from normal TCP session teardown;
- do not redirect stderr to /dev/null when calling file(1). This way the
diagnostics from file(1) will now at least be visible in the debug session;
- determine_file_types: make 'bzip compressed' pattern match older
bzip format (v1) as well; thanks to Davaeron;
- sub run_command: explicitly close STDERR before reopening it; this way
the reporting of possible problems in each operation would be separate,
and it seems to avoid a rare problem (open STDERR '/dev/null' failing)
reported by Sam Hart;
- don't attempt to do lookups in regexp table $viruses_that_fake_sender_re
if it is undefined; reported by Chris Paul;
- remove existing X-Spam-Score along with other X-Spam* header fields
if spam scanning is enabled;
- do not skip inserting 'X-Spam-Status: No, hits=- ... WHITELISTED'
for whitelisted senders which caused SA to be bypassed - artificially
assume score is -10 for the purpose of comparing it to tag_level;
reported by Mike Vanecek;
- explicitly qualify wblist.rid field in $sql_select_white_black_list
just in case someone has a field 'rid' in the other table;
- fix showing the value of LC_TYPE environment variable in the log,
show LC_CTYPE as well;
- SAVI-perl: remove MIME option from the default set of options.
Even with recent versions of Sophos SAVI there are cases where the
library goes into a spin while trying to decode broken MIME message
(same applies to Sophie - one may want to change its configuration);
- updated vfind entry in @av_scanners to work with the new version,
changing '--vexit {}' to '--vexit {}/*'; thanks to Lowell Filak;
- updated kavdaemon entry in amavisd.conf; thanks to Michael Hall and
Daniel Melanchthon;
- added option '-packed' to 'FRISK F-Prot Daemon' entry;
thanks to Manfred Gloiber;
- NAI uvscan entry: commented pre/post actions in the entry to show
how to remove environment variable LD_PRELOAD after finishing,
suggested by David Tilley; added option --mime, suggested by Max and
Kevin W. Gagel; added note on how to treat password-protected files
as viruses, by Seth Parker;
- added 'Dr.Web Daemon' entry which talks directly to Dr.Web daemon over
its Unix socket, speeding up a single-file check more than 200 times;
provided by Andrew I Baznikin;
- another entry for Symantec AntiVirus Scan Engine provided by Guido
R. Rolon A.; I'm not sure which is which, check your documentation;
- added 'dumaru', 'parite', 'gibe' and some other virus names to
$viruses_that_fake_sender_re;
- add examples to $viruses_that_fake_sender_re to show how to make
a default result true, and only list exceptions;
- placed a comment in amavisd.conf pointing out the proper syntax
for $hdrfrom_notify_* variables; thanks to Wouter de Jong;
- bump up the size of $sa_mail_body_size_limit to 150 kB in amavisd.conf
- documentation updates; fixed typos and spelling mistakes in the
documentation files and in amavisd.conf comments;
- new documentation file README.protocol, specifying the new (and the old)
protocol between helper programs and amavisd daemon, to be made available
with the next major release. The description of the current (traditional)
protocol was contributed by Stephane Lentz.
---------------------------------------------------------------------------
August 25, 2003
Patch: amavisd-new-20030616-p5
- fix 'Modification of non-creatable array value attempted' bug when
no 'Received' header field was present in an infected mail;
reported by Paul Miner;
- caching of SQL lookups on white/blacklist was based on sender address only,
instead of sender _and_ recipient. This could lead to white-/blacklisting
of one recipient to affect other recipients of the same message;
reported and debugged by Paul Gamble;
- added LDFLAGS to helper-progs/Makefile.in. It is needed at least on NetBSD.
Patch by Julian C. Dunn (the NetBSD package maintainer for amavisd-new);
- more obvious logging of HOLD reason in sendmail/milter setup
(reported by Pascal Martinez);
- $MAXLEVELS zero or undef should disable the limit according to docs,
but was not honoured; reported by Rob Hutton;
- if notifications delivery encounters a temporary failure (4xx), propagate
this status to the final result instead of only logging a warning;
- amavisd.conf:
* VirusBuster entry changed to match newer version of the product;
information from Marcus Schopen;
* another entry in amavisd.conf for KasperskyLab kavscanner (v4.5?),
contributed by Simone Marx;
---------------------------------------------------------------------------
August 12, 2003
Patch: amavisd-new-20030616-p4
- revert to using alarm() instead of Time::HiRes::alarm(). It is nonstandard
to mix the two, and is causing problems on some operating systems
(e.g. Solaris); thanks to Geoff Gibbs;
- rise log level for log entries on intentional mail drops in case a
DSN (a bounce) should be sent, but will not be; such as on rejected
bounces, viruses with forged names, and spam from mailing lists.
The new log entries now say: '... Not sending DSN ...' and provide
more information on the reason for dropping DSN;
- if sender is whitelisted, don't insert 'X-Spam-Flag: YES' header field,
don't append spam address extension, and don't quarantine. This makes it
less surprising, although previous behaviour was according to documentation.
NOTE: the documentation still describes former behaviour, this needs
to be fixed;
- report deaths of command line scanners and some external programs
distinctly from normal exits with nonzero exit status;
- fix replacing * or {}/* in the pattern with actual file names, causing
MkS_Vir (mks32), VFind, Dr.Web Daemon, and KasperskyLab aveclient to receive
its file arguments glued together; based on patch by Rafael J. Wysocki;
- update 'Panda Antivirus for Linux' entry to work with new (and older)
versions of pavcl; updated entry kindly provided by Panda Software;
- stop the timer after SMTP transaction is over to better behave in
persistent SMTP/LMTP sessions; start the timer at the beginning of a
SMTP transaction, in addition to restarting it when DATA mode is entered;
- sub mail_via_smtp_single: properly report SMTP response code when all
recipients are rejected by MTA, instead of logging a 'mail_via_smtp:'
without a value. The problem was commonly seen with the W32/Mimail-A
virus which fakes an often invalid local sender address and gets rejected
by Postfix outright; reported by Turgut Kalfaoglu;
- use SMTP response code 554 (instead of 550) for rejecting syntactically
invalid header (according to rfc2476);
- add am_id to SMTP response code generated by one_response_for_all()
to make it easier for MTA log to be correlated with amavisd-new log;
some cosmetic improvements in the generated SMTP response text;
- added 'Return-Path:' in notifications to make it more obvious to see
envelope address from reports;
- the 'Message-ID:' in neutral DSN notifications template was inadvertently
pushed into the DSN body;
- report undefined spam score in X-Spam-Status header field as 'hits=-'
instead of 'hits=0.0' which can be misleading;
- indicate blacklisting in X-Spam-Status header field of quarantined messages;
- add X-Envelope-From header field to quarantined messages;
- added virus names: tanatos, lentin, bridex (alternative names
for bugbear, yaha and braid) to the $viruses_that_fake_sender_re;
thanks to Harrie Overdijk;
- set environment variables LINES and COLUMNS to sensible defaults
to avoid some external program get puzzled about the terminal settings
(e.g. older versions of pavcl from Panda Software);
- another attempt at fixing the Subject header field duplication. The patch
amavisd-new-20030314-p2 fixed the case of entirely missing Subject header
field, but did not fix the case of Subject header field present but with
an empty text. Reported by Steven Cobb and Francois Rolland;
- rewritten 'Received' header fields parsing to better cope with valid,
as well as with more common cases of broken syntax; used when trying
to report originator IP address for believed-to-be-faked senders;
- more permissive parsing of SMTP addresses and options on MAIL FROM
and RCPT TO commands;
- 'neutral' (=space) field in SQL black/whitelists now terminates the lookup
search, avoiding fallback to static black/whitelists. It enables recipient
to explicitly express its neutral stance towards the sender, overruling the
site default;
- $sa_mail_body_size_limit now takes into account some portion of the
mail header size so that huge mail headers that can cause slow SA calls
are avoided (such degenerate cases were reported by Ralf Hildebrandt);
- taint fix in read_l10n_templates (as used by the Debian distribution),
patch by Henrique de Moraes Holschuh;
- don't send recipient notifications to recipients that have
bypass_virus_checks/bypass_banned_checks; suggested by Joe Breu;
- replace /bin/false with a more usual /usr/bin/false as a last resort exit;
- fetch modules 'Net::Ping' and 'bytes', which seem to be needed in certain
chrooted setups;
- change log level from 0 to 1 for the log entry 'BAD HEADER from';
reported by Thomas Lamy via Debian bug reports;
same for 'unrar: all %d members are encrypted';
- fix typo in variable name: $spamassasin_obj -> $spamassassin_obj
- explicitly set pipes and sockets to binmode, as is a default since Perl 5.8.1
- documentation updates:
* new file LDAP.schema, by Jacques Supcik, PhD
* updated README.chroot to tell that /dev/urandom is needed
in chroot jail (otherwise creation of MIME notifications fails);
thanks to Lynn Duerksen and Jimmy Porter;
* updated README.sendmail-dual, thanks to Robert LeBlanc and Stephane Lentz;
* added URLs of external programs to INSTALL;
* small updates to other doc files;
---------------------------------------------------------------------------
June 28, 2003
Patch: amavisd-new-20030616-p3
- avoid 'savemail: cannot save rejected email anywhere' sendmail panic when
feeding mail via LMTP and using D_BOUNCE settings in dual-sendmail setup.
---------------------------------------------------------------------------
June 27, 2003
Patch: amavisd-new-20030616-p2
- when running chrooted, fix pre-loading of modules needed by SpamAssassin
and Razor agents, which SpamAssassin forgets to pre-load by itself;
reported by Neil Camara;
- updated KasperskyLab AVPDaemonClient entry to protect against
carriage returns in collected virus names; thanks to Harrie Overdijk;
- updated 'Symantec CommandLineScanner' entry to look also for the new
scanner name 'savsecls'; thanks to Guido Rolon;
- documented extended uses of read_hash() in the release notes;
- updated README.chroot;
---------------------------------------------------------------------------
June 24, 2003
Patch: amavisd-new-20030616-p1
- bug fix: allow stderr to be joined with stdout (>&1), instead of creating
a file with a name "&1" when calling command-line scanners or doing
LHA decoding (it got broken in 20030616); noticed by Henrik Larsson;
- bug fix: the X-Spam-Level header field would carry 64 asterisks instead
of none when spam is configured to pass (mail tagging only) and when SA was
not called (e.g. sender is whitelisted, ...) and tag_level or tag2_level
is set to 0 or less; reported by Ralf Hildebrandt;
- bug fix: untaint e-mail addresses when forwarding via pipe;
reported by Sam Tilders;
- modify code to match documentation: $relayhost_is_client should influence
$notify_method too, not just $forward_method (adding an extra argument to
mail_dispatch() was necessary to make this possible); reported by Zhu Yicun;
- SQL failure modes consistency: when initial connect to SQL failed, previous
versions of amavisd-new would fall back to static defaults, disabling
SQL lookups. Now initial connect failure is fatal, just like if connect
failure occurred during operation - mail flow just stops during SQL servers
outage, and is resumed when servers become reachable again;
- commented-out adding of 'X-Spam-Report' header field to retain behaviour
from amavisd-new-20030314 - I find this header too long and intrusive
for regular use. Uncomment it if adding X-Spam-Report is desired;
- support lzop (.lzo) and freeze (.F) compressors if available;
- use external program cpio (if available) to efficiently and safely handle
the following archive formats: cpio binary, HPUX binary cpio, cpio crc,
old ASCII cpio, new ASCII cpio, and HPUX old ASCII cpio, as well as
POSIX.1 tar (also GNU tar) and old tar if allowed (see below)
without reading whole archive members into memory;
- if your cpio(1L) can read tar format (as is common on FreeBSD and Linux),
it is recommended to uncomment the following line (in file 'amavisd'):
# /^\.tar$/ && defined $cpio && return do_cpio($part,$tempdir);
which will cause tar archives to be safely and efficiently decoded
by cpio(1L) instead of Archive::Tar, which loads the whole archive
into memory;
- use separate file(1) 'classifications' ".uue" and ".hqx" (instead of ".asc")
for uuencoded and binhex formats;
- merge 'H+BEDV AntiVir' and 'CentralCommand Vexira Antivirus'
entries into a single entry - they are basically the same product
(the Vexira entry didn't work for new version of Vexira);
thanks to Vivek Khera;
- do_unrar: some switches to rar/unrar are not recognized by older versions
of rar/unrar; issue a more informative log when this happens, and retry
without newer switches for compatibility; thanks to Geoff Gibbs;
- crop backtrace on SA timeouts at 980 characters;
- preload Perl modules Net::DNS::* if spam scanning is enabled; they could not
be loaded if running in a chroot jail, so SpamAssassin did not use Razor;
- documentation: collected documentation pieces pertaining to LDAP
into README.lookups; fix SQL example table data in README.lookups;
( the patched MIME-tools by David F. Skoll is recommended over 5.411,
as it better handles broken/bad MIME syntax:
http://www.mimedefang.org/ -> Download section )
---------------------------------------------------------------------------
June 16, 2003
amavisd-new-20030616 release notes
This is mostly a maintenance release. Only a handful of new features
were added that were small or easy enough to provide.
- revised the incoming SMTP session abort procedure to properly shut down
the remote SMTP client in case of unexpected trouble (e.g. disk full)
during SMTP data reception;
- subroutine mail_via_smtp_single() overhaul to produce cleaner diagnostics
and to properly abort outgoing SMTP session in case timeout occurs
in SMTP 'DATA' mode (avoids recipients repeatedly receiving partial
message in successive delivery attempts when the receiving MTA responds
very slowly); problem reported by Scott Vintinner;
- fixed the generation of LMTP response code when D_BOUNCE was chosen,
but DSN was not actually sent because sender was thought to be faked
or mail came in from a mailing list. Since DSN was not really sent,
the final response remained at 5xx (causing bounce by MTA) instead of
being converted to 2xx as D_BOUNCE would suggests; reported by Peter Bates;
- security: avoid inadvertently untainting values (e.g. mail addresses)
at several places throughout the program;
- work around a Perl 5.8.0 taint bug where global variable $1 could become
tainted, making further untainting attempts unsuccessful;
- work around another Perl 5.8.0 taint bug where Encode::encode fills up
all available memory when given a tainted string with a non-encodable
character. New subroutine safe_encode() provides a wrapper around
the Encode::encode;
- treat 'Maximum number of files exceeded' the same as 'Exceeded storage
quota', i.e. inserting the X-Amavis-Hold header field; noted by
Christopher Odenbach;
- call exec with explicitly specified program path, preventing exec
from even considering calling a shell;
- catch and better report the Perl's failure to fork on open;
- avoid inappropriately reporting 'Illegal seek' in run_av (and elsewhere)
even though nothing went wrong; reported by William Yodlowsky;
- new subroutine flatten_and_tidy_dir() recursively descends into a directory
containing potentially unsafe files with unpredictable names, soft links,
etc., rename each regular nonempty file giving it a generated name, and
discard all the rest.
- do_unrar and do_unarj now let the archiver itself recursively unpack
the archive and then use flatten_and_tidy_dir() to tidy up the result.
With previous method it was not possible to extract archive members
with names containing non-ASCII characters from rar archives due to
a bug in unrar and rar programs. Problem reported by Pan Bambaryla.
Protection from mail bombs in do_unrar is retained.
- do_unrar: use program rar or unrar, whichever is available. Both
rar and unrar recognize the same options to extract. (amavisd.conf:
$unrar=['rar','unrar'] ). Under some circumstances unrar falls in a
loop while rar extracts the archive correctly.
- do_unrar: don't bail out on exit status 1 (warning) from unrar after
collecting file names from the archive. Be more careful when extracting
file names, such as archive members with names starting with '-' or spaces;
Disable showing archive comment and file comments which can easily
break member-name parsing. Avoid extracting files in subdirectories
twice: once when directory itself is listed, and the second time when
each file in a directory is listed. Licence note: unrar is free,
rar is not ( see http://www.rarlab.com/ for fresh unrar sources ).
- do_unarj: prefer more versatile arj over unarj (amavisd.conf:
$unarj=['arj','unarj'] ). Both programs support subcommand 'e' that we use;
suggested by Guillem Jover (via Debian support); ARJ since 2.78/3.10
is Open Source (GPL license). When arj is available (recognizing several
new archive types, archive versions and options), put these options
to good use, recovering several archive members that are unaccessible
by the demo program unarj, such as old member versions, memebers with
equal names, etc. Disable showing comments.
- add X-Virus-Scanned header only if a mail was actually scanned
for viruses (i.e. av scan not bypassed for this set of recipients);
pointed out by Phil Regnauld;
- updated entry for 'KasperskyLab AvpTeamDream' virus scanner;
by Daniel Melanchthon;
- truncate instead of round the spam level when producing the bar
in the X-Spam-Level header field for compatibility with SA.
Clip the level bar at 64 characters instead of 60 (more would not
be allowed by RFC 2822, although SA used to crop at 100);
- subroutine hdr() incorrectly fixed an illegal header field body
which did not have a space after a line fold; noticed by Virna Gupta;
(the buglet did not show unless one modified the program);
- replace several remaining regular expressions /...$/ with /...$(?!\n)/
as a matter of principle (as was done in amavisd-new-20030314-p2 for the
more urgent cases). The use of simple $ was almost always subtly
semantically wrong, but fortunately in most cases without consequences;
- specify open mode explicitly in several 'open's, and protect special
characters in file names from being 'too cleverly' interpreted by
Perl open (still not all cases done);
- SpamAssassin changes umask to 0077 - restore our mask after the SA call;
- be a bit more careful regarding platform-independency, distinguishing
between use of \012 and \015 against \r and \n (note the \n on Mac is CR);
- change log level of several less important av scanner log messages
from 2 to 3;
- small documentation updates in amavisd,conf;
- updated README.lookups: the new substring capture and reference mechanism
in regexp lookups explained. The SQL example modified to work under
PostgreSQL, thanks to Phil Regnauld;
- updated documentation: README.sendmail-dual, README.postfix, README.chroot,
new file: README.exim_v4_app2 - by Louis Erickson;
NEW FEATURES
- provide a command line option 'debug-sa', which is equivalent
(but more convenient) then setting $sa_debug to true and starting
as '# amavisd foreground';
- list version numbers of the more important Perl modules at startup;
- lookup_re(): RE pattern now allows for capturing of parenthesized substrings,
which can then be referenced from the result string using the $1, $2, ...
notation, as with the Perl m// operator. The number after the $ may be
a multi-digit number. To avoid possible ambiguity the ${n} or $(n) form
may be used. Substring numbering starts with 1. Nonexistent references
evaluate to empty strings. If any substitution is done, the result inherits
the taintedness of the key. Keep in mind that the $ character needs to be
backslash-quoted in qq() strings (but not in q() strings). Example:
$virus_quarantine_to = new_RE(
[ qr'^(.*)@example\.com$'i => 'virus-${1}@example.com' ],
[ qr'^(.*)(@[^@]*)?$'i => 'virus-${1}${2}' ] );
- read_hash() now returns the hashref (first argument) also as a return
value, making it easier to create new hashes (instead of only adding values
to existing ones). Also the first argument (hashref) is now optional,
and if missing a new hash is created. The following three cases are now
equivalent:
a) $sa_tag2_level_deflt = {};
read_hash($sa_tag2_level_deflt, '/var/amavis/tag2_levels.dat');
b) $sa_tag2_level_deflt = read_hash({}, '/var/amavis/tag2_levels.dat');
c) $sa_tag2_level_deflt = read_hash('/var/amavis/tag2_levels.dat');
- extend the semantics of the configuration variable $inet_socket_port,
which can now either be a scalar as before (i.e. a single port number),
or a ref to a list of port numbers, e.g:
$inet_socket_port=[10024,10026,10028]; # accept SMTP on all these TCP port
When $relayhost_is_client is true, the semantics of $forward_method has
changed slightly: instead of taking the port number from $forward_method,
it is now calculated as being one higher than the port number on which
the incoming SMTP connection came in (one from the $inet_socket_port list).
This allows for multiple MTA pairs on the same host to share the same
amavisd daemon without having to use multiple loopback interfaces.
Useful for example if incoming and outgoing mail is handled by separate
mailers on the same host.
- add field 'spam_quarantine_to' to SQL table 'policy' - a patch
provided by Vivek Khera. For compatibility a missing field defaults
to undef, causing lookup search to proceed with the next lookup map;
- add fields: spam_lover, banned_files_lover, bad_header_lover,
bypass_banned_checks and bypass_header_checks to SQL table 'policy'
for completeness; For compatibility a missing field defaults to undef,
causing lookup search to proceed with the next lookup map;
- add a 'SPAM-TAG, ...' log entry at log level 2, which is produced when
inserting spam-related header fields for each cluster of recipients
with same settings. It complements the 'SPAM, ...' log entry, which
is triggered at kill level. Feel free to adjust their log levels
if you think one or the other is redundant;
- insert a 'X-Spam-Report' header field at tag2 level when spam is passed.
Feel free to comment out the ...append_header('X-Spam-Report'... line
if this is undesired.
- fold-in the 'bad_headers.patch' to the main code
(available in the distribution since amavisd-new-20030314-p1),
but renamed *bad_headers_lovers* to *bad_header_lovers* and added
$final_bad_header_destiny and *bypass_header_checks*.
To refresh memory:
Enables checking headers of each mail for invalid (non-encoded) 8-bit
characters, and produces a bounce (non-delivery status notification,
or a SMTP REJECT if desired) with the full explanation of the problem,
with offending header fields trimmed, sanitized and included in the text.
New setting $final_bad_header_destiny, defaults to D_PASS for backwards
compatibility, but a value D_BOUNCE is suggested. Similarly to other
*lovers* settings, a hash and an ACL lookup %bad_header_lovers and
\@bad_header_lovers_acl are available, and a setting $warnbannedsender.
Certain recipients may be exempt from the checking (*bad_header_lovers*).
Similarly mail from mailing lists (Precedence: list or bulk), and mail
with null reverse-path mail (e.g. bounces) is passed, even if violating
the RFC 2822 header syntax. A log entry is produced nevertheless.
Postfix users: this is similar to the Postfix strict_7bit_headers=yes
functionality, but produces a much more informative problem report
(non-delivery notification) to the sender. One difference is that
amavisd-new header check takes into account only the RFC 2822 header,
not MIME headers in the mail body. It is reasonably efficient to use
amavisd-new as a header checker only, without any anti-virus or anti-spam
checks, if desired.
- added configuration settings $warnbannedsender and $warnbannedrecip,
separating this function from $warnvirussender and $warnvirusrecip,
which previously applied to both virus and banned files;
- new macro %c which evaluates to spam level/hits (mnemonic: sCore)
as provided by SpamAssassin; useful in $log_templ;
- new configuration settings *bypass_banned_checks* and *bypass_header_checks*;
- new parameter $spam_quarantine_bysender_to (in contrast to the usual
$spam_quarantine_to) makes possible collecting quarantined spam
by sender names or domains, instead of (or in addition to) the more
usual by-recipient map;
- new configuration setting $remove_existing_spam_headers (defaults to true);
- bring-in the localization (l10n) support contribution from Debian Linux,
by Henrique de Moraes Holschuh; see README.l10n, and comments at the
call to read_l10n_templates() in file amavisd.conf;
- supply X-Envelope-From and X-Envelope-To header fields containing envelope
addresses to the messaged passed to SpamAssassin for checking;
- there is now an official OID: 1.3.6.1.4.1.15312.2.2
under which LDAP schema for amavisd-new LDAP lookups is to be defined
(iso.org.dod.internet.private.enterprise. 'ijs.amavisd-new.ldap' );
---------------------------------------------------------------------------
May 6, 2003
Patch: amavisd-new-20030314-p2
- blacklisted sender only caused spam headers to be inserted,
but did not cause rejecting the message and other 'evasive actions'
due to over-optimization; reported by Lawrence Farr, Robin Elfrink
and Eric Vollmer;
- fix improper evaluation of bypass_spam_checks* in multi-recipient mail
where some recipients match 'bypass_spam_checks*' and others do not.
Spam-check was bypassed if _any_ (instead of _all_) recipient matched
'bypass_spam_checks*'; thanks to Joseph W. Breu;
(it got broken in amavisd-new-20030314, the amavisd-new-20021227 was ok);
- adding address extensions was not done for *_lovers, only for
*_destiny=D_PASS; reported by Steve Khoo. The "deal_with_spam" section
needed to be overhauled for this reason. For consistency with the
documentation and with the change in adding address extensions,
quarantining is now done if mail is considered spam, regardless of
it being delivered (e.g. to spam lovers), or not;
- if only spam checking is performed, but no virus scanning nor banned
file names checking is enabled, and the spam is detected but configured
to pass, and the Subject is configured to be edited, the Subject header
field was duplicated because the existing Subject header field was not
detected.
Solution: make MIME entity available even if virus and banned files
scanning is not performed. Header information is needed by
add_forwarding_header_edits_per_recip() when deciding whether to add
or replace the Subject header field in case spam is passed.
(workaround for previous versions: don't leave $banned_filename_re
undefined. If not needed, set it to an empty list: new_RE() );
- when quarantining to a mbox file, excape 'From ' with '>' only if it
follows an empty line, as sendmail mail.local man page prescribes;
- added parameter $notify_xmailer_header to control the body of the header
field X-Mailer as placed in the notification messages (e.g. bounces)
generated by amavisd-new; by default this header is no longer inserted,
as it was felt it revealed too much information. Choose from:
$notify_xmailer_header = undef; # no X-Mailer inserted (default)
$notify_xmailer_header = ''; # X-Mailer: MIME-tools x.xxx (Entity x.xxx)
$notify_xmailer_header = 'your-text-here'; # X-Mailer: your-text-here
- replace regular expressions /...$/ with /...$(?!\n)/ or with string
equality operator in some places (many more to come in the next major
release); the use of simple $ was almost always subtly semantically
wrong, but in most cases without consequences;
- enable timeouts after SMTP connection has been established, to be able
to handle cases where MTA is very slow in accepting checked mail,
which could result in multiple deliveries if MTA on the input side
times out;
- debugging aids: call stack backtrace is logged if
Mail::SpamAssassin::NoMailAudit::check exceeds allowed time;
- LDAP enhancement: allow "hostname" to be a string or a reference to an
array, in which case each entry will be tried in order until a connection
is made; by Jacques Supcik;
- X-Spam-Status now says WHITELISTED if sender was whitelisted;
contributed by Joseph W. Breu;
- if mail rejection was due to sender blacklisting, say so in the SMTP
response; thanks to Robin Elfrink;
- updated file bad_headers.patch to apply cleanly against this version.
The bad_headers.patch will be folded-in with the next major release,
please try it out, comments welcome. For patch description see release
notes for amavisd-new-20030314-p1;
- specify option 'ExactAddresses=>1' in a call to Net::SMTP::new()
in anticipation for the next release of Net::SMTP;
- amavisd.conf: updated documentation / comments;
- amavisd.conf: removed option '-ni' in the DrWeb virus scanner entry;
thanks to Mike Boev and the FreeBSD community
( http://www.freebsd.org/cgi/query-pr.cgi?pr=50893 )
- amavisd.conf: added entry for 'MkS_Vir daemon', thanks to Dariusz Grzegorski;
- amavisd.conf: updated entry for 'F-Secure Antivirus' to cope with fsav 4.50
as well as with 4.1x; solution by Juhan Tamsalu;
- documentation: updated README.chroot; new README.sendmail describing
dual-MTA sendmail setup, small updates to other doc files;
March 21, 2003
Patch: amavisd-new-20030314-p1
- fix passing of remote MTA reject code back to the input side
(DSN was not sent if MTA rejected the message);
- fix broken logic which allowed a virus to pass if a recipient was a
'banned file lover' and the same message also contained a 'banned file';
(or the other way around: pass a banned file if containing a virus and sent
to virus lovers); thanks to Ortwin Gentz, I'm glad somebody is looking
over my shoulder;
- remove a log entry: one_response_for_all: SHOULDN'T HAPPEN
The test was inappropriate, but harmless;
- enhanced sub run_av() to fix a problem with MkS_Vir virus scanner,
where a '*' in the template command was not expanded to actual
file names; reported by Pan Bambaryla;
- trim virus scanner output (in sub run_av) to a manageable size
of 900 characters so as not to clutter log entries and notifications
too much; provoked by the Panda pavcl;
- move the check for 'Precedence:' mail header field from
sub delivery_status_notification() to its caller for flexibility;
- replaced the text in notification templates:
The message WAS delivered to:
into:
The message WILL BE delivered to:
to reflect the truth. The message may still be rejected
by the MTA when forwarding takes place.
- small documentations edits/updates;
- cosmetic: trim trailing whitespace in the program;
- cosmetic: make shorthand subroutine nf() locally scoped; by Jacques Supcik;
- make available an experimental patch (file: bad_headers.patch,
not applied by default), which, when applied (with patch(1) utility)
to amavisd-new, makes it check headers of each mail for invalid
(non-encoded) 8-bit characters, and produces a bounce (non-delivery
status notification, or a SMTP REJECT if desired) with the full
explanation of the problem, with offending header fields trimmed,
sanitized and included in the text.
Certain recipients may be exempt from the checking (*bad_header_lovers*).
Similarly mail from mailing lists (Precedence: list or bulk), and mail
with null reverse-path mail (e.g. bounces) is passed, even if violating
the RFC 2822 header syntax. A log entry is produced nevertheless.
Postfix users: this is similar to the Postfix strict_7bit_headers=yes
functionality, but produces a much more informative problem report
(non-delivery notification) to the sender. One difference is that
amavisd-new header check takes into account only the RFC 2822 header,
not MIME headers in the mail body. It is reasonably efficient to use
amavisd-new as a header checker only, without any anti-virus or anti-spam
checks, if desired.
The patch is experimental in the sense that it may change in future
versions, but is fully functional when applied;
- Jacques Supcik cleaned up and generalized his LDAP lookups
support code (which is off by default). In his words:
The major change is in the definition of ldap lookups.
In the config file, you have to enable ldap first:
| $enable_ldap = 1;
Then you can define defaults for your ldap queries:
| $default_ldap = {
| hostname => 'localhost', tls => 0,
| base => 'ou=hosting,dc=example,dc=com', scope => 'sub',
| query_filter => '(&(objectClass=amavisAccount)(mail=%m))'
| };
And then the lookups themselves:
| $virus_lovers_ldap = {res_attr => 'amavisVirusLover'};
| $banned_files_lovers_ldap = {res_attr => 'amavisBannedFilesLover'};
| ...
With this method, you can define every parameter individually.
You could have a different ldap server for each lookup! Like that,
the configuration is closer to the one in postfix or courier.
The hashes are converted into lookups objects in the
Amavis::Lookup::LDAP class method.
---------------------------------------------------------------------------
March 14, 2003
amavisd-new-20030314 release notes
MAIN NEW FEATURES AT A GLANCE
- per-user white- and blacklisting, including a SQL lookup mechanism;
refined semantics for white/blacklists;
- easier to run in chroot jail, see README.chroot;
- can specify two sets of virus scanners: the main list and a backup list,
which gets consulted only if all scanners from the main list fail;
- split REJECT into separate destiny settings: REJECT and BOUNCE,
giving you a choice of who is responsible for sending a non-delivery
notifications to the sender; see comments in amavisd.conf;
- edited configure file amavisd.conf, providing more informative comments,
cleaner examples and simplified common settings;
- made it work better with the Unicode-aware Perl 5.8.x (not against it)
and distinguish between character data / byte data / octet data (binary),
provide UTF-8 (Unicode) support for notification messages,
while avoiding some possible nasty surprises in the UTF-8 locale
(which is enabled by default in Red Hat 8.0);
- SQL SELECT clause settable from the config file;
- can store quarantined or forwarded messages as BSMTP files;
Changes since amavisd-new-20021227 patch level 2
BUG FIXES
- fixed caching problem when bypass_*_checks are used selectively: when only
one of the virus/spam tests were performed but the other bypassed due to
a per-recipient setting, the result of a non-performed test was inserted
into cache as clean. This could cause tests to be bypassed for subsequent
message with the same body but different recipients with different bypass*
settings, but only within a lifetime of a child process (10 messages
by default). Noticed by Jürgen Louis Fluk, thanks!
- changed caching of SQL lookups to match the documented behaviour,
i.e. SQL lookup result is cached (for the benefit of field lookups)
for a single message only, and no longer for the child lifetime.
It is possible to revert to the previous behaviour by commenting-out
one line (look for $sql->clear_cache). Pointed out by Paul Miner;
- see also further down about the field 'banned_files_lover'
in the SQL table 'policy', and the UTF-8 (Unicode) workarounds.
SECURITY
- running amavisd-new in chroot jail is now easier to set up due to added
configuration variable $MYHOME and its simplification effects on other
settings in the config file. Relevant settings are better commented;
- avoid the need to have a shell in chroot jail (use of the new subroutine
run_command() instead of Perl functions qx() and system() );
- new file README.chroot with guidelines on how to run amavisd-new
in a chroot jail, to prevent possible vulnerabilities from affecting
the rest of your system;
- added a security notice (web page and other places) that the Unix utility
file(1) 3.39 and older contains an exploitable security vulnerability,
which can cause system access with access rights of the user running
amavisd-new daemon (not root, unless you violated the recommendations).
Please upgrade file(1) to 3.41 or newer:
ftp://ftp.astron.com/pub/file/file-3.41.tar.gz
- new section (Security considerations) on the web page;
CLEANUPS AND CHANGES IN amavisd.conf
- improve explanation texts (comments) in amavisd.conf,
edit the file to improve legibility and structure, change some examples.
The daemon is thought to be compatible with amavisd.conf from previous
version, but it may be worthwhile to switch to the new file to avoid
asking FAQ;
- introduce a configuration variable $MYHOME, which is never used directly
by amavisd daemon, but is a handy way of setting default values for
other configurable variables in the config file. Also makes default
settings more chroot-friendly.
- introduce a configuration variable $mydomain, which is never used directly
by amavisd daemon, but is a handy way of setting default values for
other configurable variables when there is only a single domain (with
subdomains) behind the mailer. This avoids the need to edit half-a-dozen
configuration variables in simple setups;
- moved debugging-related settings to a separate section
(Section VIII - Debugging);
- make SQL SELECT clause settable from the config file;
- renamed variable @local_domains to @local_domains_acl for consistency;
The old name became a synonym for @local_domains_acl, so compatibility
with existing config files is retained;
- comment out the example: read_hash(\%whitelist_sender, ...);
it slipped into the amavisd-new-20021227 released code from my tests;
- comment out the setting $relayhost_is_client=1; It is only useful
for multi-access daemon setup, and potentially confusing for others;
VIRUS SCANNERS
- introduced a list of backup virus scanners: @av_scanners_backup,
complementing the normal list of virus scanners.
If no virus scanners from the @av_scanners list produce 'clean' nor
'infected' status (e.g. they all fail to run or the list is empty),
then all scanners in the @av_scanners_backup list are tried.
When there are both daemonized and command-line scanners available,
it is customary to place slower command-line scanners in the
@av_scanners_backup list;
- removed the Mac and SafeMacDfHandling options from SAVI-Perl,
as they are no longer supported by Sophos library since 2003-03;
set new option NamespaceSupport to SOPHOS_DOS_AND_MAC_FILES;
thanks to Paul B. Henson;
- changed a default to initialize SAVI-Perl only at startup time,
and no longer every time the child process is started. This is because
starting with SAVI 3.0 (March 2003) the initialization is slower.
NOTE: it is now necessary to restart amavisd master process
after SAVI database is updated, to make it notice new virus descriptions.
- in 'MkS_Vir for Linux' entry replaced options -e -c with -s,
to make mks work, suggested by Pan Bambaryla;
- added status 8 ('corrupted') to the list of clean statuses for
'KasperskyLab AVPDaemonClient', as is already the case for some other
virus scanners. See amavisd-new web page (Security considerations)
if the idea bothers you;
- RAV command line scanner no longer supported off-the-shelf,
as interfacing to it seems to be violating their license terms.
Btw, if RAV command line scanner does not work as user amavis,
check permissions and ownerships of /usr/local/rav8/* or reinstall
RAV as the user that will run amavisd; thanks to Oyku Gencay;
- RAV does not like '</dev/null' (segfault) (reported by Johnny Ljunggren),
most others don't care, but possibly Panda needs it. Tried to remove it,
but later put is back, as I don't like external command to have access
to the amavisd socket over which SMTP or AM.CL session is running;
- H+BEDV AntiVir: changed option -allfiles to --allfiles;
thanks to Dr Patrick Atlas, confirmed in the H+BEDV documentation;
- comment out the default entry for Sophos sweep, as the name 'sweep'
clashes with the Debian sweep package (audio samples editor).
If you uncomment the entry, make sure the correct 'sweep' is found
in the specified path; thanks to Guido Guenther and Brian May;
- added av definition for BitDefender (based on amavis-ng code by
Hilko Bengen); thanks to Stephane Lentz for testing and suggesting
command line options;
- added av definition for Ikarus AntiVirus for Linux, based on
amavisd-0.1 code by Lars Hecking;
PERL 5.8 UNICODE AND INTERNATIONALIZATION (I18N)
- make program aware of the type of data coming-in from external files,
pipes and sockets - distinguish between binary data, byte data, and
character data / plain text.
This is necessary in order to be able to survive in the UTF-8 locale-aware
Perl 5.8.0 environment (e.g. to avoid 'Malformed UTF-8 character'
and 'Wide character in print' warnings/errors, among others).
- check character ranges in regular expressions throughout the program,
and fix/rewrite them to avoid assumption that all characters are in the
range 0..255, which is no longer true! Some nasty surprises could have
been lurking in that corner.
Note the Perl 5.8.0 terminology (see perluniintro and Encode man page):
* character: a character in the range 0..(2**32-1) (or more)
(what Perl's strings are made of);
* byte: a character in the range 0..255
(a special case of a Perl character);
* octet: 8 bits of data, with ordinal values 0..255
(term for bytes passed to or from a non-Perl context, e.g. a disk file)
- modified sanitize_str() and hdr() to cope with UTF-8 characters;
- modified string_to_mime_entity() and hdr() to encode RFC2047-permitted
headers according to $hdr_encoding, and notification body according
to $bdy_encoding settings (both default to iso-88591-1, aka Latin1).
Notification template texts are supposed to be stored internally as
native or UTF-8 Perl strings, which is handled (mostly) transparently
by Perl 5.8.x;
- added option $hdr_encoding_qb to be able to explicitly specify
MIME encoding ("Q" or "B") for $hdr_encoding MIME charset;
- rfc2822_timestamp: produce correct date format regardless of
LC_TIME locale setting;
- read_text() subroutine (which may be used in amavisd.conf to read
notification text templates) may be given a second argument, specifying
character encoding used in the external file. If Perl is Unicode-aware
(5.8.0), this argument is given to binmode to influence the reading
of the file and its conversion into internal Perl string representation.
When notification text is generated from the internal Perl string
representations (UTF-8), the following settings influence the character
encoding (charset in MIME terminology):
$hdr_encoding = 'iso-8859-1'; # ... header field bodies
The header is first encoded into $hdr_encoding, which is then encoded
into 7-bit ASCII as required by RFC 2821 and defined by RFC 2047.
The specified encoding (charset) will appear in the RFC 2047 Q-encoded
strings in header fields where this is allowed, i.e. in Subject,
Comments, and X-* header fields. In other header fields where RFC 2047
encoding is not allowed, encoding into plain ASCII is used, where invalid
characters are treated as Perl Encode module does it, i.e. by converting
them to '?'.
$bdy_encoding = 'iso-8859-1'; # ... notification body text
Notification body text is encoded from the internal Perl representation
according to $bdy_encoding. The encoding (charset) is also inserted
in MIME header as Content-Type.charset.
OTHER
- make it work with Net::Server version 0.85, which changed the handling
of the CHLD signal, this is breaking qx and system calls used by previous
versions of amavisd-new;
- with Net::Server 0.85 you may see a message:
Net::Server: Couldn't POSIX::setuid to ... []
If the process UID remained 0 (root), the program will terminate,
otherwise just consider the message harmless (or complain to the
author of Net::Server);
- temporarily removed utility amavisdconf, it is not yet updated to cope
with the new configuration variables;
- determine_file_types: 'script text executable' is now considered .txt,
not .exe; thanks to Stephane Lentz and Michael Dengler;
- report correct system error if Convert::UUlib::decode fails
(e.g. permission denied); thanks to Nick Murtagh for figuring out
that UUDecode() in uunconc.c tries to decode into a temporary file
in /root/tmp/whatever;
- avoid the need for a Unix utility file(1) to support option -b;
suggested by Clifton Royston considering OpenBSD;
- new macro %b for customized notifications and log: it evaluates to
the message digest of mail body (MD5, hex). Used as a caching key,
can serve to identify messages in quarantine with same body text.
- provide a new BSMTP output method that can be used as $forward_method,
$notify_method, $virus_quarantine_method or $spam_quarantine_method.
It is simple in a sense that it generates one file for each message,
so it is not terribly efficient. The parameter after the colon is used
as a file name prototype, where %i in the prototype name is replaced
by current time, %n by amavis internal message id, and %b by message
digest of the body (MD5,hex). For example:
$forward_method = "bsmtp:$TEMPBASE/out-%i-%n.bsmtp";
$spam_quarantine_method = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp";
The bsmtp file has ".tmp" appended to a name during writing, and is
renamed to the final name when it is done, to avoid being seen by possible
fetcher prematurely. It is experimental in a sense that configuration
and implementation details may change in the future, including possible
changes in the old 'local:' method;
- besides the default (implied) quarantine method 'local:', it is now possible
to specify 'bsmtp:...' as a quarantine method, and have different settings
for virus and spam quarantines. New settings: $virus_quarantine_method
and $spam_quarantine_method, e.g.:
$virus_quarantine_method = "bsmtp:$QUARANTINEDIR/virus-%i-%n.bsmtp";
$spam_quarantine_method = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp";
- if ***SPAM*** is to be inserted into Subject header field,
but the original mail does not contain the Subject field, insert one;
suggested by Christopher Odenbach;
- per-recipient SQL field and static flag (spam_modifies_subj) for modifying
Subject: header field when spam is passed;
- updated README.lookups, describing new SQL fields and tables;
- split kill_level setting into two independent settings: tag2_level,
and kill_level. A quick reference:
tag_level controls adding the X-Spam-Status and X-Spam-Level headers,
tag2_level controls adding 'X-Spam-Flag: YES', and editing Subject,
kill_level controls 'evasive actions' (reject, quarantine, extensions);
The default is tag2_level=kill_level, which retains compatibility.
- work around a bug in Perl 5.8.0 in UTF-8 environment, which could
produce error "Bad RFC822 field name 'C0c0\x{0}\x{0}T-Type'" when building
notifications (the Perl bug is triggered in Mail::Header::_tag_case
by statement $tag =~ s/\b([a-z]+)/\L\u$1/gio; reported on several
Red Hat 8.0 hosts, seemingly only on Intel platforms).
Many thanks to Chris Kloiber and Steve Sloan for making available
their test machines for debugging.
- replace all calls to qx by run_command, to avoid 'Malformed UTF-8 character'
when reading from external utilities that do not honour locale settings,
e.g. Unix file(1) utility, which may return non-printable characters with
codes above 127. This also ensures better error reporting.
run_command() moved into Util module, part of its code moved into a
new subroutine run_command_copy().
- improved diagnostics when certain external programs fail to run,
such as when fork or exec fails;
- new LDAP lookups (experimental), by Jacques Supcik, PhD,
IP-Plus Internet Services - Swisscom Enterprise Solutions Ltd,
Genfergasse 14, 3050 Bern, Switzerland (http://www.ip-plus.net/)
Many thanks for the contribution!
- changed enhanced status code 5.7.0 to 5.7.1 when message is considered
bad (virus, banned file, spam);
- when sending message via SMTP, don't go into DATA part if any RCPT TO
returned 4xx code, since without implementing proper queueing we wouldn't
be able to handle the situation unless message came in via LMTP;
- updated README.postfix to describe:
* the now recommended Postfix 2.0 setup with pre-cleanup service,
instead of the (post-)cleanup2;
* how to specify a backup content filter(s) in case the primary fails;
* the Postfix setting if you want X-Amavis-Hold header field to cause
message to be placed on hold by Postfix, e.g. when amavisd-new encounters
a mail bomb which can not be checked for viruses.
- unconditionally delete old headers X-Spam-Status, X-Spam-Level,
X-Spam-Flag and X-Spam-Report, when anti-spam code is enabled.
Previously deletion was done only when new headers were to be inserted.
- replace sloppy file accessibility tests (Perl functions -e, -f, -d, -r, ...)
with more reliable method, which also produces more informative
diagnostics;
- set $SIG{CHLD}='DEFAULT' in Net::Server hooks to make qx work
in Net::Server::Single environment (useful for debugging),
as well as with the version of 0.85 of Net::Server;
- adjust the log level in do_ascii() (reported by Pan Bambaryla),
and in few other places;
- fix $log_templ - an unconditional comma was causing the log entry to be
always issued, regardless of mail content; reported by Stephane Lentz;
- enhance reject/discard/pass destinies with reject/bounce/discard/pass,
making it more flexible to specify what action to perform on detection
of bad content; (thanks again to Chris Hastie for earlier discussion
on the topic); see explanations in amavisd.conf;
- added hash lookup table %local_domains for those that have a great number
of hosted domains and prefer to read a list from a separate file;
- a special shorthand provision when SQL lookups are used: when a match
for recipient address or domain is found in SQL tables (regardless of
field values), the recipient is considered local, regardless of static
@local_comains_acl or %local_domains lookup tables. This simplifies
life when a large number of dynamically changing domains is hosted.
To overrule this behaviour, add an explicit boolean field 'local'
to table 'users' (missing field defaults to true, meaning record presence
implies locality).
- changed SQL table 'policy' field name 'banned_files_lover' to a name
'banned_files_lovers' (added an 's') for consistency with variables
%banned_files_lovers and @banned_files_lovers_acl, and updated
README.lookups;
- SQL field treated as boolean now recognize '\0' character as false
(not to be confused with NULL SQL field, which is still mapped to undef);
- when generating Received: header field, move receiving port number
into the second comment field - some software were too picky
about the syntax of the first comment field ('TCP-info' in rfc2821);
- consistency: apply late 'string sanitation'; meaning that strings
such as virus names, banned attachment names, e-mail addresses, etc,
are sanitized at the point where the string is leaving amavisd-new,
not at the point where it enters is. This allows the quotation rules
to depend on the output properties (log entry, notification text,
SMTP response text, ...) / (ASCII, iso-8859-1, UTF-8, binary, ...);
- delegate responsibility to call sanitize_str to write_log,
instead of having a caller do it;
- wrap long syslog entries at 980 characters (less than 1023 minus prefix);
thanks to Ted Cabeen;
- skip spam-related actions if $spam_level is undefined (e.g. when
anti-spam code is not enabled) so that missing $sa_kill_level_deflt
setting in amavisd.conf won't cause a surprise. Thanks to Marcio Merlone.
- new version of amavis-milter.c based on amavis-milter.c 1.1.2.3.2.36
(from amavisd-0.1);
- use the /m regexp modifier by default when parsing av-scanner output
for clean and infected regular expressions, making it easier to specify
patterns, and to make it like the expression which collects virus names;
thanks to Daniel Hunziker for reporting a problem with Symantec
CommandLine Scanner;
- refined the semantics of sender whitelisting/blacklisting,
add per-recipient sender while/blacklisting mechanisms:
ENVELOPE SENDER WHITELISTING / BLACKLISTING - GLOBAL (RECIPIENT-INDEPENDENT)
WHITELISTING: use envelope sender lookups to ENSURE DELIVERY from whitelisted
senders even if the message is recognized as spam. Effectively, for the
specified senders, message recipients temporarily become 'spam_lovers', with
further processing being the same as otherwise specified for spam lovers.
It does not turn off inserting spam-related headers, if they are enabled.
BLACKLISTING: messages from specified senders are DECLARED SPAM.
Effectively, for messages from blacklisted senders, spam level
is artificially pushed high, and the normal spam processing applies,
resulting in 'X-Spam-Flag: YES', high 'X-Spam-Level' bar and other usual
reactions to spam, including possible rejection. If the message nevertheless
still passes (e.g. for spam loving recipients), it is tagged as BLACKLISTED
in the 'X-Spam-Status' header field, but the reported spam value and
set of tests in this report header field (if available from SpamAssassin,
which may not have been called) is not adjusted.
A sender may be both white- and blacklisted at the same time,
settings are independent. For example, being both white- and blacklisted,
message is delivered to recipients, but is tagged as spam.
If ALL recipients of the message either white- or blacklist the sender,
spam scanning (calling the SpamAssassin) is bypassed, saving on time.
ENVELOPE SENDER WHITELISTING / BLACKLISTING - PER-RECIPIENT
The same semantics as for global white/blacklisting applies, but this
time each recipient (or its domain, or subdomain, ...) can be given
an individual lookup table for matching senders. The per-recipient lookups
override the global lookups, which serve as a fallback default.
One can specify a two-level lookup table: the key for the outer table is
recipient, and the result should be an inner lookup table (hash or ACL
or RE), where the key used will be the sender.
Per-recipient white/blacklisting is also available as a SQL lookup.
See configuration variable $sql_select_white_black_list and README.lookups .
INCOMPATIBILITIES WITH amavisd-new-20021227
- the previous configuration file amavisd.conf from version
amavisd-new-20021227 is usable with amavisd-new-20030314.
Nevertheless it may be a good idea to adopt the new amavisd.conf,
as it describes new options, is cleaner and better commented;
- reordered hash lookups from:
user+foo@sub.example.com, user@sub.example.com,
sub.example.com, .sub.example.com, .example.com, .com, . ,
user+foo@, user@
into:
user+foo@sub.example.com, user@sub.example.com,
user+foo@, user@,
sub.example.com, .sub.example.com, .example.com, .com, .
to make user@ still useful even when catchall ( . ) is used,
and to bring it closer to the recommended SQL order;
Not likely to affect existing setups, as previously the combination
of user@ and catchall was not useful;
- SQL lookups:
* incompatible change (avoiding unexpected behaviour): when the requested
SQL field name does not exist in the SQL table, return a sensible default
value for boolean fields (according to field name and its type),
instead of 1 as previously. Logs a warning when no default is available.
Thanks to Ortwin Gentz for noticing the problem;
* new fields spam_tag2_level, spam_modifies_subj and local, with useful
defaults if absent, so it is not necessary to immediately update an
existing database;
* changed SQL table 'policy' field name 'banned_files_lover' to a name
'banned_files_lovers' (added an 's') for consistency with variables
%banned_files_lovers and @banned_files_lovers_acl;
---------------------------------------------------------------------------
amavisd-new-20021227-p2 (2003-01-10):
- sender notification were not sent for messages containing banned
filenames but no virus; fixed (thanks to Philip Ross);
- added virus name 'Sobig' to the default $viruses_that_fake_sender_re
(amavisd.conf);
- extended default $banned_filename_re to match also a longer
(4 character) middle file extension (e.g. Movie_0074.mpeg.pif)
(amavisd.conf);
- add explicit path to the program vscan in the 'Trend Micro FileScanner'
entry (amavisd.conf), thanks to Matthew Hobbs;
- improvements in the setup of helper programs (files configure.in,
Makefile.in, config.h, amavis.c and amavis-milter.c), by Radu Greab;
- add Sophos error code 527 ('Not supported in this SAVI implementation')
to the list of (mostly)harmless codes in the SAVI-Perl interface routine
(thanks to Paul B. Henson and Tim Winders for discussion);
- initialize SAVI-Perl module (if available) in each new child, instead of
only once during daemon startup time. This avoids the need to reload
amavisd for every change in the Sophos IDE database - at the expense
of additional SAVI-Perl startup time every $max_requests messages;
amavisd-new-20021227-p1 (2002-12-30):
fix the problem when using multiple virus scanners, where the
last scanner doesn't find a virus, but some other scanner does,
and amavisd-new (incorrectly) does not report a virus.
Thanks to Okke Timm, Henrik Larsson and Eduardas Paulavicius;
---------------------------------------------------------------------------
December 27, 2002
amavisd-new-20021227 release notes
amavisd-new-20021227 is primarily a maintenance release, consolidating
amavisd-new-20021116 with its mandatory patches, updating documentation,
but also brings a couple of niceties.
Changes since amavisd-new-20021116 patch level 5
- The new configure/makefile in the helper_progs subdirectory is
contributed by Stephane Lentz, based on MIMEDefang configure.in.
It is needed to build helper programs for sendmail (milter and nonmilter)
setups (but not needed for Postfix and Exim v4 setups, or any dual-MTA
setup). Similar to the prior work done by Ramiro Morales - thanks to both!
- Updated documentation (README, INSTALL, RELEASE_NOTES, README.lookups,
README.customize, README.postfix).
- new subdirectory test-messages, containing sample mail messages to test
virus and spam detection - see README there
- For daemonized antivirus scanners (only) the semantics of few parameters
in the @av_scanners list has been incompatibly changed and enhanced.
The same internal mechanism now handles both Unix sockets and inet socket.
More than one socket may now be specified, to be tried one after the other
until one is successfully connected to (useful for 'FRISK F-Prot Daemon').
Also the usual regexp pattern parameters in the @av_scanners are now
observed by Sophie and Trophie interfaces as well, bringing them more
in line with other scanners.
- The (?m) regexp flag in @av_scanners patterns is now turned-on
by default, so it is no longer necessary to specify it in @av_scanners,
and has been removed to shorten the expressions.
- The SAVI-Perl interface routine now turns on most Sophos engine options,
enabling recognition of some more file formats.
- All active antivirus scanners are now called, the scan no longer stops
when the first one finds a virus. This is on the average a very small
price to pay (only payed in virus storms), but makes possible to see
which virus scanners found the virus and which did not, and to get their
timing in the logs. The list of virus scanners that noticed the virus
is now included in the notification message to the virus administrator.
A new macro %W is available for this purpose.
- More sensible logic when checking more than one file for those
daemonized av scanners that need to be handed individual files instead
of directory name.
- The 'Clam Antivirus-clamd' daemon is now supported, both via Unix
socket, as well as via inet socket (but Unix socket should be used,
due to lack of self-protection in this respect by clamd).
clamd is a fast and open-source anti-virus scanner
( http://clamav.elektrapro.com/ ), and I recommend it to be used in
combination with other commercial virus scanners as the last resort
scanner, to avoid bouncing mail in case primary scanner(s) stumble
across some unhandled situation, like corrupted or encrypted mail.
- New utility program 'amavisdconf' to check the values of configurable
variables set by /etc/amavisd.conf or defaulted (modeled after
Postfix utility 'postconf', but -e option is not yet supported).
Usage:
amavisdconf [-d | -n] [-c conf-file] [variable]
Options:
(none) show the value as will be used by amavisd;
-d show default value as in the absence of the config file;
-n only show variable if its value is different from the default;
-c conf-file ... use the specified file instead of /etc/amavisd.conf
If a variable is specified show only that variable, otherwise show all
configurable variables. (Note: the leading $, @ or % must be included
with the variable. Make sure to protect $ from shell evaluation,
e.g. by enclosing variable name in single quotes, or prepending
a backslash:
$ amavisdconf -d '$forward_method'
$ amavisdconf -n
This is the first attempt at such a utility, some details would need
to be polished (e.g. how to report code reference), but it is hoped
to be usable already in its present form.
- Two new subroutines (read_hash and read_text) are now available
for use from amavisd.conf file, with a primary intent to move larger
texts out of the amavisd.conf file:
* read_hash allows for lookup hashes (e.g. white/blacklist,
static %spam_lovers table, ...) to be read from a file during startup;
It takes quoted form of addresses, optionally enclosed in angle
brackets, and converts them into unquoted (raw) form, as needed
for the hash keys. Format: one address per line, everything from
'#' to end of line is a comment, leading and trailing whitespace
removed, empty (or comment-only) lines are ignored.
* read_text allows for reading more complex strings (e.g. multi-line,
or including special characters such as backslash) from a file.
Useful to override the default template text for notification messages.
See example usage in amavisd.conf.
- Three synonym (alias) subroutine names are available for use
in amavisd.conf to shorten the expressions:
new_RE same as Amavis::Lookup::RE->new
ask_daemon same as Amavis::AV::ask_daemon
sophos_savi same as Amavis::AV::sophos_savi
amavisd.conf file is changed to use the short names, but full
subroutine names are still valid.
- Provide protection mechanism for SMTP/LMTP server code to limit the
number of recipients in a mail transaction to some sane value
($smtpd_recipient_limit, default 1000). Even if the number is exceeded,
mail will still be normally delivered in more than one transaction
(provided your MTA is rfc2821 compliant).
- SMTP client code now correctly handles the situation where remote
SMTP server returns '452 Too many recipients'. When this happens the
mail forwarding or submission is split into several SMTP transactions,
and as long as the remote SMTP server is willing to accept at least
one recipient per transaction, mail delivery is assured.
- Auto white-listing (AWL) by SpamAssassin may be turned on by
setting $sa_auto_whitelist to true (contributed by Hamish Marson).
This only works as one global database, the per-recipient AWL
is presently not available.
- New configuration variables $hdrfrom_notify_sender, $hdrfrom_notify_admin,
and $hdrfrom_notify_spamadmin, to remove the overloading (double semantics)
from variables $mailfrom_notify_*. Now $mailfrom_notify_* control ONLY
envelope addresses, while $hdrfrom_notify_* only controls From header
in notifications. The default values for $hdrfrom_notify_* are chosen
to preserve compatibility.
- The spam headers in quarantined message now reflect the lowest
tag and kill levels when more than one recipient is given, instead
of reporting the levels for the first specified recipient.
- Tested with SpamAssassin 2.50 and with its new Bayesian classification
(used as a global database by amavisd-new). Given two sets of spam
and non-spam (=ham) messages, one can build a database by using new
SpamAssassin utilities sa-learn-spam and sa-learn-nonspam as user amavis,
and the database built this way will be used by amavisd-new/SpamAssassin.
- Initialize SpamAssassin as $daemon_user (no longer as root),
as a safety measure, and to ensure the files it creates are still
accessible after the UID change.
Incompatible changes to 20021116:
- Different parameters in @av_scanners list for daemonized antivirus
scanners (as described above).
- lookup_hash: an undefined hash value was previously treated as a
special case, resulting in value true. This is no longer the case,
which brings it in line with other lookup methods. It is not likely
anyone will be affected, as this practice was only documented but
not encouraged by any example.
---------------------------------------------------------------------------
Here is a brief description of patches that needed to be applied to
amavisd-new-20021116, and were available from the amavisd-new web page:
patch5 (2002-12-09):
- fix a simple test to decide if the anti-virus and anti-spam code is needed;
- use the /m regexp modifier by default when parsing av-scanner output
for virus names - seems like (?m) in qr is ignored (have we found
a Perl bug?);
- fix DSN to report original smtp response in case reporting MTA is remote
(thanks to Chris Hastie for his keen eye in understanding DSN);
- fix reporting quarantine mailbox name(s) in case of per-recip quarantine
or quarantine forwarding to MTA;
- macro %q (quarantine id) is now a list of quarantine mailbox names or
addresses (may have more than one entry in case of per-user quarantines);
- macro %v (av-scanner output) is now a list of scanner output lines,
to facilitate formatting of multi-line av-scanner output;
- fix spam admin notifications, which used the virus template instead of
the spam template;
- fix problem of undefined entity causing spam notification failure
in case other content checking (virus/banned) is not enabled;
- make SQL lookups (DBI access) optional - no DBI code is loaded if
@lookup_sql_dsn of empty;
- support SAVI-Perl module 0.10 or later (direct access to the Sophos library
engine) ( http://www.csupomona.edu/~henson/www/projects/SAVI-Perl/ );
- add macro %j that evaluates to message 'Subject:' header field;
- fix minor SMTP protocol inconsistency when multiple MAIL FROM
commands are issued by the client;
- more informative timing report when more than one av-scanner is used;
- make sendmail initial submission happy when null reverse-path is specified
(instead of the correct empty argument, which works but makes sendmail
issue a warning, we now use <> in arguments to specify null paths),
(thanks to Sebastian Hagedorn and Dale Perkel);
- replace log entry "spam from=<%s>, to=%s" with "SPAM, <%s> -> %s ...".
- dropped (redundant) historical %bypass_checks @bypass_checks_acl
(use bypass_virus_checks and %bypass_spam_checks instead);
patch4 (2002-11-21):
In amavisd:
- SQL lookup fix;
- finished per-recipient header edits and split-forwarding, where
one multi-recipient message may be forwarded as several messages
if different headers need to be inserted for different recipients
(not with milter);
- the check for banned file contents now looks for both the short
and the original full type classification as provide by 'file' utility
(thanks to Wolfgang Lumpp for the idea);
- #!/usr/local/bin/perl => #!/usr/bin/perl
- cosmetics
In amavisd.conf:
- suggested group name change $daemon_group 'sweep'=>'amavis';
so that non-Sophos users are not puzzled where 'sweep' group comes from;
- added path and fixed args for KasperskyLab AVPDaemonClient
(thanks to Mike Hall);
- added path for Trend Micro FileScanner vscan
(thanks to Eduardas Paulavicius);
- added 'use strict;' and '1;' at the end; prevent read_config from failing
if the config file does not return true (thanks to Vivek Khera);
- cosmetics, fixed typos in comments;
patch2 & patch3 (2002-11-18, 2002-11-19):
- fixes a problem with 'recip_done' object method (causing TEMPFAIL)
when $warnvirusrecip is enabled (thanks to Chris Hastie, Dale Perkel,
and Didi Rieder);
- pass null return path to sendmail in such a way to make it happy
(thanks to Sebastian Hagedorn and Dale Perkel);
- added a safety precaution to prevent somebody shooting himself in the
foot by specifying an empty $forward_method while using the SMTP in/out
setup (e.g. Postfix), which could send mail down the bit bucket;
patch1 (2002-11-17):
- fixes a problem in milter setup where per-recipient response codes
needed to be handled differently (the problem was causing a TEMPFAIL)
(thanks to Stephane Lentz and Didi Rieder for the initial milter testing
and problem reports);
---------------------------------------------------------------------------
November 16, 2002
amavisd-new-20021116 release notes
- provide a mechanism to load only required code sections
(anti-virus scanning, anti-spam scanning, SMTP/LMTP server module,
traditional amavis client & milter server module),
resulting in reduced memory usage and less installation dependencies;
- introduce the per-recipient status handling and make possible some
per-recipient functionality that was not possible before, such as
handling the per-recipient spam thresholds. This required a major
rewrite and cleanup of some sections, hopefully providing cleaner
mechanism for possible future advances in this area.
Consider for example a previously incorrectly handled situation
where there is more than one recipient, and some (but not all) of them
get a reject from the (outgoing) relay MTA - it is not possible for a
single SMTP response or status code to describe the situation;
- generate proper (non-)delivery status notifications (DSN), compliant with
rfc1892 (now rfc3462) and rfc1894 (now rfc3464). This form supersedes
the simple virus/spam sender notifications, but also covers the area of
genuine delivery failures, such as selective recipient rejects by the
outgoing relay MTA, which could previously lead to multiple deliveries;
NOTE: to avoid sender getting two non-delivery messages (one from
MTA, and another from amavisd), do not set $warnvirussender and
$final_virus_destiny=-1 (REJECT) at the same time (and equivalently
for spam settings);
- enhanced mail system status codes (rfc1893 (now rfc3463), and rfc2034)
are now included with all SMTP responses and DSN notifications;
- added checking for banned MIME types and names. If any mail part
matches, the whole mail is rejected, much like the way viruses
are handled. A list in object $banned_filename_re can be defined
to provide a list of Perl regular expressions to be matched against
each part's:
* Content-Type value (both declared and effective mime-type),
including the possible security risk content types
message/partial and message/external-body, as specified by rfc2046;
* declared (recommended) file names as specified by MIME subfields
Content-Disposition.filename and Content-Type.name, both in their
raw (encoded) form and in rfc2047-decoded form if applicable;
* file content type as guessed by 'file' utility and classified into
short type names such as .asc, .txt, .html, .doc, .jpg, .pdf,
.zip, .exe, ... - see subroutine determine_file_types().
This step is done only if $bypass_decode_parts is not set.
NOTE: by default the $final_banned_destiny is set to 1 (pass),
so detected banned file names only cause a header line to be added,
quarantining, and added address extension - but the mail is delivered
nevertheless. This default is set so that we can get initial experience
without being too obtrusive. Change as you prefer.
- besides SMTP/ESMTP protocol, the server now also accepts LMTP protocol
(rfc2033). This now explains why I had to implement PIPELINING,
ENHANCEDSTATUSCODES and 8BITMIME SMTP extensions, as these are required
by rfc2033.
One advantage of using LMTP to feed content filter is that LMTP uses
per-recipient status response, as opposed to 'one-size-fits-all'
SMTP status response, which require that the SMTP client (e.g. content
filter) is capable of generating DSN.
Another advantage is specific to the Postfix setup, as Postfix
LMTP client supports multiple transactions per session, saving on
connection teardown/reconnect for every message being checked.
To enable LMTP feed from Postfix, add 'max_use=10' to main.cf, and
replace last argument 'smtp' with 'lmtp' in the master.cf line:
'smtp-amavis unix - - n - 2 lmtp'. No changes are needed in amavisd-new,
both protocols can coexist, the distinction is based on HELO/EHLO
vs. LHLO command.
Btw, older versions of Postfix lmtp client inappropriately lowercased
the envelope addresses. This is fixed in Postfix Snapshot 1.1.11-20021015,
and in the regular Postfix release 1.2 (when available). Lowercasing
can be a problem for addresses where local part is case-sensitive,
although such setups are rare.
- improved per-recipient SMTP response code handling when sending mail
via SMTP. Previously one rejected recipient (by MTA) in a multi-recipient
message caused the whole message to me rejected. This is now correctly
handled with the new ability to send DSN;
- added command line option '-c config-file' so one can override the
default location of the configuration file (/etc/amavisd.conf);
- explicitly set PATH and HOME environment variables
(settings: $path and $helpers_home)
- added another form of lookups: Perl regular expression matching.
See README.lookups for details. Corresponding new variables in amavisd.conf
are: $virus_lovers_re, $spam_lovers_re, $bypass_virus_checks_re,
$bypass_spam_checks_re, $local_domains_re, as well as $banned_filename_re,
$viruses_that_fake_sender_re, $keep_decoded_original_re,
$whitelist_sender_re, $blacklist_sender_re.
- besides whitelist_sender* lookup tables, there is now also
a blacklist_sender* set of tables, which causes mail to be declared spam
and to skip remaining spam checks. See examples in amavisd.conf.
- provide a configurable lookup table $keep_decoded_original_re
of file types, for cases where unpacker is not very trustworthy.
The lookup key is what 'file' utility returned. If the part contents
match the lookup table, we keep both the original and the unpacked file
for virus scanner to check;
- provide a configurable list of regular expressions
$viruses_that_fake_sender_re, which recognizes viruses that usually
fake envelope sender address. Don't send sender notification if a match
is found. For syntax of the new regexp lookup tables see README.lookups;
- specifying per-recipient quarantine address or location is now possible
by setting $virus_quarantine_to and $spam_quarantine_to be a ref to a
hash lookup table. Thanks to Vivek Khera for the idea;
- no 'configure' and 'make' for the daemon; all configuration is done
via amavisd.conf configuration file at the daemon startup time;
Also the ./mta, ./av and ./notify configuration subdirectories
are now gone;
- provided a sample init shell script amavisd_init.sh (edit to will,
and move it to /etc/init.d/amavisd if you want). Based on the script
from Wil Cooley;
- one-shot debugging mechanism: if envelope sender matches @debug_sender_acl
lookup table, turn debugging fully up just for this one message and
cause temp file and directories not to be cleaned for this message.
This facilitates debugging a particular problem even in the presence
of regular traffic;
- cleaner debug log entries for multi-transaction SMTP/LMTP protocol sessions;
- cleaner log entries - avoid misleading and incomplete text
when quarantine is disabled; thanks to Michael Leone;
- avoid using (nonstandard) field width in the %e format specifier
when calling strftime, which lead to bad date syntax on some systems;
- local time zone offset is now automatically computed, no more '-0000'
in rfc2822 dates;
- put Message-ID field value in angle brackets as required by rfc2822
when generating notifications;
- add a In-Reply-To field to notifications when original Message-ID is known;
- don't bounce a virus (or spam) back to a mailing lists even if
$final_virus_destiny (or $final_spam_destiny) is set to REJECT;
a patch by Brian May from the Debian support crew, thanks;
- because some external module may play games with STDIN and STDOUT
(like SpamAssassin seems to do when local_tests_only=>0)
run the input protocol directly on the Net::Server's socket,
not on STDIN and STDOUT, which are aliases to the socket;
- a patch to amavis-milter.c by Didi Rieder to support REJECTing mail,
by instructing sendmail to return non-delivery notification to sender;
- a modified amavis.c helper program to make possible to invoke
local delivery agent from it, for those still using such a setup;
- a new macro %l is available for use in notification messages (via
'expand'); it evaluates to true (1) if the sender matches @local_domain,
and returns empty otherwise; by default it is now used in creating
a 'Subject:' line, inserting word 'LOCAL' before the sender name
when appropriate;
- new macros %D and %N expand to lists of recipients that got the mail
delivered (%D), or not-delivered (%N). Union of both sets gives %R,
i.e. a list of all recipients as specified in the envelope;
- dropped macro %a
- improved parsing of e-mail addresses according to rfc2821
(full address literals syntax, etc.);
- proper line folding for generated rfc2822 header fields;
- proper SMTP response wrapping (for very long responses such as
the ones that include a trouble report) according to rfc2821;
- allow for obsolete rfc822 syntax of permitting whitespace before
colon in header field name;
- added optional spam-sender nondelivery notifications, based on patch
from Lazslo E. Miranda (lazslo@dcc.ufmg.br) and Fernando F. Morais
(frota@cecom.ufmg.br);
- dynamically change process name (Perl variable $0) to reflect
the process state; suggested by Chip Paswater. Not all operating
systems make this process state visible by ps(1);
- determine location of external programs (or their absence) at
startup time; in amavisd.conf one may specify absolute path or just
rely on PATH. This mechanism is also used to determine absolute path
of the daemon itself, making reload (after HUP) more predictable;
- explicitly specify lock file for serialization to be used by the
Net::Server::PreForkSimple module, instead of relying on default
provided by POSIX::tmpnam. The default approach has a possibility
for a minor security problem, because the lockfile is created with
open()..., so it will follow symlinks. Observed, and a patch
provided, by Jarno Huuskonen;
- new defense against mail bombs: for the cumulative total of all
decoded mail parts we set max storage size. The formula is:
quota = max($MIN_EXPANSION_QUOTA,
$mail_size*$MIN_EXPANSION_FACTOR,
min($MAX_EXPANSION_QUOTA, $mail_size*$MAX_EXPANSION_FACTOR))
In plain words (later condition overrules previous ones):
allow MAX_EXPANSION_FACTOR times initial mail size,
but not more than MAX_EXPANSION_QUOTA,
but not less than MIN_EXPANSION_FACTOR times initial mail size,
but never less than MIN_EXPANSION_QUOTA
- if the permitted quota is exceeded (or the defense triggered by other
similar safeguards), the virus scanning is skipped to protect the
virus scanner from tripping over the mail bomb, a header field
is inserted:
X-Amavis-Hold: ... reason ...
and a log entry 'Placing on HOLD: reason' at level 0 is produced.
Also the temporary directory is preserved.
The 'X-Amavis-Hold: ...' header field can be used by your MTA to
put the message 'on hold' (freeze). If MTA is not set up to catch
messages with this tag (the default), they are passed normally to
recipients. This is likely to be the best action under the circumstances.
- tested with razor-agents-2.20 and SpamAssassin-2.43 (Razor2 is now called
by SpamAssassin, and no longer directly by amavisd-new). New configuration
variable $helpers_home, which defaults to $TEMPBASE. Thanks for hints
by Chris Hastie and John Stewart;
- to avoid CPU loop in SpamAssassin-2.43 and earlier, my patch needs
to applied to SA - see amavisd-new web page;
- to avoid taint problem in Razor 2.20 (if SpamAssassin-2.43 is configured
to call it), my patch needs to be applied to it - see amavisd-new web page;
- chroot available (but not well tested): $daemon_chroot_dir = '/var/amavis'
- provide a fail-over mechanism for SQL database connect - given a list
of SQL servers/databases, pick the first that is available.
Thanks to Ken McKittrick for making available the patch,
and to Ben Ransford <ben@terakeet.net> for writing it.
- remove existing virus-related and spam-related headers (some of them
optionally) if we'll be providing our own; suggested by Borut Mrak;
- avoid 'insecure dependency' in lookup_sql when calling DBI::execute
(thanks to ric* at mpc.com.br)
- fixed macro %H to provide original header lines, not the ones
stored internally by MIME::Entity, which may have been modified;
(thanks to Chris Hastie for noticing the problem);
- no longer inserts X-Razor-id header field;
INCOMPATIBLE CHANGES:
- when specifying boolean values to variables in the amavisd.conf file,
please specify 1 (or old style "yes") for true, and 0 or "" or undef
for false. The old style "no" yields true for Perl, and is only still
supported (converted to 0) for some traditional variables for
compatibility with amavisd(-snapshot);
- removed variables $sendmail_wrapper*, and changed the syntax for
specifying $forward_methor/$notify_method, with the intention to
do all mail sending settings at one place. See amavisd.conf for examples;
- variable $mailto is now deprecated (but still works as a fallback default
for compatibility with previous version). Use $virus_admin and $spam_admin
lookup tables instead, they also offer a mechanism to specify per-domain
administrator address;
- variable $warnadmin is no longer used. Use $virus_admin and $spam_admin
lookup tables instead. Not specifying administrator address
(e.g. leaving $virus_admin, $spam_admin and $mailto undefined)
turns off admin warnings.
- dropped variable $LOGDIR, the variable $LOGFILE now specifies the full path;
- renamed: $warnsender -> $warnvirussender, $warnrecip -> $warnvirusrecip
- dropped variables $enable_relay, $sendmail_cf_orig, $QMAILDIR;
they were never used in amavisd-new;
- dropped $MAX_ARCHIVE_NESTING, which is replaced by new storage limitations:
$MIN_EXPANSION_QUOTA, $MIN_EXPANSION_FACTOR
$MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR
- SQL database is now specified differently (@lookup_sql_dsn)
MTA-SPECIFIC: sendmail/milter
- $notify_method now specifies deferred delivery mode ('-odd') by default,
when submitting notifications to sendmail. This is to avoid calling milter
immediately during submission, which in turn calls amavisd-new, possibly
leading to a deadlock situation when the number of amavisd-new child
processes is small. Seems like this change is needed since Sendmail 8.12
or so.
The following recommendation is from mimedefang-filter man page:
| You MUST run a client-submission queue processor if you use
| Sendmail 8.12. We recommend executing this command as part of the
| Sendmail startup sequence:
| sendmail -Ac -qp1m
DECODERS-UNPACKERS:
- rewritten decode_parts() to allow for retaining source text if the
unpacker sw is considered unreliable. This more cleanly resolves
the problem reported on the amavis-user mailing list on 2002-06-06:
"Amavisd passing through VBS/VBSWG.gen@MM"
A nice side-product is that a directory search is avoided for
each nested unpacking step;
- refined do_unzip to control and limit the size of decompressed
members (among others it defends against the 42.zip-type bombs);
- rewritten external decompressors interface to the gzip/bzip2/compress
family. Instead of using a 'system' call, they are now called through
fh_copy, making possible to control and limit the size of the
decompressed contents on the fly, avoiding denial-of-service attacks.
Affected: b(un)zip2, g(un)zip, (un)compress;
- fh_copy now uses IO::Handle object to assure the forked process
gets reclaimed even in case of aborted contents extraction;
stdin gets redirected to /dev/null or to a specified input file
for the exec'd process;
- Convert::UUlib is called again (the amavisd-new-20020630 removed its
usage due to problems with improper decoding). This time the originals
are kept, so that virus checker sees both the original and the
attempted-decoded part;
- completely rewritten do_ascii and its usage of Convert::UUlib
to fix:
* coding error (bitwise op treated as logical op and improperly negated):
... if (!$uu->state || !FILE_OK || -z $newpart);
* it never reset the state, so if a successfully decoded ASCII file
contained another ASCII file, each decoding level would decode all
previously decoded parts again, plus add new ones at this level;
recursion would not stop until the hard limit, resulting in TEMPFAIL;
* it never checked nor reported errors that should have been detected
(I/O errors, out of memory, trouble accessing or creating files);
* more informative log entry;
- save MIME preambles and epilogues (if nonempty) as extra (pseudo)parts
to be scanned. This also mitigates the problem of syntactically-incorrect
MIME mail as produced by some user agents, which (rightfully) gets
treated as one long preamble by MIME-Tools, and previously went by
unchecked;
- supports unpacking arc archives using 'nomarch' (by Russell Marks,
http://rus.members.beeb.net/nomarch.html), (thanks to David D. Kilzer
for the initial code). Using 'nomatch' fixes a nasty habit of arc
which gratuitously appends a form-feed at the end of file when using
the 'p' (pipe) option, which might mislead a virus scanner.
Besides, 'nomarch' is GPL licensed;
- fixed a 'broken pipe' problem when calling unrar,
thanks to Ricardo Campos Passanezi and Rainer Link;
VIRUS SCANNERS:
- rewritten interfacing for most command-line virus scanners.
All settings for them is now done in amavisd.conf. New ones may be
added without having to modify the daemon source. More complex scanners
(e.g. daemonized scanners) still need to have a corresponding interface
routine in the daemon;
- rewritten Sophie and Trophie interfaces to be more resilient
to Sophie/Trophie daemon restarts during virus database reloads,
avoiding an unnecessary retry (TEMPFAIL) - thanks to Cor Bosman
for the suggestion and code, and to Dale Perkel for testing the
Trophie interface;
- make sophos sweep tolerant to encrypted attachments: if all files are
password protected, then the scanner failure is ignored and the message
is allowed to pass. Based on patch by Radu Greab;
- updated nai uvscan interface to recognize the result
'Found trojan or variant Exploit-CodeBase !!!', a patch by Anton Berezin;
- Clam Antivirus supported;
- fixed a problem in 'avp' scanner interface ($TEMPBASE not imported),
thanks to Joshua E Warchol;
- avpdc (KasperskyLab AVPDaemonClient): recognize additional exit codes,
based on patch from Christian Hammers;
- Panda new regexps, no TERM vt100 setting (thanks to Benjamin Zwittnig);
- Trend vscan exit code seems to be the number of infected parts.
Updated the test to reflect that, based on observation from Stephane Lentz;
- MksVir scanner interface returns error code 2 if viruses are removed
(if --clean option is passed) - a patch by Robert Litwiniec;
----------------------------------------------------------------------------
June 30, 2002
amavisd-new-20020630 release notes
Since it seems like several people are adapting amavisd-new
in details to their requirements, and certain improvement requests
have much in common, I'm making available the 20020630 release,
shortly before leaving on vacation. As this means my support
will be absent for the coming few weeks, and this release is
perhaps by few days premature, please consider it primarily a
development and new features release.
It is available at the usual location at:
http://www.ijs.si/software/amavisd/
(or ask Google about 'amavisd-new')
Having said that, it is still a fully functional and tested version,
and it is running in production at a couple of sites now. Also it is
my version of choice in view of dependability, having it running
at our site while I'm away. If you get into trouble, you can still go
back to amavisd-new-20020517 with which it is fully upwards compatible.
The main changes and features since amavisd-new-20020517 are:
- the code is thoroughly rearranged, interfaces cleaned, separated
into namespaces (packages), several sections generalized (e.g. lookups,
appending/editing header lines). The AV scanner and unpacking sections
are still mostly the same and compatible with amavisd, so whatever
improvements and new AV scanner support becomes available for amavisd,
applies almost without a change to amavisd-new;
- SMTP on the input side (used with Postfix and Exim) now talks ESMTP
(rfc2821) and not just rfc821, including some SMTP extensions:
command pipelining (rfc2920), message size declarations (rfc1870), and
8bit-MIME transport (rfc1652). The main reason for this was the change
in recent Postfix versions which can now do MIME transformations to support
7bit transports (implied by SMTP). To ensure the transparent 8bit path and
avoid message transformation by MTA, amavisd-new needs to declare it does
present an 8bit-clean path. A side benefit is a little speedup in passing
chunks of mailing list addresses due to pipelining support;
- split certain previous amavisd.conf settings (variables)
into several variables or lookup tables:
* sender address for notifications:
$mailfrom -> $mailfrom_notify_admin, $mailfrom_notify_sender,
$mailfrom_notify_recip, $mailfrom_notify_spamadmin
(these may also be empty to specify null reverse path <>,
which is most useful for sender notification);
* administrator address for notifications:
$mailto -> $virus_admin, $spam_admin (per-sender lookups)
* %bypass_checks -> %bypass_virus_checks, %bypass_spam_checks
- supports SQL database lookups via Perl module DBI (interface to
popular database types). Some examples are provided, if you need
other SQL lookups just modify the code by analogy. SQL lookups
are most useful for per-user settings of virus_lovers, bypass_virus_checks,
bypass_spam_checks, spam thresholds etc. when the user base is large
and subgroups can not be identified through their (sub)domains.
Another use is for dynamically changing settings without having
to restart amavisd-new;
- can optionally insert 'Received:' header if acting as a mail relay
(not with milter); does loop detection as required by rfc2821 section 6.2;
- notification messages now contain 'Date:' and 'Message-id:' headers;
- quarantined viruses contain X-AMaViS-Alert header line with names
of detected viruses;
- quarantined spam contains X-Spam-Status and X-Spam-Level header lines;
- optionally send spam admin notifications, which include the full
SpamAssassin spam report and message header;
- when started as root, changes UID and GID to $daemon_user, $daemon_group;
- to facilitate startup scripts and debugging, supports few simple
command line parameters:
amavisd ... standard run: changes uid/gid and daemonizes
amavisd start ... same thing
amavisd debug ... starts with full debug level, stays attached
amavisd reload ... finds amavisd master process and sends it a HUP
amavisd stop ... finds amavisd master process and sends it a TERM
- some more (minor) configurable options:
$daemonize, $pid_file,
$replace_existing_extension, $localpart_is_case_sensitive
- no longer calls Convert::UUlib for uuencoded, xxencoded, and binhex
attachments. The first two are handled by MIME::Parser, the remaining
are likely to be handled by anti-virus scanners, especially when some
virus would use such encoding. The use of Convert::UUlib is dropped
because it was causing recent problems with garbling virus so that it
could no longer be recognized by AV scanners, and because the underlying
library does not seem very dependable. See the thread 'Amavisd passing
through VBS/VBSWG.gen@MM' from the beginning of June 2002 in amavis-user
mailing list archives;
- quarantine files now include internal amavis id in the file name,
instead of the process number; previous naming scheme could stumble
across a name contention on a busy system;
- only a recommendation: logging via syslog is now preferred to direct
logging to a file. It serializes the logging, and avoids locking/unlocking
and reopening a log file by amavisd* for every log entry.
The syslog daemon does it more efficiently and reliably.
NOTE for Linux users: make sure you prefix the file name in syslogd.conf
with a '-' tag to disable fsync after every write; this is most necessary
for heavy logging such as from MTA and/or amavisd-new. The downside
is that you may lose the last few log entries in case of machine crash.
See man page of your syslogd for details.
- changed examples according to rfc2606 and recent complaints on the
postfix-users list.
CAVEATS:
- header rewriting is only available in SMTP-in/SMTP-out setup,
i.e. with Postfix and Exim, but not with sendmail milter setup;
- the SQL lookups are a very recent addition and not so well tested
as the rest of the program. Also since SQL lookups are supposed
to introduce per-user settings (e.g. spam thresholds), the code
still does not accommodate it, and spam thresholds of the first
recipient in a message affects the whole message;
- as Razor2 is still pretty much unstable, I left the spam_scan()
routine much as it was in the May version. Contributions welcome;
- as I ran out of my time for this release, I didn't prepare a
separate version without SpamAssassin and Razor 1.20 support,
so you will need to strip it out if you do not need it.
Similarly for the required DBI Perl module. I'm very sorry;
- the time stamp used in the 'Date:' header in notifications,
and in an optional 'Received:' header, does not contain true time zone
offset, but -0000 instead (standard meaning for unknown). I did not want
to include a fat Perl package for handling time zones. For now just edit
the subroutine rfc2822_timestamp() if you want to change that;
- amavisd.conf settings $daemon_user and $daemon_group are not
automatically set by ./configure options. Please set them manually.
I would like to thank many people on the amavis-user list, on the
postfix-users list, and in private conversations, who contributed
valuable ideas and improvements, and offered much appreciated
encouragements.
---------------------------------------------------------------------------
May 17, 2002
Available at:
http://www.ijs.si/software/amavisd/
amavisd-new-20020517 is primarily a response to popular demand for
Mail::SpamAssassin support. If amavisd-new-20020424 meets your needs,
there is no urgent need to upgrade. There are no incompatible changes
between these two versions, except some new (optional) amavisd.conf
variables, so you may keep old amavisd.conf file if you wish.
A new file README.exim is now provided, so that Exim 4.x is now a
supported and tested configuration, besides the usual Postfix and
sendmail/milter MTAs. Also included is a brand new qmail amavis client
by Lars Hecking (untested, please try it), plus his updated version of
amavis.c (no longer needed in recommended Postfix and Exim configurations,
and Sendmail milter and qmail configurations use a different client).
Several files are unchanged from the base amavisd CVS release 2002-05-13
and do not reflect the amavisd-new state: FAQ, HINTS, INSTALL, BUGS,
NEWS, TODO, ChangeLog, tests/, doc/. Please start with this file
README.amavisd-new-RELNOTES and follow it. More recent instructions
and last-minute changes are available from the web page.
Changes since amavisd-new-20020424:
- supports Mail::SpamAssassin and Vipul's Razor (1.20 required)
for spam checking (but not for modifying mail body - only add headers
and/or address extension, or reject/discard/quarantine spam).
NOTE: spam checks are off in the default amavisd setup.
Copy file ./amavis/amavisd.in.all to ./amavis/amavisd.in
before running ./configure and make, to use the
spam-check-enabled version!
Amavisd calls Mail::SpamAssassin directly, avoiding the need to set up
spamc/spamd or to chain filters. This is more efficient, one daemon
less to worry about, although maybe less flexible for some taste.
At the moment the SA per-user database is not used. Feel free to
experiment with it and let me know what you came up with.
The usual SA config files are observed, but remote tests are disabled
by default (Razor is handled directly, RBL lookups can/should be
performed early by MTA which knows what IP address mail came from,
and that information is (mostly) lost afterwards). If you feel otherwise,
change the hard-wired settings in the call:
$spamassasin_obj = Mail::SpamAssassin->new(
{ dont_copy_prefs => 1, local_tests_only => 1 } )
(also the Razor score contribution is hard-wired,
modify: '$razor_spam_found ? 3 : 0' to will).
Amavisd-new handles Razor checks directly to be able to exercise
more control over it than would otherwise be available through SA:
timing, signatures are needed for insertion into header, skips one-liner
body checks which are common Razor false-positives. To avoid SA calling
Razor again, either keep the default setting 'local_tests_only=>1',
or set 'score RAZOR_CHECK 0' in the SA configuration.
Added are 'whitelist_sender' hash and ACL lookups (see README.lookups),
which approves spam from specified SMTP originator addresses - SpamAssassin
can only check and whitelist rfc822 headers, not the envelope addresses,
and I see no way of passing envelope addresses to it - which is a pity,
as important information is lost.
SpamAssassin checks are computationally quite expensive compared to other
amavisd activities, and the time needed for SA check goes up significantly
with the message size. SA check are skipped (but not the Razor check)
if mail size exceeds 64k. My analysis shows that presently less than 1%
of spam exceeds 64k characters, and this is probably well below the
false-negative SA rate, so it is not worth wasting time to check large mail.
remaining changes:
- new file README.exim (thanks to Jochen Erwied, Patrice Fournier
and Igor D'Astolfo);
- updated README.postfix to describe how to avoid running header_checks,
body_checks and dns_lookups in Postfix twice;
- put back file README.customize, which was omitted from amavisd-new-20020224
by mistake (but available with previous version and on the web);
- added introductory paragraph to README, the rest still needs to be updated;
- added missing last-step check for '.' in hash lookups to match examples
and make it more useful;
- added timeouts to certain tasks to make it better suitable for unattended
operation (less, but still subject to certain DoS attacks, similar to
the official amavisd). The main reason for adding timeouts is that we
don't want spam checking to slow down amavisd operation too much: if it
takes too long, just skip it (assume not spam) and move on;
- change unmangle_sender() to believe sender address for Klez viruses;
this is sometimes wrong, but seems like people prefer to sacrifice
few false accusations in favour of some warranted sender notifications.
Feel free to hack this routine at will (and publish good ideas),
it is intended to be modified;
- use Perl module Errno instead of errno.ph to avoid using broken
file errno.ph on some popular platforms;
- make possible to have per-sender-domain administrator e-mail address
for admin notifications (hash lookup %mailto, sub warn_admin() )
- relax temporary file/dir protection to allow them to be readable
by the group. This makes possible for anti-virus (daemon) checker
to run under a different user (but in the same group). Doing so should
be safer as it makes impossible for virus-checking daemon to clobber
files. It is only supported in the all-SMTP configuration. To use
it with traditional amavis clients (e.g. milter), you will have to
modify their sources to change umask and mods for file/dir they create;
- avoid changing sender address <> to <""> under certain circumstances
(not strictly wrong, but still a bad idea);
- replaced header 'X-Razor-Warning:' with SA-compatible 'X-Spam-Status:' ;
the 'X-Razor-id:' is still provided to facilitate user spam reporting;
- added '--' between options and argument when calling $sendmail_wrapper
to be triply sure we avoid problems with some mailers (note that
$sendmail_wrapper is only still needed in the sendmail setup,
Postfix and Exim do not need it in the recommended configuration);
- tidy the %local_delivery_aliases mechanism for local delivery / quarantine;
- allow MIME::Parser to decode uuencoded parts, if it feels
it can (and should) do it;
- write warnings from MIME::Parser to the log (at log level 1 or higher),
instead of discarding them.
The package is available at the usual location at:
http://www.ijs.si/software/amavisd/
where also the most up-to-date version of FAQ
and certain other fresh documentation files live.
---------------------------------------------------------------------------
April 24, 2002
amavisd-new-20020424 is primarily a maintenance release
to summarize one week's worth of experience with amavisd-new-20020418
and to implement some good ideas from the amavis-user mailing list.
It also brings one or two new features.
It is available at:
http://www.ijs.si/software/amavisd/
or more specifically, at:
http://www.ijs.si/software/amavisd/amavisd-new-20020424.tar.gz
Changes since amavisd-new-20020418:
- removed Perl 'my' declaration from configurable variables which prevented
them from being changed in amavisd.conf (thanks to Sebastian Hagedorn
and Wouter de Jong for reporting)
- introduced child timeouts to prevent bad amavis client from monopolizing
a child forever (thanks to Sebastian Hagedorn for reporting the problem)
- supported and documented (in README.postfix) configuration where multiple
remote or local SMTP-in/SMTP-out MTAs (e.g. Postfix) can use the same
amavisd server, by making it deliver checked mail back to the same
IP address it came from (see variable $relayhost_is_client in amavisd.conf)
(thanks to Wouter de Jong for the splendid idea which fitted naturally
into the overall scheme)
- in certain log messages include the SMTP-in and SMTP-out MTA IP address
- new access list checking for IP addresses - used to limit SMTP access
to authorized MTAs only: @inet_acl; access control is now enabled by default
- slightly more sophisticated hash-based access list lookups, modeled after
Postfix map lookups. The sequence now goes through the following steps:
- hash lookup for user+foo@do.ma.in
- hash lookup for user@do.ma.in (only if $recipient_delimiter is nonempty)
- hash lookup for do.ma.in
- hash lookup for .do.ma.in
- hash lookup for .ma.in
- hash lookup for .in
- hash lookup user+foo@
- hash lookup user@ (only if $recipient_delimiter is nonempty)
- updated README.lookups
- cleaner quarantine code; new variable $mailfrom_quarantine allows
to choose either the original envelope sender, or admin-specified
fixed sender address; include a special per-user quarantine example
(look for 'trouble-user-quarantine' in the amavisd)
- fixed problem with localized system error messages - use numeric errno
instead of strings in Sophie and Trophie clients (thanks to Igor D'Astolfo
for reporting the problem with Italianized version of Linux and for a good
suggestion)
- successfully tested with Exim
(thanks to Igor D'Astolfo for testing and for reporting
the <<doubly-quoted>> sender (but not recipient!) problem)
- wrong variable used in the spam section ($final_virus_destiny
instead of $final_spam_destiny) (thanks to Wayne Smith for reporting
and for testing Vipul's Razor 'plugin' patch)
- changed pattern match in the ./av/oav to support new version of
OpenAntiVirus ScannerDaemon (thanks to Rainer Link)
- changed 'configure' to make it recognize Sophie 1.33rc1
(thanks to Igor D'Astolfo and Lars Hecking)
- include acinclude.m4/acx_pthread.m4 macro
(thanks to Rainer Link)
- updated README.postfix: includes instructions on how to avoid body_checks
and header_checks for reinserted mail (contributed by Wayne Smith,
works nicely, requires Postfix version 1.1.7-20020331 or later)
- clarified comments in amavisd.conf
- optionally keep existing X-Virus-Scanned: header lines, or remove them
before adding our own header line - see $remove_existing_x_scanned_headers
(requested by Darryl Harvey)
- avoid historical (misleading) parameter name $localhost_ip;
use $relayhost and $relayhost_port instead, but take old variables into
account for backwards compatibility with existing amavisd.conf files
- Here is an overall picture (sequence of events)
of how pieces fit together:
bypass_checks? ==> PASS
no viruses? ==> PASS
log virus if $log_templ is nonempty
quarantine if $virus_quarantine_to is nonempty
notify admin if $warnadmin
notify sender if $warnsender
notify recips if $warnrecip
final_destiny==pass? ==> PASS
virus_lovers? ==> PASS
DISCARD or REJECT (depending on final_*_destiny)
---------------------------------------------------------------------------
April 18, 2002
This it to announce the second release of amavisd-new-20020418,
available at:
http://www.ijs.si/software/amavisd-new-20020418.tar.gz
It is a version of amavisd (a daemonized AMaViS, which is an interface
between MTA and virus scanners), based on amavisd CVS from today (20020418)
(same configuration, amavis clients in C, instructions, AV client code),
while also being a successor of the initial release of amavisd-new-20020329,
which is a performance-enhanced pre-forked Net::Server -based amavisd
with SMTP-in/SMTP-out capability, written in Perl.
(I tried to make this release based on amavisd-snapshot-20020300, but failed,
as that version still has $errval semantics bug (the bitwise-operations
problem was discussed on the AMaViS-user list some time ago).
This was fixed in the CVS version and most AV clients were changed then.
As the CVS version contains AV client code for new AV scanners, this makes
it incompatible with config stuff from amavisd-snapshot-20020300.
If you have problems with ./configure or make, these problems would be
common to both versions. Also the unpackers and decompressors code is mostly
the same for all recent amavisd versions, so any problems in this arena
(like DoS mishandling) are most likely to be common for both the official
amavisd and the amavisd-new version.
Compared to the first release of amavisd-new-20020329, the second release
brings further significant performance improvements especially in the
SMTP-in/SMTP-out configuration, e.g. with Postfix, but also brings
some interesting new features and new configuration possibilities.
No important bug fixes were needed, so upgrade at your leisure if you are
running my initial version. The Postfix users would perhaps want to rush
a bit though, to put new performance improvements into use.
The summary of changes since the initial version:
- significant SMTP-in speedups (25% with fast AV scanner), file reuse
- pass reject reason to MTA on the input side
- more informative MTA log entries in the SMTP-in/SMTP-out setup
- amavis internal id (am_id) in log entries and passed to MTA in SMTP response
- ISP features: specify subgroups of users who want to receive viruses
- address extensions: e.g. user@domain -> user+virus@domain if virus detected
- can specify final_virus_destiny: reject, discard, pass
- quarantine new options: save to individual file, save to mailbox, pass to MTA
- new headers in quarantined viruses preserve envelope addresses
and quarantine id (similar to the suggestion from Furio Ercolessi - see code)
- detailed timing breakdown report for each passed message
- anti-spam hooks and examples, example patch to integrate Vipul's Razor client
- body cache now always enabled - heavy speedup for mailing list bursts
- Sophie 1.33-ready
- rewritten Trophie client, based on new Sophie client code
- rewritten README.postfix, describing new setup possibilities
- new file README.lookups (to be used with virus_lovers and bypass_checks)
- new file README.customize (same as in the initial version)
- new file README.performance (unfinished)
- new amavisd.conf options, documented there
- code heavily commented, cleaned, generalized again
- does not accuse innocent users of sending viruses if we are suspicious
of sender address (see FAQ below) (same as in the initial version)
CAVEATS:
- no test mode;
- only Postfix and sendmail milter are fully supported and tested;
to integrate with other mailers one would need some understanding of their
operation to set up properly; contributions are welcome;
(P.S. note: this is probably a non-issue, related to the file system problem:
All amavisd versions seem to share one still unresolved problem, probably
with amavis-milter.c client, its use of libmilter, or perhaps even in the
libmilter code itself - see recent thread on the AMaViS-user list
(subject: Leftover email.txt files with amavisd 'standard' also).
The problem is most pronounced in burst of heavy traffic.
)
FAQ:
- Net::Server 0.82 triggers a Perl 5.005 bug (the problem is obvious:
you get syntax errors). Either upgrade to Net::Server 0.83, or
upgrade your Perl - 5.6.x should be ok.
- if you intend to play with customized notifications, it is wise
to remove the '-t' option from $sendmail_wrapper_args in amavisd.conf.
That way you are free to screw up notification mail headers any way you want,
and the message would still be delivered to the correct recipients.
Removing '-t' is now the recommended setting, but both variants should work.
If you use SMTP-out method for notifications (e.g. Postfix recommended
and default setting), this does not affect you.
- if you see virus notifications claiming the virus originator is <?>
or <?@some.domain> and sender notifications are not sent, this is not a bug,
but a feature - see comments at the subroutine unmangle_sender().
The original idea comes from Furio Ercolessi: as some viruses tend to use
forged or corrupted sender or 'From:' addresses, we try to determine
the true virus sender, and if we can not do that, we avoid accusing
innocent users of sending viruses.
- if you kill or HUP amavisd, temporary directories may be left undeleted;
this is normal and mails are not lost;
- if amavisd does not restart after receiving HUP, a possible reason may be
that amavisd can not be found in the path as set in the $ENV{PATH} variable
(near the beginning of amavisd program). Another reason may be a syntax
error if you changed the amavisd.conf file. Try to start it manually:
$ su vscan -c amavisd
If that does not make you wiser, set $DEBUG = "yes" and retry.
This is also the recommended first-time start method.
- after changing $inet_socket_bind in amavisd.conf, you must stop amavisd
and start it anew. The HUP method causes amavisd to stumble over its feet.
Below is my announcement notice for the initial version of amavisd-new,
just slightly edited to remove some mistakes.
---------------------------------------------------------------------------
Date: Sat, 30 Mar 2002 04:13:25 +0100
From: Mark Martinec <Mark.Martinec@ijs.si>
Subject: [AMaViS-user] ANNOUNCE: new amavisd - leaner and meaner
To: amavis-user@lists.sourceforge.net
Message-id: <01KFYI1DJ2O200AMKT@CATHY.IJS.SI>
I would like to announce an updated version of amavisd,
based on recent CVS code of amavisd (which is not far
from the February amavisd snapshot). It is a result of my
three weeks work on the code, caused by our needs for:
- better reliability;
- higher throughput (less overhead);
- versatility (e.g. separating amavis and MTA hosts, load sharing),
Available at:
http://www.ijs.si/software/amavisd-new-20020329.tar.gz
This is now finally a version which I can recommend to friends :)
TODO: better DoS handling in unpackers.
Main features - in brief:
- pre-forked reusable children - saving on process creations;
- persistent connections to certain AV scanners, e.g. Sophie, saving on forks;
- both SMTP and pipe (sendmail wrapper) interfaces independently available
and configurable on all three sides: input, output, notifications;
- sendmail Milter interface supported and tested; Postfix supported and
thoroughly tested (Exim untested; classical sendmail untested, no qmail);
- customizable notification messages;
- compatibility with existing configurations;
- cleanups, generalizations, speedups, fixes, better code documentation;
- HUP signal causes restart with new configuration;
- ISP feature: certain recipients may be allowed to receive viruses
(with alert header line added (not with milter), notifications are
still generated);
- anti-spam hooks, caching-ready, Vipul's Razor interface in a form
of a patch included - should make integration with SpamAssassin easier.
Install:
Unpack the tar over the checked-out CVS version of amavisd branch
(or February snapshot). It overlays some files, the rest is unchanged.
For milter interface make sure you use amavis-milter.c from CVS,
as the snapshot version contains a bug which can cause message loss
if amavisd dies.
Details:
- child and socket handling is now delegated to Perl module Net::Server,
which gives us pre-forked children which are able to do more than one
mail-check during their lifetime, saving on process creations
and giving better response time;
- Net::Server controls number of children, does signal handling,
takes care of dead children, handles listening on multiple sockets
(both Unix and TCP/UDP), delegating tasks and synchronizing 'accept's;
- as a consequence, certain AV-scanner interfaces (most notably Sophie,
and soon to follow Trophie) can keep persistent connections to the
AV checking daemon, saving on AV scanner process creations
(forks on accept), and socket setups/teardowns;
- on the input side: both SMTP (TCP) and traditional amavis client
protocol (Unix socket) are now accepted - even both at the same time
(by default) to ease transition to SMTP interface;
- new input-side SMTP interface (SMTP server) is easier to set up
(no need to worry about file ownerships, UIDs and GIDs),
and more versatile (e.g. SMTP responses carry more information
that sysexits.h-based status codes, SMTP can talk to remote host);
It is based on Perl native I/O and it is quite fast - no OO overhead,
_not_ SMTP::Server-based (which I do not consider production quality);
- a nice by-product of SMTP input interface is the extra information
available in MTA logs, e.g. Postfix log:
postfix/smtp[7656]: DA7B147FA6:
to=<user@domain>, relay=localhost[127.0.0.1], delay=5,
status=sent (250 Ok, discarded - VIRUS: EICAR-AV-Test)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- notifications messages can be delivered either via SMTP or
as traditionally via pipe to sendmail wrapper / MTA pickup.
This means that in the Postfix setup for example, one needs
to worry only about one output interface (SMTP or pipe).
This also saves unnecessary work of checking the just-generated
notification messages for viruses;
- the output pipe interface is now more careful with status
checking and forking;
- output SMTP interface error handling more compact;
- sender notifications can be sent to more than one contact address
(not used at the moment, but the mechanism is available);
- sender address unmangling patch incorporated (but I'm willing
to take it out if considered inappropriate for the base distribution);
- see file ./amavis/mta/postfix_init for comments describing
how to select SMTP- or pipe-based output interface
(for re-injection and notifications);
- customizable notification messages and log entry text
(see README.customize);
- clean notion of when mail addresses are in their quoted and
when in unquoted form (RFC2821). All internal handling uses unquoted
form, addresses get quoted as required by the output interface,
and quoting gets stripped away as required by each input interface;
- besides traditional choices where a mail can be forwarded/accepted,
discarded, or temporary failure/retry indicated, there is now
a fourth choice: REJECT. Depending on the MTA this requires a
message rejection to be done by the input side MTA itself.
This comes handy in cases when amavisd accepts a non-infected mail,
but outgoing MTA does not want to take it back for final delivery,
e.g. in case of some policy violation. Traditionally amavisd
would indicate temporary failure on its input side, causing the
message to be retried and re-scanned over and over again,
without having a chance of ever being accepted;
- all existing virus scanners are still there (one little change
in each: replaced a call to do_virus with return 1);
Sophie client rewritten to take advantage of persistent connections;
- decoding sections are mostly unchanged. This area needs more work
in the future;
- includes exit status codes from sysexits.ph
instead or having them hard-wired;
- anti-spam code easier to integrate into amavisd due to some code
rearrangements; a patch to integrate Vipul's Razor is included
as an example, but it may be better to tie amavisd with SpamAssassin.
Anti-spam code will NOT be integrated into amavisd, but the least
we can do is making it easier for people to add their own code;
- cache-ready (example in the included Razor patch). This works by
calculating a message digest (hash, signature) of the message body
and keeping it in storage for a short while, e.g. for the lifetime
of a child process (10 consecutive requests by default).
If another message with the same body content arrives in the near future
we can skip a virus check. This comes handy where mailing list traffic
is frequent, especially if we have to deal with poorly done mailing
list managers or heavy spam traffic;
To support this concept the functions of MIME decoding and
unpacking of archives are now separated.
- MIME parsing is now supplied with our own Filer subclass.
This was necessary to avoid MIME parser complaints when it tried
to reconstruct file name extensions from file names in strange
character sets. Its work was completely unnecessary and harmful,
since we want to supply our own file names and do not care
for file name extensions.
- as a consequence of our own Filer subclass, we now avoid the first
(or the only) directory traversal (reading) in the first decoding pass;
Not much, but every little bit counts.
- when using sendmail wrapper it is no longer necessary to supply
the sendmail -t option. If -t is not specified, sendmail wrapper
receives addresses via command arguments (exec, no shell),
which is more reliable than having to parse mail headers - which are
now more error-prone due to user-customizable notifications.
The -t option is still supported though, but not recommended.
- do_unzip no longer complains with multi-line backtrace when it dies;
just a single message is issued, like with other decoders;
- HUP signal causes restart and re-reading of config file;
- many new comments, code unifications, supplied some missing
error checks, code generalizations;
MTA support:
- thoroughly tested with Postfix in all combinations of input,
output, and notifications interfaces; in production use;
- big thanks to Sebastian Hagedorn who helped to test the milter
interface on his Solaris!
- Exim and traditional sendmail interface untested, but should
work without much work.
- Qmail interface is still missing, like in current CVS amavisd.
I believe the amavis client for qmail can be written as
a few-dozen line Perl program. Anybody want to try?
A quick cookbook on how to set up Postfix / amavisd interface
using SMTP on both amavisd input and output side (including notifications).
Amavisd by default now accepts both Unix socket and SMTP on the loopback
interface, so the transition is easier: first install new amavisd,
then at some other time change Postfix configuration (if desired) to:
master.cf:
# MTA -> amavisd
smtp-amavisd unix - - n - 2 smtp
# amavisd -> MTA
localhost:10025 inet n - n - - smtpd -o content_filter=
main.cf:
# choose transport to amavisd
content_filter = smtp-amavisd:localhost:10024
[see new file README.postfix for details]
The amavisd can now easily be located on a different host than MTA,
also Postfix load-balancing transport methods can be used (e.g. multiple
MX records). Although amavisd now talks SMTP and incorporates some rudimentary
defenses against malicious SMTP clients, do not expose its SMTP server
directly to the world - always front-end it with MTA. By default it
binds to the local interface only, but other access restrictions are
also available.
Happy amavising! Experiences and comments are most welcome.