DATABASE_README.html [plain text]
<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Postfix Lookup Table Overview</title>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix
Lookup Table Overview</h1>
<hr>
<h2>Overview </h2>
This document covers the following topics:
<ul>
<li><a href="#intro">The Postfix lookup table model</a>
<li><a href="#lists">Postfix lists versus tables </a>
<li><a href="#preparing">Preparing Postfix for LDAP or SQL lookups</a>
<li><a href="#detect">Maintaining Postfix lookup table files</a>
<li><a href="#safe_db">Updating Berkeley DB files safely</a>
<li><a href="#types">Postfix lookup table types</a>
</ul>
<h2><a name="intro">The Postfix lookup table model</a></h2>
<p> Postfix uses lookup tables to store and look up information
for access control, address rewriting and even for content filtering.
All Postfix lookup tables are specified as "<a href="DATABASE_README.html">type:table</a>", where
"type" is one of the database types described under "<a
href="#types">Postfix lookup table types</a>" at the end of this
document, and where "table" is the lookup table name. The Postfix
documentation uses the terms "database" and "lookup table" for the
same thing. </p>
<p> Examples of lookup tables that appear often in the Postfix
documentation: </p>
<blockquote>
<pre>
/etc/postfix/<a href="postconf.5.html">main.cf</a>:
<a href="postconf.5.html#alias_maps">alias_maps</a> = hash:/etc/postfix/aliases (local aliasing)
<a href="postconf.5.html#header_checks">header_checks</a> = <a href="regexp_table.5.html">regexp</a>:/etc/postfix/header_checks (content filtering)
<a href="postconf.5.html#transport_maps">transport_maps</a> = hash:/etc/postfix/transport (routing table)
<a href="postconf.5.html#virtual_alias_maps">virtual_alias_maps</a> = hash:/etc/postfix/virtual (address rewriting)
</pre>
</blockquote>
<p> All Postfix lookup tables store information as (key, value)
pairs. This interface may seem simplistic at first, but it turns
out to be very powerful. The (key, value) query interface completely
hides the complexities of LDAP or SQL from Postfix. This is a good
example of connecting complex systems with simple interfaces. </p>
<p> Benefits of the Postfix (key, value) query interface:</p>
<ul>
<li> You can implement Postfix lookup tables first with local
Berkeley DB files and then switch to LDAP or MySQL without any
impact on the Postfix configuration itself, as described under "<a
href="#preparing">Preparing Postfix for LDAP or SQL lookups</a>"
below.
<li> You can use Berkeley DB files with fixed lookup strings for
simple address rewriting operations and you can use regular expression
tables for the more complicated work. In other words, you don't
have to put everything into the same table.
</ul>
<h2><a name="lists">Postfix lists versus tables </a></h2>
<p> Most Postfix lookup tables are used to look up information.
Examples are address rewriting (the lookup string is the old address,
and the result is the new address) or access control (the lookup
string is the client, sender or recipient, and the result is an
action such as "reject"). </p>
<p> With some tables, however, Postfix needs to know only if the
lookup key exists. The lookup result itself is not used. Examples
are the <a href="postconf.5.html#local_recipient_maps">local_recipient_maps</a> that determine what local recipients
Postfix accepts in mail from the network, the <a href="postconf.5.html#mydestination">mydestination</a> parameter
that specifies what domains Postfix delivers locally, or the
<a href="postconf.5.html#mynetworks">mynetworks</a> parameter that specifies the IP addresses of trusted
clients or client networks. Technically, these are lists, not
tables. Despite the difference, Postfix lists are described here
because they use the same underlying infrastructure as Postfix
lookup tables. </p>
<h2><a name="preparing">Preparing Postfix for LDAP or SQL lookups</a>
</h2>
<p> LDAP and SQL are complex systems. Trying to set up both Postfix
and LDAP or SQL at the same time is definitely not a good idea.
You can save yourself a lot of time by implementing Postfix first
with local files such as Berkeley DB. Local files have few surprises,
and are easy to debug with the <a href="postmap.1.html">postmap(1)</a> command: </p>
<blockquote>
<pre>
% <b>postmap -q info@example.com hash:/etc/postfix/virtual </b>
</pre>
</blockquote>
<p> Once you have local files working properly you can follow the
instructions in <a href="ldap_table.5.html">ldap_table(5)</a>, <a href="mysql_table.5.html">mysql_table(5)</a>, <a href="pgsql_table.5.html">pgsql_table(5)</a>
or <a href="sqlite_table.5.html">sqlite_table(5)</a>
and replace local file lookups with LDAP or SQL lookups. When you
do this, you should use the <a href="postmap.1.html">postmap(1)</a> command again, to verify
that database lookups still produce the exact same results as local
file lookup: </p>
<blockquote>
<pre>
% <b>postmap -q info@example.com <a href="ldap_table.5.html">ldap</a>:/etc/postfix/virtual.cf </b>
</pre>
</blockquote>
<p> Be sure to exercise all the partial address or parent domain
queries that are documented under "table search order" in the
relevant manual page: <a href="access.5.html">access(5)</a>, <a href="canonical.5.html">canonical(5)</a>, <a href="virtual.5.html">virtual(5)</a>,
<a href="transport.5.html">transport(5)</a>, or under the relevant configuration parameter:
<a href="postconf.5.html#mynetworks">mynetworks</a>, <a href="postconf.5.html#relay_domains">relay_domains</a>, <a href="postconf.5.html#parent_domain_matches_subdomains">parent_domain_matches_subdomains</a>. </p>
<h2><a name="detect">Maintaining Postfix lookup table files</a></h2>
<p> When you make changes to a database while the mail system is
running, it would be desirable if Postfix avoids reading information
while that information is being changed. It would also be nice if
you can change a database without having to execute "postfix reload",
in order to force Postfix to use the new information. Each time
you do "postfix reload" Postfix loses a lot of performance.
</p>
<ul>
<li> <p> If you change a network database such as LDAP, NIS or
SQL, there is no need to execute "postfix reload". The LDAP, NIS
or SQL server takes care of read/write access conflicts and gives
the new data to Postfix once that data is available. </p>
<li> <p> If you change a <a href="regexp_table.5.html">regexp</a>: or <a href="pcre_table.5.html">pcre</a>: file then Postfix may or
may not pick up the file changes immediately. This is because a
Postfix process reads the entire file into memory once and never
examines the file again. </p>
<ul>
<li> <p> If the file is used by a short-running process such as
<a href="smtpd.8.html">smtpd(8)</a>, <a href="cleanup.8.html">cleanup(8)</a> or <a href="local.8.html">local(8)</a>, there is no need to execute
"postfix reload" after making a change. </p>
<li> <p> If the file is being used by a long-running process such
as <a href="trivial-rewrite.8.html">trivial-rewrite(8)</a> on a busy server it may be necessary to
execute "postfix reload". </p>
</ul>
<li> <p> If you change a local file based database such as DBM or
Berkeley DB, there is no need to execute "postfix reload". Postfix
uses file locking to avoid read/write access conflicts, and whenever
a Postfix daemon process notices that a file has changed it will
terminate before handling the next client request, so that a new
process can initialize with the new database. </p>
</ul>
<h2><a name="safe_db">Updating Berkeley DB files safely</a></h2>
<p> Although Postfix uses file locking to avoid access conflicts
while updating Berkeley DB or other local database files, you still
have a problem when the update fails because the disk is full or
because something else happens. This is because commands such as
<a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> overwrite existing files. If the update
fails in the middle then you have no usable database, and Postfix
will stop working. This is not an issue with the CDB database type
available with Postfix 2.2 and later: <a href="CDB_README.html">CDB</a>
creates a new file, and renames the file upon successful completion.
</p>
<p> With multi-file databases such as DBM, there is no simple
solution. With Berkeley DB and other "one file" databases, it is
possible to add some extra robustness by using "mv" to REPLACE an
existing database file instead of overwriting it: </p>
<blockquote>
<pre>
# <b>postmap access.in && mv access.in.db access.db</b>
</pre>
</blockquote>
<p> This converts the input file "access.in" into the output file
"access.in.db", and replaces the file "access.db" only when the
<a href="postmap.1.html">postmap(1)</a> command was successful. Of course typing such commands
becomes boring quickly, and this is why people use "make" instead,
as shown below. User input is shown in bold font. </p>
<blockquote>
<pre>
# <b>cat Makefile</b>
all: aliases.db access.db virtual.db ...etcetera...
# Note 1: commands are specified after a TAB character.
# Note 2: use <a href="postalias.1.html">postalias(1)</a> for local aliases, <a href="postmap.1.html">postmap(1)</a> for the rest.
aliases.db: aliases.in
postalias aliases.in
mv aliases.in.db aliases.db
access.db: access.in
postmap access.in
mv access.in.db access.db
virtual.db: virtual.in
postmap virtual.in
mv virtual.in.db virtual.db
...etcetera...
# <b>vi access.in</b>
...editing session not shown...
# <b>make</b>
postmap access.in
mv access.in.db access.db
#
</pre>
</blockquote>
<p> The "make" command updates only the files that have changed.
In case of error, the "make" command will stop and will not invoke
the "mv" command, so that Postfix will keep using the existing
database file as if nothing happened. </p>
<h2><a name="types">Postfix lookup table types</a> </h2>
<p> To find out what database types your Postfix system supports,
use the "<b>postconf -m</b>" command. Here is a list of database types
that are often supported: </p>
<blockquote>
<dl>
<dt> <b>btree</b> </dt>
<dd> A sorted, balanced tree structure. This is available only on
systems with support for Berkeley DB databases. Database files are
created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. The lookup
table name as used in "btree:table" is the database file name
without the ".db" suffix. </dd>
<dt> <b>cdb</b> </dt>
<dd> A read-optimized structure with no support for incremental updates.
Database files are created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command.
The lookup table name as used in "<a href="CDB_README.html">cdb</a>:table" is the database file name
without the ".cdb" suffix. This feature is available with Postfix 2.2
and later. </dd>
<dt> <b>cidr</b> </dt>
<dd> A table that associates values with Classless Inter-Domain
Routing (CIDR) patterns. The table format is described in <a href="cidr_table.5.html">cidr_table(5)</a>.
</dd>
<dt> <b>dbm</b> </dt>
<dd> An indexed file type based on hashing. This is available only
on systems with support for DBM databases. Database files are
created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. The lookup
table name as used in "dbm:table" is the database file name without
the ".dir" or ".pag" suffix. </dd>
<dt> <b>environ</b> </dt>
<dd> The UNIX process environment array. The lookup key is the
variable name. The lookup table name in "environ:table" is ignored.
</dd>
<dt> <b>hash</b> </dt>
<dd> An indexed file type based on hashing. This is available only
on systems with support for Berkeley DB databases. Database files are
created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. The database
name as used in "hash:table" is the database file name without the
".db" suffix. </dd>
<dt> <b>internal</b> </dt>
<dd> A non-shared, in-memory hash table. Its content are lost when
a process terminates. </dd>
<dt> <b>ldap</b> (read-only) </dt>
<dd> Perform lookups using the LDAP protocol. Configuration details
are given in the <a href="ldap_table.5.html">ldap_table(5)</a>. </dd>
<dt> <b>mysql</b> (read-only) </dt>
<dd> Perform MySQL database lookups. Configuration details are given
in <a href="mysql_table.5.html">mysql_table(5)</a>. </dd>
<dt> <b>netinfo</b> (read-only) </dt>
<dd> Perform Netinfo database lookups. </dd>
<dt> <b>nis</b> (read-only) </dt>
<dd> Perform NIS database lookups. </dd>
<dt> <b>nisplus</b> (read-only) </dt>
<dd> Perform NIS+ database lookups. Configuration details are given
in <a href="nisplus_table.5.html">nisplus_table(5)</a>. </dd>
<dt> <b>pcre</b> (read-only) </dt>
<dd> A lookup table based on Perl Compatible Regular Expressions.
The file format is described in <a href="pcre_table.5.html">pcre_table(5)</a>. The lookup table
name as used in "<a href="pcre_table.5.html">pcre</a>:table" is the name of the regular expression
file. </dd>
<dt> <b>pgsql</b> (read-only) </dt>
<dd> Perform PostgreSQL database lookups. Configuration details
are given in <a href="pgsql_table.5.html">pgsql_table(5)</a>. </dd>
<dt> <b>proxy</b> (read-only) </dt>
<dd> Access information via the Postfix <a href="proxymap.8.html">proxymap(8)</a> service. The
lookup table name syntax is "<a href="proxymap.8.html">proxy</a>:<a href="DATABASE_README.html">type:table</a>". </dd>
<dt> <b>regexp</b> (read-only) </dt>
<dd> A lookup table based on regular expressions. The file format
is described in <a href="regexp_table.5.html">regexp_table(5)</a>. The lookup table name as used in
"<a href="regexp_table.5.html">regexp</a>:table" is the name of the regular expression file. </dd>
<dt> <b>sdbm</b> </dt>
<dd> An indexed file type based on hashing. This is available only
on systems with support for SDBM databases. Database files are
created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. The lookup
table name as used in "sdbm:table" is the database file name without
the ".dir" or ".pag" suffix. </dd>
<dt> <b>sqlite</b> (read-only) </dt>
<dd> Perform SQLite database lookups. Configuration details are given
in <a href="sqlite_table.5.html">sqlite_table(5)</a>. </dd>
<dt> <b>static</b> (read-only) </dt>
<dd> Always returns its lookup table name as lookup result. For
example, the lookup table "static:foobar" always returns the string
"foobar" as lookup result. </dd>
<dt> <b>tcp</b> </dt>
<dd> Access information through a TCP/IP server. The protocol is
described in <a href="tcp_table.5.html">tcp_table(5)</a>. The lookup table name is "<a href="tcp_table.5.html">tcp</a>:host:port"
where "host" specifies a symbolic hostname or a numeric IP address,
and "port" specifies a symbolic service name or a numeric port
number.
</dd>
<dt> <b>texthash</b> (read-only) </dt>
<dd> This produces similar results as hash: files, except that you
don't have to run the <a href="postmap.1.html">postmap(1)</a> command before you can use the
file, and that texthash: does not detect changes after the file is
read. The lookup table name is "texthash:filename", where the file
name is taken literally; no suffix is appended. </dd>
<dt> <b>unix</b> (read-only) </dt>
<dd> A limited way to query the UNIX authentication database. The
following tables are implemented:
<dl>
<dt> <b>unix:passwd.byname</b> </dt>
<dd>The table is the UNIX password database. The key is a login
name. The result is a password file entry in passwd(5) format.
</dd>
<dt> <b>unix:group.byname</b> </dt>
<dd> The table is the UNIX group database. The key is a group name.
The result is a group file entry in group(5) format. </dd>
</dl> </dd>
</dl>
</blockquote>
<p> Other lookup table types may be available depending on how
Postfix was built. With some Postfix distributions the list is
dynamically extensible as support for lookup tables is dynamically
linked into Postfix. </p>
</body>
</html>