4.5.1 - Function

The Delivery Task is a NAFlet which delivers stored messages onwards to their next-hop destination. It is one of the big two Mailismus naflets, the other being the Submit Task.

The Delivery task is composed of a marshalling agent called the SMTP Sender (or Forwarder), which obtains pending messages from the queue (see §4.3), and hands them to SMTP Clients.
The SMTP Clients connect to the remote SMTP servers and perform the necessary SMTP protocol operations to deliver the messages.

The Delivery task works by fetching a batch of messages from the queue, and allocating this batch optimally to a set of SMTP-Client instances. By "optimally", we mean that if the same message has multiple recipients in the same domain, they will be allocated to the same client, so that they can all be delivered within a single SMTP connection. Mailismus also batches additional messages for the same domain into a single connection.
This cycle is continuously repeated at configurable intervals. The intention is that the interval should be very short if the previous batch did find some messages, and longer if no messages were waiting, to avoid hogging the machine if there is no work to do.
If the batch of messages exceeds configured limits on the number of total connections or connections per domain, then the excess messages will be carried over into the next batch, where they will be nearer the front of the queue.

Once this is understood, you can see that most of the Sender config deals with the tuning of this batch processing in terms of limits and intervals, while the SMTP-Client config is focussed mainly on protocol options.

4.5.2 - Task Config

<deliver>
    <relays> ... </relays>
    <audit rot="monthly">%DIRLOG%/audit/Y%Y%/delivered-%DT%.log</audit>
    <queuecache>2500</queuecache>
    <maxconnections>2500</maxconnections>
    <maxserverconnections>20</maxserverconnections>
    <maxmessages>100</maxmessages>
    <maxrecips>50</maxrecips>
    <interval_low>100</interval_low>
    <interval_high>15s</interval_high>
    <interval_error>3m</interval_error>
    <delay_start>30s</delay_start>
    <client>
        ...
    </client>
</deliver>

relays
See section §4.5.4 below.

audit
This specifies an audit log, reporting the delivery of each message to each recipient.
It is meant to serve as a formal record of activity, suitable for archiving (as opposed to the more debug-oriented trace logs). It records the delivery timestamp, the sender and recipient email addresses, and the IP addresses the message was received from and delivered to.
This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This config item has no default and auditing will be turned off if it is omitted.

interval_low
If any entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to a tenth of a second.

interval_high
If no entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
It defaults to 15 seconds, and this low/high split delay allows us to avoid thrashing the queue when it has no pending messages.

interval_error
If we encountered an error when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to 3 minutes.

delay_start
This specifies the pause on initial system startup, before we first poll the queue.
Defaults to 30 seconds.

maxconnections
This specifies the max number of simultaneous outgoing SMTP connections.
Defaults to 2,500 unless in slave-relay mode, in which case it defaults to 500.
It should be set to the same size as queuecache as any larger is pointless (but harmless, as will never be exercised) but any smaller means that there will be leftover recipients in the cache after each delivery batch, which means the queue-manager is doing wasted work to load and restore them.

maxserverconnections
This specifies the max number of simultaneous outgoing SMTP connections to any one destination (remote domain or relay). Of course multiple connections to one destination domain might in practice be spread amongst multiple remote servers, but we treat it as as one target, and limit the total connections to it.
Defaults to 20, and is irrelevant in slave-relay mode, since all connections would be made to the smarthost, and there is no limit.

maxmessages
This specifies the max number of messages we will batch into a single SMTP connection.
This setting is the one that mainly controls how long it can take to process a single batch, and it defaults to 100. This limit is based on ensuring that a batch completes in a reasonable time, to minise the latency before we flush it back to the queue.

maxrecips
This specifies the max number of common recipients we will batch into a single message.
Defaults to 50 (well below RFC-5321 minimum of 100, just to be conservative).

queuecache
This defaults to 2,500 and specifies the max size of the delivery batch, measured as the number of cached queue entries (each queue entry constitutes a unique recipient/message combination).
It is subject to the limit imposed by the queue manager itself, with its maxmemoryqueue setting (see §4.3.1).

4.5.3 - SMTP Client

The client's config block is listed below, with largely default values.

<client>
    <transcript>%DIRLOG%transcripts/smtpclient-%DT%.log</transcript>
    <announcehost>hostname</announcehost>
    <maxpipeline>25</maxpipeline>
    <timeout>1m</timeout>
    <mindatarate>1000000</mindatarate>
    <sayHELO>N</sayHELO>
    <fallbackHELO>N</fallbackHELO>
    <sendQUIT>Y</sendQUIT>
    <waitQUIT>Y</waitQUIT>
    <fallbackMX_A>N</fallbackMX_A>
    <anonssl latent="Y" ... />
    <remotenets>
        <remotenet ip="%SYSNAME%/24">
            ...
        </remotenet>
    </remotenets>
</client>

transcript
This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This config item has no default and transcripts will be turned off if it is omitted.
See §4.4.7 for a description of the transcript contents.
The only difference with the server transcripts, is that in this case, we're the ones initiating the connections and sending the commands.

announcehost
This specifies how the client will announce itself in its greeting message (SMTP: HELO or EHLO).
If absent it defaults to the setting in the application block - see §4.1
Unlike its namesake in the SMTP-Server config, it is critical for clients that this hostname is a valid DNS entry which matches the IP address we call out on, and ideally has a reverse entry as well (ie. the in-addr.arpa DNS domain).

maxpipeline
This controls how aggressively the client pipelines its outgoing requests, when the remote Server advertises its support for the ESMTP pipelining extension.
The default is 25, meaning we will have at most 25 outstanding requests, before we pause to wait for responses.
Setting this to 1 effectively disables pipelining, regardless of what the Server supports. A value of 0 is also interpreted as being 1.

timeout
This specifies the max time to wait for a response from the server, after which the client will declare failure and abandon the connection.
The default is 1 minute.

mindatarate
This specifies the minimum data rate (in bps - bits per second) that you expect when sending the message body, and is used to extend the timeout interval during this phase of the connection if the message is sufficiently large.
This setting can only increase the timeout interval in effect, not reduce it, and if the message is judged too small to exceed the timeout setting, the latter will remain in effect.
The default value is 1,000,000 (but must be entered without punctuation) or in other words 1 Mbps.
Regardless of your outgoing bandwidth, it is probably inadvisable to assume that incoming data on a busy server at the other end exceeds 1Mbps per connection.

sayHELO
This tells the Client to use plain SMTP rather than ESMTP, by issuing the SMTP HELO greeting rather than EHLO.
The default is to announce our ESMTP capability, by saying EHLO.

fallbackHELO
This tells the Client to fall back to issuing a HELO command if the EHLO. is rejected.
The SMTP standard recommends falling back to HELO, but also mandates that servers must support EHLO, and you're very unlikely to come across any servers that don't support EHLO nowadays.
For that reason, this setting defaults to No (don't fall back) as falling back merely delays recognition of the far more likely possibility that our EHLO was rejected because of who we are, rather than because the server doesn't support EHLO.

sendQUIT
Setting this to No enables the client to accelerate the standard SMTP end-of-connection dialogue, by disconnecting abruptly without sending the SMTP QUIT command.
This default to Yes (ie. send QUIT), as it is not generally considered polite behaviour to take this shortcut. It does not affect the message delivery either way, as the message transfer phase of the connection is already over at this stage.

waitQUIT
If we issue an SMTP QUIT command, this controls whether we wait for the server's response, or accelerate the process by disconnecting immediately after sending the QUIT.
Clearly the message transfer phase is over at this, and we've already issued our final SMTP squawk, but abruptly disconnecting before the server has had a chance to respond to the QUIT may result in spurious errors in its logfiles, so for the sake of orthodoxy and politeness, the default is Yes, ie. we do wait for the QUIT response.

fallbackMX_A
This enables what is known as the "Implicit MX" rule, whereby if the destination email domain has no mail relays specified in the DNS, then we fall back to treating it as a simple hostname.
For example: If sending a message to somebody@mydomain.org but the domain mydomain.org had no MX RRs (Resource Records), then we would simply connect to the host called mydomain.org (if any).
The Implicit MX rule has been mandated by the SMTP standards ever since RFC 821 was first published in 1982, but Mailismus defaults this setting to No on the grounds that it's very unlikely any valid email domain would rely on it today.
To formally conform to the standard, you need to intervene and set this to Yes.
Since the absence of any MX RRs generally means that the recipient address is invalid, enabling this setting will invariably result in a wasted hostname lookup after the MX lookup failed, and what is likely to be an unsuccessful SMTP connection if the hostname does happen to exist, so we strongly recommend leaving this off.

anonssl
This is an optional config element which enables SSL to be used when connecting to the general population of SMTP servers (ie. quite apart from the specifically configured relays).
It contains the standard SSL attributes described in section §8 of the NAF Guide, and section §4.5.4 below (Relays) describes how these SSL settings are treated in the context of the SMTP client.
Obviously it makes no sense to require SSL support in the general population of SMTP servers on the Internet, so if you do use this config element, you should set its latent attribute to True, and mandatory should be left as False (as hinted in the above config block).
It is unlikely you would ever use anonssl and we are not necessarily recommending it, but it is there just in case you want to take up the invitation from any random SMTP server that advertises SSL capability (via the STARTTLS SMTP extension).

remotenets
This serves the same purpose as in the Server config. It is an optional block containing one or more nested remotenet blocks. Each remotenet block has an ip attribute which specifies one ore more IP address blocks in CIDR notation (delimited by vertical bars), but you may specify a hostname in place of the IP part (as illustrated above).
The purpose of this is that each remotenet block identifies a family of remote servers (ie. those whose IP matches the CIDR block) to which you want to apply alternative config settings. There are no new settings defined for the remotenet block, but it may override most of the settings defined in the common client config.
The full list of settings that may be overridden within a remotenet block is:
• announcehost
• maxpipeline
• timeout
• mindatarate
• sayHELO
• fallbackHELO
• sendQUIT
• waitQUIT
• anonssl

4.5.4 - Relays

This optional config block allows Mailismus to function as a distribution node, by specifying message-routing rules based on either the recipient domain or the sender's email address. This is as opposed to the usual means by which mailservers (including Mailismus) generally route a message to the recipient domain, which is to look up its MX relays in the DNS and then forward the message to one of them.
These routing rules therefore override MX lookup, and allow you to manually define next-hop routing for specific addresses. Mailismus will compare the sender and recipient of each outgoing message to all the configured relay nodes, and if it finds a match, it forwards the message to that relay node. If no matches, DNS/MX-based forwarding is then performed as usual.

You can also specify a catch-all relay (known as the default relay), to which messages for all other domains are sent. This means no DNS lookups will ever be done to determine the next hop.
The relay1.mydomain.com entry in the example config block below is a default relay, as it has no destination-domains or source-routing rules specified.
If the default relay is the only entry in the relays block, then it becomes what is known as a smarthost, ie. all our outgoing messages are relayed through it. In this case, Mailismus is said to be in slave-relay mode.

In addition to supporting bespoke routing rules, the relays block also allows Mailismus to act as a secondary MX relay for various domains. In that case the domain's primary MX server address would be specified here as one of the relay nodes, along with the domain name (see the destdomains attribute below).
This functionality is required for backup mailservers because when a mailserver is configured as a secondary MX relay, it obviously can't use DNS lookup to locate the message destination, as it's own hostname would be returned as one of the MX relays, leading to a circular loop.

The relays config block can contain any number of relay entries (or none, as the entire block is optional), and the example block below illustrates various forms the relay entries can take.
The examples below are far from exhaustive, and most combinations of the various relay attributes can appear in most entries.
In particular, authentication and SSL are optional and either, both or neither can be specified in each individual relay definition.

It is also important to note that the destdomains and senders attributes are allowed to appear in the same relay definition, which would simply mean that both the source and destination routing rules point at the same next-hop relay.
If a message would match both a source-routing rule and one of the destination rules, then source-routing takes precedence, and it will get forward to the associated relay.

<relays>
    <relay address="relay1.somedomain.com"/>
    <relay address="relay2.somedomain.com:8025" destdomains="domain1.com"/>
    <relay address="relay3.somedomain.com" senders="srcdomain1,senderB@srcdomain2"
                    sendernets=\"192.168/16, 127.0.0.1\">
        <auth enabled="Y" initrsp="N">
            <username>myname1<username>
            <password>mypass1<password>
        </auth>
    </relay>
    <relay address="192.168.101.1:8025" destdomains="domain3a.com, domain3b.com">
        <auth enabled="Y" override="external"/>
        <ssl ... />
    </relay>
    <relay address="relay4.somedomain.com:8025" destdomains="domain4.com">
        <ssl ... />
    </relay>
    <relay address="relay5.somedomain.com" senders="sender1@srcdomain1">
</relays>

• address
  This is the only mandatory attribute within a relay block, and it specifies the relay's network address.
  The destination address can be specified as either a hostname or a dotted IP, with or without a colon-separated TCP port. If no TCP port is specified, then it defaults to the standard SMTP port of 25.
• destdomains
  This specifies the destination domain(s) to which this relay definition applies, and effectively constitutes a destination-routing rule. It consists of a comma-separated list of one or more domain names, each of which must be unique among all the defined relays.
  If a recipient address matches one of these domains, then Mailismus will forward the outgoing message to this relay.
  This attribute is optional since as explained above, if both destdomains and senders are omitted, this entry becomes a default relay. However, you can only specify one default relay, so at least one of these attributes are required for the other relays.
  While it is possible to define multiple relay blocks all containing one destination domain that points to the same address, it is better to list all such domains in the same relay block, as that enables more optimal batching of related recipients into SMTP messages. That is, provided there are no other differences such as specific authentication procedures for a particular domain, in which case it should of course have its own relay definition.
• senders
  This specifies a source-routing rule and consists of a comma-separated list of one or more sender addresses, which can be expressed as either a full username@domain email address, or a wildcard domain name. Each such form must be unique across all the defined relays.
  The most specific match takes precedence, so in the above example, messages from sender1@srcdomain1 would be forwarded to relay5, while messages from any other user in srcdomain1 would be forwarded to relay3.
  This attribute is optional, but as explained above, there can be only one relay which specifies neither senders nor destdomains
• sendernets
  This is an optional attribute which is only relevant if senders is specified (and it is optional even then) and which restricts the source IPs to which this source-routing rule should be applied.
  It specifies one or more hostnames (expressed as either a hostname or a dotted decimal IP) and/or subnets (specified in CIDR notation, with a trailing routing prefix).
  Using the example config block above to illustrate its functionality, only messages which arrived into Mailisms from 192.168.x.x or localhost would be eligible to have the relay3 source-routing rule applied to them. All other messages will automatically be a non-match.
  Although this attribute is optional, it is recommended that it always accompanies senders, to guard against routing loops.
• auth
  This block defines any SMTP-Authentication settings that are required to connect to the relay. If enabled, the SMTP client will attempt to authenticate to the relay before sending any messages.
  • enabled
    This defaults to false, and specifies whether SMTP-Auth is actually enabled for this relay.
  • override
    Mailismus supports three SASL authentication methods, namely Plain, CRAM-MD5 (specified as cram_md5) and External.
    By default, Mailismus will automatically adapt to whatever mechanisms the remote server advertises that it supports, and if it doesn't support any of these three, Mailismus will attempt to continue without authenticating.
    This optional attribute tells Mailismus to unconditionally authenticate using the specified mechanism, and it will not bother checking what mechanisms the remote server advertises.
  • initrsp
    This is a low-level protocol setting, to do with the fact that SASL permits some authentication data to be sent along with the initial authenticate command, thus reducing the number of round trips required.
    This is clearly a good thing and might seem like an obvious choice, but not all servers support the so-called initial-response feature, so this is set to false by default, to guarantee interoperability.
  • username and password
    These attributes specify the credentials to supply to the remote server, when performing the SASL authentication.
    Note that in the examples above, these parameters were omitted for the SASL-External case. That is because it uses a client certificate as its credentials, and any username/password parameters would be ignored.
• ssl
  This defines any SSL parameters required to connect to the relay. SSL configuration is specified in the NAF Guide (see section §8) and the only thing left to add here is the treatment of its latent and mandatory attributes in the context of the SMTP Client.
  If latent is set to its default value of False, then the connection will be established in SSL mode to start with.
  If latent is True, then Mailismus will switch to SSL mode immediately after the server greeting (ie. before commencing a message or doing any authentication), but only if the remote server advertises SSL capability (via the STARTTLS extension).
  The client is normally prepared to continue in non-SSL mode if the server doesn't support SSL, but if mandatory is set to True, then it will abort the connection immediately after the initial server greeting.