4.5 - The Delivery Task

4.5.1 - Function

The Delivery Task is a NAFlet which delivers stored messages onwards to their next-hop destination. It is one of the big two Mailismus naflets, the other being the Submit Task.

The Delivery task is composed of a marshalling agent called the SMTP Sender (or Forwarder), which obtains pending messages from the queue (see §4.3), and hands them to SMTP Clients.
The SMTP Clients connect to the remote SMTP servers and perform the necessary SMTP protocol operations to deliver the messages.

The Delivery task works by fetching a batch of messages from the queue, and allocating this batch optimally to a set of SMTP-Client instances. By "optimally", we mean that if the same message has multiple recipients in the same domain, they will be allocated to the same client, so that they can all be delivered within a single SMTP connection. Mailismus also batches additional messages for the same domain into a single connection.
This cycle is continuously repeated at configurable intervals. The intention is that the interval should be very short if the previous batch did find some messages, and longer if no messages were waiting, to avoid hogging the machine if there is no work to do.
If the batch of messages exceeds configured limits on the number of total connections or connections per domain, then the excess messages will be carried over into the next batch, where they will be nearer the front of the queue.

Once this is understood, you can see that most of the Sender config deals with the tuning of this batch processing in terms of limits and intervals, while the SMTP-Client config is focussed mainly on protocol options.

4.5.2 - Task Config

    <relays> ... </relays>
    <audit rot="monthly">%DIRLOG%/audit/Y%Y%/delivered-%DT%.log</audit>

See section §4.5.4 below.

This specifies an audit log, reporting the delivery of each message to each recipient.
It is meant to serve as a formal record of activity, suitable for archiving (as opposed to the more debug-oriented trace logs). It records the delivery timestamp, the sender and recipient email addresses, and the IP addresses the message was received from and delivered to.
This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This config item has no default and auditing will be turned off if it is omitted.

If any entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to a tenth of a second.

If no entries were found when we polled the queue to build the latest cached batch, this is the pause before polling again.
It defaults to 15 seconds, and this low/high split delay allows us to avoid thrashing the queue when it has no pending messages.

If we encountered an error when we polled the queue to build the latest cached batch, this is the pause before polling again.
Defaults to 3 minutes.

This specifies the pause on initial system startup, before we first poll the queue.
Defaults to 30 seconds.

This specifies the max number of simultaneous outgoing SMTP connections.
Defaults to 2,500 unless in slave-relay mode, in which case it defaults to 500.
It should be set to the same size as queuecache as any larger is pointless (but harmless, as will never be exercised) but any smaller means that there will be leftover recipients in the cache after each delivery batch, which means the queue-manager is doing wasted work to load and restore them.

This specifies the max number of simultaneous outgoing SMTP connections to any one destination (remote domain or relay). Of course multiple connections to one destination domain might in practice be spread amongst multiple remote servers, but we treat it as as one target, and limit the total connections to it.
Defaults to 20, and is irrelevant in slave-relay mode, since all connections would be made to the smarthost, and there is no limit.

This specifies the max number of messages we will batch into a single SMTP connection.
This setting is the one that mainly controls how long it can take to process a single batch, and it defaults to 100. This limit is based on ensuring that a batch completes in a reasonable time, to minise the latency before we flush it back to the queue.

This specifies the max number of common recipients we will batch into a single message.
Defaults to 50 (well below RFC-5321 minimum of 100, just to be conservative).

This defaults to 2,500 and specifies the max size of the delivery batch, measured as the number of cached queue entries (each queue entry constitutes a unique recipient/message combination).
It is subject to the limit imposed by the queue manager itself, with its maxmemoryqueue setting (see §4.3.1).

4.5.3 - SMTP Client

The client's config block is listed below, with largely default values.

    <anonssl latent="Y" ... />
        <remotenet ip="%SYSNAME%/24">

This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This config item has no default and transcripts will be turned off if it is omitted.
See §4.4.7 for a description of the transcript contents.
The only difference with the server transcripts, is that in this case, we're the ones initiating the connections and sending the commands.

This specifies how the client will announce itself in its greeting message (SMTP: HELO or EHLO).
If absent it defaults to the setting in the application block - see §4.1
Unlike its namesake in the SMTP-Server config, it is critical for clients that this hostname is a valid DNS entry which matches the IP address we call out on, and ideally has a reverse entry as well (ie. the in-addr.arpa DNS domain).

This controls how aggressively the client pipelines its outgoing requests, when the remote Server advertises its support for the ESMTP pipelining extension.
The default is 25, meaning we will have at most 25 outstanding requests, before we pause to wait for responses.
Setting this to 1 effectively disables pipelining, regardless of what the Server supports. A value of 0 is also interpreted as being 1.

This specifies the max time to wait for a response from the server, after which the client will declare failure and abandon the connection.
The default is 1 minute.

This specifies the minimum data rate (in bps - bits per second) that you expect when sending the message body, and is used to extend the timeout interval during this phase of the connection if the message is sufficiently large.
This setting can only increase the timeout interval in effect, not reduce it, and if the message is judged too small to exceed the timeout setting, the latter will remain in effect.
The default value is 1,000,000 (but must be entered without punctuation) or in other words 1 Mbps.
Regardless of your outgoing bandwidth, it is probably inadvisable to assume that incoming data on a busy server at the other end exceeds 1Mbps per connection.

This tells the Client to use plain SMTP rather than ESMTP, by issuing the SMTP HELO greeting rather than EHLO.
The default is to announce our ESMTP capability, by saying EHLO.

This tells the Client to fall back to issuing a HELO command if the EHLO. is rejected.
The SMTP standard recommends falling back to HELO, but also mandates that servers must support EHLO, and you're very unlikely to come across any servers that don't support EHLO nowadays.
For that reason, this setting defaults to No (don't fall back) as falling back merely delays recognition of the far more likely possibility that our EHLO was rejected because of who we are, rather than because the server doesn't support EHLO.

Setting this to No enables the client to accelerate the standard SMTP end-of-connection dialogue, by disconnecting abruptly without sending the SMTP QUIT command.
This default to Yes (ie. send QUIT), as it is not generally considered polite behaviour to take this shortcut. It does not affect the message delivery either way, as the message transfer phase of the connection is already over at this stage.

If we issue an SMTP QUIT command, this controls whether we wait for the server's response, or accelerate the process by disconnecting immediately after sending the QUIT.
Clearly the message transfer phase is over at this, and we've already issued our final SMTP squawk, but abruptly disconnecting before the server has had a chance to respond to the QUIT may result in spurious errors in its logfiles, so for the sake of orthodoxy and politeness, the default is Yes, ie. we do wait for the QUIT response.

This enables what is known as the "Implicit MX" rule, whereby if the destination email domain has no mail relays specified in the DNS, then we fall back to treating it as a simple hostname.
For example: If sending a message to somebody@mydomain.org but the domain mydomain.org had no MX RRs (Resource Records), then we would simply connect to the host called mydomain.org (if any).
The Implicit MX rule has been mandated by the SMTP standards ever since RFC 821 was first published in 1982, but Mailismus defaults this setting to No on the grounds that it's very unlikely any valid email domain would rely on it today.
To formally conform to the standard, you need to intervene and set this to Yes.
Since the absence of any MX RRs generally means that the recipient address is invalid, enabling this setting will invariably result in a wasted hostname lookup after the MX lookup failed, and what is likely to be an unsuccessful SMTP connection if the hostname does happen to exist, so we strongly recommend leaving this off.

This is an optional config element which enables SSL to be used when connecting to the general population of SMTP servers (ie. quite apart from the specifically configured relays).
It contains the standard SSL attributes described in section §8 of the NAF Guide, and section §4.5.4 below (Relays) describes how these SSL settings are treated in the context of the SMTP client.
Obviously it makes no sense to require SSL support in the general population of SMTP servers on the Internet, so if you do use this config element, you should set its latent attribute to True, and mandatory should be left as False (as hinted in the above config block).
It is unlikely you would ever use anonssl and we are not necessarily recommending it, but it is there just in case you want to take up the invitation from any random SMTP server that advertises SSL capability (via the STARTTLS SMTP extension).

This serves the same purpose as in the Server config. It is an optional block containing one or more nested remotenet blocks. Each remotenet block has an ip attribute which specifies one ore more IP address blocks in CIDR notation (delimited by vertical bars), but you may specify a hostname in place of the IP part (as illustrated above).
The purpose of this is that each remotenet block identifies a family of remote servers (ie. those whose IP matches the CIDR block) to which you want to apply alternative config settings. There are no new settings defined for the remotenet block, but it may override most of the settings defined in the common client config.
The full list of settings that may be overridden within a remotenet block is:

4.5.4 - Relays

This optional config block allows Mailismus to function as a distribution node, by specifying message-routing rules based on either the recipient domain or the sender's email address. This is as opposed to the usual means by which mailservers (including Mailismus) generally route a message to the recipient domain, which is to look up its MX relays in the DNS and then forward the message to one of them.
These routing rules therefore override MX lookup, and allow you to manually define next-hop routing for specific addresses. Mailismus will compare the sender and recipient of each outgoing message to all the configured relay nodes, and if it finds a match, it forwards the message to that relay node. If no matches, DNS/MX-based forwarding is then performed as usual.

You can also specify a catch-all relay (known as the default relay), to which messages for all other domains are sent. This means no DNS lookups will ever be done to determine the next hop.
The relay1.mydomain.com entry in the example config block below is a default relay, as it has no destination-domains or source-routing rules specified.
If the default relay is the only entry in the relays block, then it becomes what is known as a smarthost, ie. all our outgoing messages are relayed through it. In this case, Mailismus is said to be in slave-relay mode.

In addition to supporting bespoke routing rules, the relays block also allows Mailismus to act as a secondary MX relay for various domains. In that case the domain's primary MX server address would be specified here as one of the relay nodes, along with the domain name (see the destdomains attribute below).
This functionality is required for backup mailservers because when a mailserver is configured as a secondary MX relay, it obviously can't use DNS lookup to locate the message destination, as it's own hostname would be returned as one of the MX relays, leading to a circular loop.

The relays config block can contain any number of relay entries (or none, as the entire block is optional), and the example block below illustrates various forms the relay entries can take.
The examples below are far from exhaustive, and most combinations of the various relay attributes can appear in most entries.
In particular, authentication and SSL are optional and either, both or neither can be specified in each individual relay definition.

It is also important to note that the destdomains and senders attributes are allowed to appear in the same relay definition, which would simply mean that both the source and destination routing rules point at the same next-hop relay.
If a message would match both a source-routing rule and one of the destination rules, then source-routing takes precedence, and it will get forward to the associated relay.

    <relay address="relay1.somedomain.com"/>
    <relay address="relay2.somedomain.com:8025" destdomains="domain1.com"/>
    <relay address="relay3.somedomain.com" senders="srcdomain1,senderB@srcdomain2"
        <auth enabled="Y" initrsp="N">
    <relay address="" destdomains="domain3a.com, domain3b.com">
        <auth enabled="Y" override="external"/>
        <ssl ... />
    <relay address="relay4.somedomain.com:8025" destdomains="domain4.com">
        <ssl ... />
    <relay address="relay5.somedomain.com" senders="sender1@srcdomain1">