4.9 - POP Retriever

This Mailismus component is a specialised POP3 client which acts as a POP Retriever.
It can download messages from a remote mailbox into either:
• A local mailbox, managed by the Mailismus MS.
• A local directory, where the messages are saved as simple files.
• A Mailismus MTA - the messages are injected into its SMTP server.

The POP Retriever is a scheduled batch-mode utility which downloads messages from multiple POP3 mailservers, allowing you to consolidate messages held at multiple ISPs (or other sources) in one place on your local machine.
As such, it is unique within Mailismus in being a personal utility rather than a server component. Even if you are not running any servers (SMTP, IMAP, or POP) you can still make use of this POP3 Retriever.
Having consolidated all your messages locally however, this utility goes well with either the IMAP or POP3 servers, which can then be used to access the messages using your favourite email client (Thunderbird, Outlook, Eudora, etc).

The Retriever requires the Message-Store (see section §4.2) and Directory subsystems to be configured (unless in download-as-file or SMTP-injection mode - see below).

As you will note below (see the client/recip config setting), the POP3 Retriever is configured on a per-POP3-user-account basis.


4.9.1 - Task Config

The POP3 retriever config block consists of some settings that apply to all clients, followed by zero or more client definitions.
A client represents the client side (local) of a session with a nominated server (remote), so each client is responsible for downloading messages from its associated server.

<pop3download>
    <audit rot="monthly">%DIRLOG%/audit/Y%Y%/pop3download-%DT%.log</audit>
    <transcript>%DIRLOG%/transcripts/pop3download-%DT%.log</transcript>
    <timeout>4m</timeout>
    <delay_start>1m</delay_start>
    <clients>
        <client> ...  </client>
        <client> ...  </client>
    </clients>
</pop3download>

audit
This specifies an audit log, reporting the download of each message.
It is meant to serve as a formal record of activity, suitable for archiving (as opposed to the more debug-oriented trace logs).
This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This config item has no default and auditing will be turned off if it is omitted.

transcript
This file records a transcript of the POP3 protocol dialogue.
This config item supports the same attributes as the GreyLog loggers (rotation, buffering, etc - see GreyLog Guide).
This setting has no default and transcripts will be turned off if it is omitted.

timeout
This specifies the response timeout. A a client will abort the session if the server takes longer than this to respond to any command.
Defaults to 4 minutes.

delay_start
This specifies the initial interval before running any POP3 clients, on Mailismus startup. Defaults to 1 minute.


4.9.2 - Clients

A client config block is shown below, with largely default values.

<client id="id1" recip="localuser1" freq="5m" runs="0" preserve="N" capa="N"
            omitreceivedheader="N">
    <downloads_directory>/path/to/directory</downloads_directory>
    <server address="pop.gmail.com:110">
        <username>myname</username>
        <password>mypassword</password>
        <authtypes initrsp="N">USERPASS | SASL_PLAIN</authtypes>
    </server>
    <ssl> ... </ssl>
</client>

client/id
This is a mandatory parameter specifying an arbitrary tag that serves to uniquely identify this client definition.

client/recip
This is a mandatory parameter specifying a local username (as defined in your Message-Store) and downloaded messages will be written to that user's Inbox.
Eg. /path/to/username/Maildir/new

client/freq
This specifies how frequently to check for remote messages, and defaults to 5 minutes.
A value of 0 (zero) means to run just once, and this client will be deactivated after its initial download session.

client/runs
This specifies how often to run the run this POP3 retrieval, and defaults to 0 (zero), meaning no limit.
If it is set to a finite value, then it overrides the single run usually implied by freq=0 and the retriever will run that many times, in immediate succession. In other words, freq=0 merely means that runs defaults to 1 rather than 0, but it doesn't constrain it.

client/preserve
By default, Mailismus deletes messages from the remote server after deleting them, which is the standard behaviour of POP clients.
If this option is set to Yes, then it will leave the messages on the remote server.
If omitted, this option defaults to No.

Note that when Mailismus connects to a remote POP3 server, it always downloads every message it finds and there's no recognition of new or old messages, so this option could result in you creating many local duplicates of each message.
Therefore this option would be of most use when combined with freq=0, ie. a one-time download.

client/omitreceivedheader
By default, Mailismus adds a Received trace header to each downloaded message, as it writes it to the local mailbox.
If this option is set to Yes, then it will not do so.
If omitted, this option defaults to No.

client/capa
This option defaults to No, but if set to Yes, it causes Mailismus to issue the POP3 CAPA command immediately on connecting to the remote server, and again after completing a STLS command (to switch to SSL mode).
This serves no functional purpose, but in combination with the Transcript logs, it can be used to troubleshoot a POP3 session by letting you examine the remote server's supported capabilities.

downloads_directory
This is an optional item which is blank by default, but if specified, it means we are in download-as-file mode, and rather than delivering the downloaded messages to a local Inbox, they will instead be saved as discrete text files in the specified directory.
If the directory path contains the token %U% it will be replaced with a username as specified in the client/recip setting above, but beyond that, the client/recip setting has no other significance when we're in download-as-file mode.

server/address
This is the server address to connect to, and can be specified as either a hostname or a dotted IP address.
The example config above illustrates how to explicitly specify a TCP port, but it may be omitted, in which case it defaults to 995 for a pure SSL connection, else to 110.

server/username
server/password
These specify the credentials to use for logging into the remote POP server.

server/authtypes
This specifies the authentication method(s) to use when logging into the server. More than one can be specified, in which case the downloader will try each one in turn, until it succeeds.
If omitted, this defaults to the sequence shown above, ie. POP3's built-in USER and PASS commands, followed by SASL-Plain.

The full set of allowed authentication methods is:
USERPASS, APOP, SASL_PLAIN, SASL_CRAM_MD5, SASL_EXTERNAL

SASL-External is only applicable in SSL mode, and you have to supply a client certificate (see the ssl config section below).
The server/username and server/password config settings will be ignored for this method, and may be omitted, as your certificate alone will serve as your credentials.
The common expectation is that the remote server will treat your certificate's Common-Name attribute as a username, but technically it's up to the server how to interpret SASL-External.


4.9.3 - SMTP Mode

As explained above, the default behaviour of the POP retriever is to write the downloaded messages to the specified recipient's local mailbox, but there is an SMTP-injection mode, where it can instead inject messages directly into the queue of a local Mailismus MTA, from where they will be forwarded as normal.
This requires the addition of the optional smtp config section shown below (and you'll also have to set the enabled attribute to Yes - or omit it, which has the same effect).

Unless otherwise described here, all the other config settings continue to behave as described above. There is no need to repeat them all.

Also note that SMTP-mode does not require a local Message-Store to be configured.
It does however require the Mailismus MTA to be running within the same Java process, as it writes the message directly into its queue.

<client recip="localuser1">
    <server> ... </server>
    <ssl> ... </ssl>
    <smtp enabled="N">
        <sender>pop3downloader@localhost</sender>
    </smtp>
</client>

client/recip
Instead of being a local username which identifies a local mailbox, this is now interpreted as a recipient email address.
If a local Directory is defined (optional), Mailismus checks if this address corresponds to a local user or alias, and if the latter, it expands it into a list of recipient addresses, as defined for that alias (see section §4.10).

smtp/sender
This config item is required in SMTP mode, and it specifies the SMTP envelope-sender to use.
Obviously the original SMTP envelope information has been stripped by the time a message reached the remote mailbox from which we downloaded it, but we need to recreate it when reintroducing the message to our SMTP server, hence this setting.


4.9.4 - SSL

The optional SSL config block is a generic NAF facility (see NAF Guide, section §8) and the only thing left to add here is the treatment of its latent and mandatory attributes in the context of the POP3 Retriever.

If the latent SSL config setting is set to its default value of False, then connections will be established in SSL mode to start with.
If True, then the Retriever connects in non-SSL mode, and after the remote POP3 server sends its initial greeting, we will issue the STLS command to switch to SSL mode.

The mandatory SSL config setting is ignored by the Retriever.