Custom Exim Templates - SpamTagger/SpamTagger-Plus GitHub Wiki
Context for MailCleaner users
Historically, the configuration of Exim templates in MailCleaner was possible by directly editing files in /usr/mailcleaner/etc/exim/. However, this was always discouraged not only because of the potential risks from a poorly configured file, but also because it involved touching files within the Git source tree which could potentially break future updates.
With the migration to bootable container images for SpamTagger Plus, it is no longer possible to modify any of the files within the read-only filesystem, which includes all of /usr/spamtagger. In order to still allow for modifying these configurations and to resolve the longstanding issue with applying changes to the application source directly, custom Exim configurations have been moved to /etc/spamtagger/exim/ as discussed here.
Warnings
Consequences of invalid configuration files
An improperly configured Exim configuration file is very likely to break mail flow. Exim can be quite strict about syntax and is not forgiving of unknown options, etc. If it notices any of these problems it will either refuse to start for any problem which it can detect at startup or will crash when a message causes it to encounter an invalid option.
Updates
Due to the migration to bootable containers, it is not necessary to maintain special accommodations for different versions of Exim. This method of distributing the OS (including the Exim package) with the matching SpamTagger Plus application code ensures that it is not possible to update one without also updating the other. Because of this, the versioned configurations in MailCleaner, such as /usr/mailcleaner/etc/exim/exim_stage1.conf_template_4.98 are unnecessary.
However, this does add a different potential issue. Specifically, in MailCleaner's Updater4MC scripts, updates to new Exim versions would check for changes to the default Exim template files and if any had been made it would refuse to update automatically. The admin would be instructed to migrate their changes to the new version of the configuration file(s) and then manually run the update instead. This prevented important custom configurations from being overwritten and ensured that the configuration template would be compatible with the new Exim version after the update.
Since SpamTagger Plus now uses bootable container images which replace the entire root filesystem all at once it is not easy to know if an impending update will make changes to the Exim version. This means that it is currently not possible to block updates you are using a custom configuration template.
This should generally not be too much of a problem because Exim will generally emit warnings for deprecated features for a few releases before they will start causing crashes. However occasionally new releases will require new syntax as demonstrated by the 'taintedness' changes in 4.94 and so existing configuration files can cause crashes if they are not updated.
In order to give admins notice, alerts will be added to the InfoBox on the homepage of the WebUI prior to releases with new versions of Exim. However, some Exim updates can be to fix critical vulnerabilities, so the timeline for these warnings will differ depending on the severity. An issue exists to change the InfoBox in order to send alert emails to the administrator.
In the future tags may be added with the Exim release which would prevent systems from updating to the next release until a manual bootc switch is run.
Base configuration templates
Here the "base templates" will refer to those which are directly loaded by each of the Exim MTAs. These may then go on to load additional configurations.
Exim Stage 1 (Incoming MTA)
This is the most sophisticated template, which does most of the filtering which is configured in Configuration->SMTP, the initial Configuration->Domains->[select domain]->Recipient verification and many other checks. It eventually passes acceptable mail to Stage 2. Outgoing messages are delivered directly by Stage 1 without passing through the other MTA stages.
It uses the default configuration template /usr/spamtagger/etc/exim/exim_stage1.conf_template and can be overridden with /etc/spamtagger/exim/exim_stage1.conf_template.
Exim Stage 2 (Filtering MTA)
This is the simplest template. It essentially just listens to accept forwarded messages from Exim Stage 1 and then saves them to a directory where they will be filtered by MailScanner.
It uses the default configuration template /usr/spamtagger/etc/exim/exim_stage2.conf_template and can be overridden with /etc/spamtagger/exim/exim_stage2.conf_template.
Exim Stage 4 (Outgoing MTA)
This template handles final delivery of all incoming emails. After MailScanner finishes scanning the message which were queue by Stage 2, it provides the results to the SpamHandler module. SpamHandler decides whether to place the message in a quarantine, drop it or to place it in the Stage 4 queue for final delivery. That decision depends on MailScanner's classification of the message as well as user preferences. Flags may be added to message that are placed in the Stage 4 queue in order to impact the delivery action. This MTA will deliver items to their final destination, perform DKIM signing and handle the Copy-To and Archiving steps.
It uses the default configuration template /usr/spamtagger/etc/exim/exim_stage4.conf_template and can be overridden with /etc/spamtagger/exim/exim_stage4.conf_template.
Additional files
The base templates have various __INCLUDE__ and .include_if_exists clauses which will pull in additional configurations from other files upon start up.
.include_if_exists
These clauses are a built-in feature of Exim. If the file following that string exists it will be included, verbatim, at that point in the configuration within the running program memory. The content of that file will not be included in the dumped configuration file, nor will there be any variable substitution (except where the variables are defined in Exim-native syntax (like how VARDIR is used in the base configurations).
Example:
.include_if_exists VARDIR/spool/tmp/exim/stage1/blacklists/ip-domain
Note that all existing instances of these clauses in the default base configurations refer to dynamically generated files. If you try to apply changes to these files they will be overwritten during the dump processes, prior to the service starting. Because of this, .include_if_exists files will only be useful if you create new ones.
__INCLUDE__
The clauses are SpamTagger-specific and are not natively understood by Exim. They exist in order to allow for the usage of __VARIABLES__ within those files. Because of this, the actual files that you will need to create in order to override the defaults or add your own __INCLUDE__s should be suffixed with _template also.
The dumper script does a "glob" search for all additional '*_template' files it the relevant /usr/spamtagger/etc/exim/stage${stage}/ and /etc/spamtagger/exim/stage${stage}/ directories, so it can include default and non-default files with preference to the custom version if it exists.
During the dump processes, the _template files will be read and have all of the same substitutions made as the base templates. The substituted text will be written in the directory /var/spamtagger/spool/tmp/exim/stage${stage}/ with _template removed from the filename. That file is then what needs to be included.
Example:
__INCLUDE__ stage1/ldap
If the file /etc/spamtagger/exim/stage1/ldap_template exists, it will have been dumped to /var/spamtagger/spool/tmp/exim/stage1/ldap with any relevant substitutions. Otherwise this will be done using /usr/spamtagger/etc/exim/stage1/ldap_template. This will be done regardless of whether the line above is included in any of the base templates.
By including the __INCLUDE__ statement above, the output file just mentioned will then be taken and inserted at that location in the dump of the base template.
Assuming that this line appears in the base template for Stage 1, you will see that the dumped contents of that file appear in the appropriate location in /var/spamtagger/spool/tmp/exim/exim_stage1.conf.
__VARIABLE__ substitutions
The base templates and __INCLUDE__ templates can contain variables formatted like __VARIABLE__. This is a spamtagger-specific syntax and is not natively understood by Exim.
These __VARIABLE__s are placeholders for values which are dynamically inserted by the configuration dumper each time the service starts. To get a comprehensive list of the existing variables which can be substituted, you will need to inspect /usr/spamtagger/bin/dump_exim_config.pl. These values are generally collected from the database, assembled into a Perl hash and then substituted in the templates with $template->set_replacements(\%hash).
If you would like to override these variables you can do so with /etc/spamtagger/exim/custom_variables.toml. This should be formatted like:
__MYVARIABLE__ = "Top-level value"
__HELO_NAME__ = "custom.hostname.com"
__SMTP_BANNER__ = "Fallback banner, never used"
REJECTDMARC = 0
[exim_conf]
__SMTP_BANNER__ = "SpamTagger MTA custom.hostname.com"
[stage1_conf]
__SMTP_BANNER__ = "Anonymous MTA custom.hostname.com"
Items which appear prior to any [table] line are for general use and will be applied to that variable if it appears anywhere. In this case __MYVARIABLE__ will be for custom use within your custom template files. __HELO_NAME__ will apply to any usage of __HELO_NAME__ (currently as part of the %sys_conf hash only).
Be careful with top-level variables as you may not notice that a variable which does not appear to be in use where you intend to use it could already be in use in a different template.
Items without surrounding __s refer to binary conditions and should only be '0' or '1'. All other values will be ignored, with errors. These enable or disable blocks of code like:
__IF__ REJECT_DMARC
# This block will be included if REJECT_DMARC == 1
...
__FI__
Items that appear after a [table] line will apply when the get_overrides function is called with that name (typically for a hash which also matches that name). In this case, the value of __SMTP_BANNER__ in %exim_conf will be overridden with "SpamTagger MTA custom.hostname.com". This is give priority over the value pulled from the database (within the dumper file) as well as the top-level instance with the value "Fallback banner, never used".
[stage1_conf], [stage2_conf], and [stage4_conf] are special cases which will further override values from [exim_conf] for that specific MTA. In this case the value for __SMTP_BANNER__ for Stage 1 will be "Anonymous MTA custom.hostname.com" while the other two stages will remain with "SpamTagger MTA custom.hostname.com".