To send mails, today some tests use msmtp and others our internal smtpc.py.
This works, but msmtp slows down the tests significantly, and smtpc.py
is also not particularly fast, and also has some limitations.
This patch introduces a new SMTP client tool written in Go, and makes
almost all the tests use it.
Some tests still remain on msmtp, mainly for client-check compatibility.
It's likely that this will be moved in later patches to a separate
special-purpose test.
With this patch, integration tests take ~20% less time than before.
This patch updates the shell scripts with some of the common best
practices, which should make them more resilient to unusual failures and
unexpected environments (in particular, directories with spaces).
Most of these were identified by shellcheck.
Most integration tests depend on the $HOSTALIASES environment variable
being functional. That variable works on most systems, but not all. In
particular, systems with `systemd-resolved` can cause the variable to be
ignored.
This was reported by Alex Ellwein in
https://github.com/albertito/chasquid/issues/20.
This patch makes the affected tests to be skipped if $HOSTALIASES is not
working properly. It also removes unnecessary hosts files from tests
which don't need it, and documents this behaviour.
Thanks to Alex Ellwein and foxcpp@ for reporting and helping investigate
this issue!
Python 2 is approaching end of life, and we only need it to run
the mail_diff test utility.
This patch updates mail_diff to run on Python 3, which only needed minor
changes.
This patch contains some changes to generate tidier DSNs, which should
make them slightly more readable.
In particular, it also makes it able to handle multi-line errors much
better than before.
Our non-delivery status notifications are quite simple today, but that
makes it much more difficult to support internationalization and
cross-language reporting.
There is a standard for internationalized DSNs, RFC 6533 (which builds
on top of the structured DSNs from RFC 3464).
This patch changes our DSN messages to be based on those standards, so
it is easier for MUAs to display reports according to the users'
languages preferences.
Note we still use message/rfc822 + 8bit to transmit the message, instead
of message/global, for compatibility reasons. This seems to be more
universally compatible, but the decision might be revisited in the
future. See RFC 5335 (section 4.6 in particular).
Despite its loose appearance, the "Received" header has a reasonably
standarized format.
We were not following the standard format as closely as we should; this
rarely causes problems in this particular case, but there's no need to
deviate from it.
This patch changes the Received header generation as follows:
- The "from" section now uses the remote address as canonical (for
non-authenticated users) which provides more valuable information
than the user-supplied EHLO address (which is also included).
- The remote authenticated user is now hidden, for additional privacy.
- Use the "with" optional clause.
- Use the standard way of printing TLS cipher suite.
- Use the standard way of printing address literals.
The nc.py script is only used in a single test, and for waiting for a
TCP port to be opened for listening.
This patch replaces it entirely, by using chamuyero for the test, and
bash for waiting on a TCP port.
This patch adds support for TLS-wrapped submission connections.
Instead of clients establishing a connection over plain text and then
using STARTTLS to switch over a TLS connection, this new mode allows the
clients to connect directly over TLS, like it's done in HTTPS.
This is not an official standard yet, but it's reasonably common in
practice, and provides some advantages over the traditional submission
port.
The default port is 465, commonly used for this; chasquid defaults to
systemd file descriptor passing as for the other protocols (for now).
Netcat's behaviour after seeing EOF from stdin seems to not be very
portable or consistent, even under the same platform.
This has caused t-05-null_address to break recently under some
conditions, for example depending on the particular Debian version of
netcat-openbsd used, and the current situation is unclear.
See https://bugs.debian.org/854292 and https://bugs.debian.org/849192
for more details.
To stop depending on this brittle behaviour, this patch unfortunately
introduces a simple python3-based netcat for our tests to use.
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
The queue currently only considers failed recipients when deciding
whether to send a DSN or not. This is a bug, as recipients that time out
are not taken into account.
This patch fixes that issue by including both failed and pending
recipients in the DSN.
It also adds more comprehensive tests for this case, both in the queue
and in the dsn generation code.
The default INFO logs are more oriented towards debugging and can be
a bit too verbose when looking for high-level information.
This patch introduces a new "maillog" package, used to log messages of
particular relevance to mail transmission at a higher level.
HELO and EHLO both take a mandatory parameter, which also should be used
in the Received header.
This patch tracks and enforces that parameter, and also updates the
Received header generation to use it.
https://tools.ietf.org/html/rfc5321#section-4.4
Currently, we do SPF checks for all connections.
However, authenticated users will be sending email from different
locations, applying SPF to them will result in false positives.
So this patch makes chasquid skip SPF checking if the connection is
authenticated.
Having the certificates inside the domain directory may cause some confusion,
as it's possible they're not for the same name (they should be for the MX we
serve as, not the domain itself).
So it's not a problem if we have domains with no certificates (we could be
their MX with another name), and we could have more than one certificate per
"domain" (if we act as MXs with different names).
So this patch moves the certificates out of the domains into a new certs/
directory, where we do a one-level deep lookup for the files.
While at it, change the names of the files to "fullchain.pem" and
"privkey.pem", which match the names generated by the letsencrypt client, to
make it easier to set up. There's no general convention for these names
anyway.
This patch adds a test for delivery status notifications and null address
deliveries, that check that chasquid can both receive and send DSNs.
To do this, we extend the mail_diff utility to support wildcards in the
comparisons, to skip over variable parts of the messages (like dates).
