To send mails, today some tests use msmtp and others our internal smtpc.py.
This works, but msmtp slows down the tests significantly, and smtpc.py
is also not particularly fast, and also has some limitations.
This patch introduces a new SMTP client tool written in Go, and makes
almost all the tests use it.
Some tests still remain on msmtp, mainly for client-check compatibility.
It's likely that this will be moved in later patches to a separate
special-purpose test.
With this patch, integration tests take ~20% less time than before.
This patch updates the shell scripts with some of the common best
practices, which should make them more resilient to unusual failures and
unexpected environments (in particular, directories with spaces).
Most of these were identified by shellcheck.
When resolving MX records, we need to distinguish between "no such
domain" and other kinds of errors. Before Go 1.13, this was not
possible, so we had a workaround that assumed any permanent error was a
"no such domain", which is not great, but functional.
Now that our minimum supported version is Go 1.15, we can remove the
workaround.
This patch replaces the workaround with proper logic using
DNSError.IsNotFound to identify NXDOMAIN results when resolving MX
records.
This requires to adjust a few tests, that used to work on environments
where resolving unknown domains (used for testing) returned a permanent
error, and now they no longer do so. Instead of relying on this
environmental property, we make the affected tests use our own DNS
server, which should make them more hermetic and reproducible.
Most integration tests depend on the $HOSTALIASES environment variable
being functional. That variable works on most systems, but not all. In
particular, systems with `systemd-resolved` can cause the variable to be
ignored.
This was reported by Alex Ellwein in
https://github.com/albertito/chasquid/issues/20.
This patch makes the affected tests to be skipped if $HOSTALIASES is not
working properly. It also removes unnecessary hosts files from tests
which don't need it, and documents this behaviour.
Thanks to Alex Ellwein and foxcpp@ for reporting and helping investigate
this issue!
This patch adds support for TLS-wrapped submission connections.
Instead of clients establishing a connection over plain text and then
using STARTTLS to switch over a TLS connection, this new mode allows the
clients to connect directly over TLS, like it's done in HTTPS.
This is not an official standard yet, but it's reasonably common in
practice, and provides some advantages over the traditional submission
port.
The default port is 465, commonly used for this; chasquid defaults to
systemd file descriptor passing as for the other protocols (for now).
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
The default INFO logs are more oriented towards debugging and can be
a bit too verbose when looking for high-level information.
This patch introduces a new "maillog" package, used to log messages of
particular relevance to mail transmission at a higher level.
Make the SMTP courier fall back to the A record when MX does not exist, as per
standard behaviour.
This is not implemented nicely, because Go's API does not give a clear signal
if the answer was that there are no MX records or something else happens.
For now, we implement it with a heuristic that should work pretty reliably,
but it's definitely not very nice.
Having the certificates inside the domain directory may cause some confusion,
as it's possible they're not for the same name (they should be for the MX we
serve as, not the domain itself).
So it's not a problem if we have domains with no certificates (we could be
their MX with another name), and we could have more than one certificate per
"domain" (if we act as MXs with different names).
So this patch moves the certificates out of the domains into a new certs/
directory, where we do a one-level deep lookup for the files.
While at it, change the names of the files to "fullchain.pem" and
"privkey.pem", which match the names generated by the letsencrypt client, to
make it easier to set up. There's no general convention for these names
anyway.
It's more convenient and in line with standard practice to fail RCPT TO if the
user does not exist.
This involves making the server and client aware of aliases, but it doesn't
end up being very convoluted, and simplifies other code.
This patch tidies up the Procmail courier:
- Move the configuration options to the courier instance, instead of using
global variables.
- Implement more useful string replacement options.
- Use exec.CommandContext for running the command with a timeout.
As a consequence of the first item, the queue now takes the couriers via its
constructor.
This patch adds a "data_dir" option, that chasquid will use to store
persistent data. It defaults to "/var/lib/chasquid" (for now, at least).
Users will come in subsequent patches.
This patch adds checks that verify:
- The envelope from must match the authenticated user. This prevents
impersonation at the envelope level (while still allowing bounces, of
course).
- If the destination is remote, then the user must have completed
authentication. This prevents unauthorized relaying.
The patch ends up adjusting quite a few tests, as they were not written
considering these restrictions so they have to be changed accordingly.
We want to be able to distinguish between connections for SMTP and connections
for submission, so we can make different policy decisions.
To do that, we first make the configuration aware of the different kinds of
addresses. This is done in this patch in a backwards-incompatible way, but at
this point in time it is ok to do so.
Then, we extend systemd's socket passing library to support socket naming, so
we can tell the different sockets apart. This is done via the
LISTEN_FDNAMES/FileDescriptorName mechanism.
And finally we make the server and connection types aware of the socket mode.
