In the loop integration test, we detect looping via checking the expvars
of chasquid, and waiting for the loop counter to be 1.
However, if chasquid is fast enough, it will go up to 2 before the
detection notices. This is because the DSN that gets generated also
loops (as expected).
Despite its loose appearance, the "Received" header has a reasonably
standarized format.
We were not following the standard format as closely as we should; this
rarely causes problems in this particular case, but there's no need to
deviate from it.
This patch changes the Received header generation as follows:
- The "from" section now uses the remote address as canonical (for
non-authenticated users) which provides more valuable information
than the user-supplied EHLO address (which is also included).
- The remote authenticated user is now hidden, for additional privacy.
- Use the "with" optional clause.
- Use the standard way of printing TLS cipher suite.
- Use the standard way of printing address literals.
Docker creates intermediate layers on each command (for most commands),
but the COPY was invalidating them too early, every time it runs it was
generating a different layer.
This patch moves the COPY down to the bottom, and adds a bit more
organization to the commands below.
This patch adds DKIM signing using https://github.com/driusan/dkim tools
to the example hook.
It also adds an optional integration test to exercise signing and
verification, and corresponding documentation.
The current dovecot config for integration test t-11-dovecot is not
compatible with dovecot 2.3. There are some new services which want to
change the group owner for some files to the default, and that does not
work when run as non-root.
The errors look like:
master: Error: service(stats): chown(/tmp/chasquid-dovecot-test/run/stats-writer, 4294967295, 127) failed: Operation not permitted
master: Error: service(imap-hibernate): chown(/tmp/chasquid-dovecot-test/run/imap-hibernate, 4294967295, 127) failed: Operation not permitted
master: Error: service(dict): chown(/tmp/chasquid-dovecot-test/run/dict, 4294967295, 127) failed: Operation not permitted
master: Error: service(dict-async): chown(/tmp/chasquid-dovecot-test/run/dict-async, 4294967295, 127) failed: Operation not permitted
master: Fatal: Failed to start listeners
(127 is the "dovecot" user)
So this patch adds some config settings to set the group manually for
these services, which is backwards compatible with 2.2.
Eventually we will stop supporting 2.2 for tests, at which point we can
change to just setting default_internal_group.
This patch adds more tests for the dovecot library, in particular:
- Protocol errors (invalid versions, etc.).
- Invalid command (cli-specific test).
- Connection breakups.
Many areas of the dovecot library are tested via chamuyero scripts, but
these were not being included in the coverage report.
This patch extends the dovecot-auth-cli tests so that they are now
coverage-aware.
This patch adds a new test, which verifies the TLS tracking.
Because we need to simulate SPF records, and Go does not support fully
intercepting DNS lookups yet, this test relies on dnsmasq to provide a
DNS resolver.
In the future, once Go supports DNS lookup interception, we can get rid
of this additional dependency.
SIGTERM can happen normally in our tests, and the current code has a
trap that makes it trigger a clean exit.
This causes some errors to be masked, as we end up calling "exit 0" when
they occur. The error message will still be displayed, but the caller
script will assume it worked.
This patch fixes the problem by adjusting the bash signal handlers, so
that we ignore SIGTERM (so bash does not get killed by the exit handler)
and exit with error on SIGINT (triggered by ctrl-c).
Note that under some conditions the SIGTERM trap is not necessary, but
this depends on the environment.
This patch adds a Docker configuration file, to build an image that can
run the integration and stress tests.
It might be used for configuring automated testing infrastructure in
subsequent patches.
Our chamuyero tests involve reading and writing utf8. This usually
works, but is dependent on the environment: on LC_ALL=POSIX environment,
for example, Python enforces ascii as the default encoding, and the
tests break.
So this patch makes chamuyero explicitly set utf8 encodings in stdout
and all the sockets.
This patch introduces stress tests, which put load on the daemon to help
sanity check its behaviour under stress.
They are separate from the existing integration tests, which focus on
correctness.
Two tests are included here: a load test, which sends emails repeatedly;
and a connection test which opens as many conections as possible.
This patch adds HTTP fetching to the integration tests.
It checks that the URLs are properly exported and that the server
replies reasonably to them. The contents are saved as they might be
useful as a debugging aid.
They're added to t-09-loop as it already was doing other HTTP fetches,
but the changes are not particularly tied to it.
The content of the pages is not checked yet, that might come in
subsequent patches.
The test to check that chasquid fails on startup if there are no valid
TLS certificates is passing, but for the wrong reasons: it fails because
there is no logging directory, not because there are no certificates.
This patch fixes the problem by moving the logs directory creation
before the first test.
This patch adds a new integration test, which executes various small
dialogs, to cover corner cases that are not well covered (according to
our coverage report).
For example, "EHLO" without domain, or invalid DATA.
While we could do them via Go tests, this way is more realistic, and the
tests are easier to write.
This patch adds some tooling and scripts to generate test coverage
information.
Unfortunately, this involves some hacks as Go does not have support for
generating coverage-enabled binaries, or merging coverage reports; but
overall it's not very intrusive.
This patch extends various packages and integration tests, increasing
test coverage. They're small enough that it's not worth splitting them
up, as it would add a lot of noise to the history.
Dovecot has options for changing the formatting of usernames; for
example, dropping the domain part, or replacing characters.
chasquid's implementation, however, fails to handle this well, as it
expects the reply to contain the username exactly as requested.
This patch fixes the problem by making chasquid ignoring the returned
username, which is unused anyway. The protocol is unambiguous enough.
Tests are also amended to always exercise this case.
This patch adds dovecot support to the chasquid daemon, using the
internal dovecot library added in previous patches.
Dovecot support is still considered EXPERIMENTAL and may be reverted, or
changed in backwards-incompatible ways.
The patch also adds the corresponding integration test, which brings up
a dovecot server with a custom configuration, and tests chasquid's
authentication against it. If dovecot is not installed, the test is
skipped.
In Go 1.10 the TLS library will start to reject DNS SANs which are not
properly formed; and in particular, if they're not IDNA-encoded. See:
- https://github.com/golang/go/issues/15196
- 9e76ce7070
The generate_cert utility will write non-IDNA DNS SANs, which the TLS
library does not like, causing our idna tests to fail.
This patch fixes this incompatibility by making generate_cert IDNA-encode
the host names when adding them to the certificate.
There's a bug in wget where it logs to files if -q is used:
https://savannah.gnu.org/bugs/?51181
It is harmless to the test, but it clutters the directory and the test
output, so this patch works around the issue by forcing it to log to
/dev/null.
The test starts by removing the previous test certificates, which may
or may not exist.
If they don't, currently "rm" fails, causing the whole test to fail.
I am surprised I did not notice this before :(
This patch fixes the bug by using "rm -f" to remove the test certs.
The outgoing security level checks are not being performed, because of a
bug: the courier thinks the "to"'s domain is always empty.
This patch fixes the bug by simplifying the logic, as there's no need
for the conditional (there is always a domain in the "to" address if it
got to the SMTP courier).
The nc.py script is only used in a single test, and for waiting for a
TCP port to be opened for listening.
This patch replaces it entirely, by using chamuyero for the test, and
bash for waiting on a TCP port.
This patch adds "chamuyero", a a tool to test and validate line-oriented
commands and servers.
It can launch and communicate with other processes, and follow a script of
line-oriented request-response, validating the dialog as it goes along.
This can be used to test line-oriented network protocols (such as SMTP) or
interactive command-line tools.
It will be used in follow up patches to test new commands and
functionality.
Currently, chasquid exits if any mode (SMTP/submission/submission+tls)
has no addresses to listen on. This means that chasquid must be given
addresses for all three.
While that's generally the expected configuration, there are cases where
users may not want to have all three.
So this patch replaces that fatal error with a warning, and only makes
chasquid exit if there are no addresses to listen on at all.
This patch adds support for TLS-wrapped submission connections.
Instead of clients establishing a connection over plain text and then
using STARTTLS to switch over a TLS connection, this new mode allows the
clients to connect directly over TLS, like it's done in HTTPS.
This is not an official standard yet, but it's reasonably common in
practice, and provides some advantages over the traditional submission
port.
The default port is 465, commonly used for this; chasquid defaults to
systemd file descriptor passing as for the other protocols (for now).
Netcat's behaviour after seeing EOF from stdin seems to not be very
portable or consistent, even under the same platform.
This has caused t-05-null_address to break recently under some
conditions, for example depending on the particular Debian version of
netcat-openbsd used, and the current situation is unclear.
See https://bugs.debian.org/854292 and https://bugs.debian.org/849192
for more details.
To stop depending on this brittle behaviour, this patch unfortunately
introduces a simple python3-based netcat for our tests to use.
The server is written assuming there's at least one valid SSL/TLS
certificate. For example, it unconditionally advertises STARTTLS, and
only supports AUTH over TLS.
This patch makes the server fail to listen if there are no certificates
configured, so the users don't accidentally run an unsupported
configuration.
The loop test can be quite slow, specially on computers without
cryptography-friendly instructions.
This patch introduces a new flag for testing, so that we can bring the
threshold down to 5. The test is just as useful but now runs in a few
seconds, as opposed to a few minutes.
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
This patch makes the hooks always have a complete set of environment
varuables, set to 0/1 or whatever is appropriate, to make it easier to
write the checks for them.
The queue currently only considers failed recipients when deciding
whether to send a DSN or not. This is a bug, as recipients that time out
are not taken into account.
This patch fixes that issue by including both failed and pending
recipients in the DSN.
It also adds more comprehensive tests for this case, both in the queue
and in the dsn generation code.
This patch changes chasquid-util's subcommands and parameters to
(hopefully) make them more user friendly and intuitive by default.
The changes include defaulting the configuration to /etc/chasquid, and
using full addresses as usernames.
It also adds some shell tests to cover most of the functionality.
The default INFO logs are more oriented towards debugging and can be
a bit too verbose when looking for high-level information.
This patch introduces a new "maillog" package, used to log messages of
particular relevance to mail transmission at a higher level.
This patch implements a post-DATA hook, which is run after receiving the
data but before sending a reply.
It can be used to implement content filtering when receiving email, for
example for passing the email through an anti-spam or an anti-virus.
HELO and EHLO both take a mandatory parameter, which also should be used
in the Received header.
This patch tracks and enforces that parameter, and also updates the
Received header generation to use it.
https://tools.ietf.org/html/rfc5321#section-4.4
We should ignore the domains' case, and treat them uniformly, specially when it
comes to local domains.
This patch extends the existing normalization (IDNA, keeping domains as
UTF8 internally) to include case conversion and NFC form for
consistency.
Currently, we do SPF checks for all connections.
However, authenticated users will be sending email from different
locations, applying SPF to them will result in false positives.
So this patch makes chasquid skip SPF checking if the connection is
authenticated.
This patch implements some measures against email loops, such as keeping
a limit on the lenght of an address, and rejecting email that has too
many Received headers.
It's not perfect (a server could be actively removing Received headers),
but it should cover the normal accidents and misconfigurations.
If there's an alias to forward email to a non-local domain, using the original
From is problematic, as we may not be an authorized sender for it.
Some MTAs (like Exim) will do it anyway, others (like gmail) will construct a
special address based on the original address.
This patch implements the latter approach, which is safer and allows the
receiver to properly enforce SPF.
We construct a (hopefully) reasonable From based on the local user, and
embedding the original From (but transformed for IDNA, as the receiver may not
support SMTPUTF8).
This patch adds initial support for SMTPUTF8, which for now consists of just
advertising it.
We support most of it, but sending emails over SMTP requires further work, as
the SMTP courier does not support this yet (it's not in Go's standard
library). That will come in subsequent patches, along with IDNA handling.
https://tools.ietf.org/html/rfc6531.html
