This patch groups a series of commands that redirect to a file into a
single grouped redirect, for readability reasons.
Suggested by shellcheck SC2129.
staticcheck found a couple of minor code cleanup improvements, like
unused variables or an out-of-order defer, mostly in tests.
This patch fixes those problems by making the necessary adjustments.
They're all fairly small, and should not change the logic in any
significant way.
There are a couple of places in the tests when we attempt to build and
run simultaneously. Here, if the build is slow, there is a race where
"text file busy" can appear.
To fix this, build to a temporary file with a random name, then
atomically rename it to the final binary name.
Dovecot 2.4 has a new configuration format, which is unfortunately
backwards-incompatible with Dovecot 2.3.
This patch adds a 2.4-compatible config, and selects which one to use
based on the Dovecot version in the environment.
In the future, once 2.4 becomes more common, we will drop the 2.3 config
from the test.
Note that we don't change the config used in the Docker image, because
that is based on Debian **stable** which is still on 2.3.
This patch implements "via" aliases, which let us explicitly select a
server to use for delivery.
This feature is useful in different scenarios, such as a secondary MX
server that forwards all incoming email to a primary.
For now, it is experimental and the syntax and semantics are subject to
change.
We have a few Python scripts which over the years ended up with a
variety of formatting.
This patch auto-formats them using `black -l 79` to make them more
uniform, and easier to read and write.
minidns supports MX records, but today it hard-codes priority=10.
This is limiting when creating test scenarios that depend on having
different MX priorities.
This patch adds support for specifying the priority in MX records.
Today, aliases parsing is too lax, silently ignoring most kinds of invalid
lines.
That behaviour can cause a lot of confusion when users think the aliases
are being parsed, and also cause problems when extending the syntax.
This patch fixes that problem by making aliases parsing return errors on
the invalid lines.
Unfortunately this will cause some previously-accepted files to now be
rejected, but it should be quite visible.
ssl.wrap_socket() has been deprecated and is no longer functional in
Python 3.12: https://docs.python.org/3/whatsnew/3.12.html#ssl.
This patch replaces it with the equivalent (in this context)
ssl.SSLContext.
Today, when starting up, if there's an error reading the users or
aliases files, we only log but do not exit. And then those files will
not be attempted to be read on the periodic reload.
We also treat "file does not exist" as an error for users file, but not
aliases file, resulting in inconsistent behaviour between the two.
All of this makes some classes of problems (like permission errors) more
difficult to spot and troubleshoot. For example,
https://github.com/albertito/chasquid/issues/55.
So this patch makes errors reading users/aliases files on startup a
fatal error, and also unifies the "file does not exist" behaviour to
make it not an error in both cases.
Note that the behaviour on the periodic reload is unchanged: treat these
errors as fatal too. This may be changed in future patches.
Unfortunately, `go get` rejects repos that have files with ':':
https://github.com/golang/go/issues/28001.
We have one such file in the tests.
This prevents some of the Go tooling from working on the latest release,
including pkg.go.dev.
So, as a workaround we use a compatible file name in the repository, and
rename it when running the test. This is very hacky, but it's okay for a
single test.
This patch adds a cross-tool integration check that uses
driusan/dkim's dkimverify to confirm it can verify our own DKIM signatures.
It is optional, since the tool may not be present.
This patch removes the integration tests that covered using driusan/dkim
and dkimpy's tools in the example hook.
Now that we have internal DKIM support, the example hook doesn't attempt
to use them, so we can remove the tests that cover it.
Those tools, and other DKIM implementations, can still be used in the
post-data hook just as before.
To send mails, today some tests use msmtp and others our internal smtpc.py.
This works, but msmtp slows down the tests significantly, and smtpc.py
is also not particularly fast, and also has some limitations.
This patch introduces a new SMTP client tool written in Go, and makes
almost all the tests use it.
Some tests still remain on msmtp, mainly for client-check compatibility.
It's likely that this will be moved in later patches to a separate
special-purpose test.
With this patch, integration tests take ~20% less time than before.
Dovecot applies an authentication penalty, where it delays failed attempts.
Because we intentionally do bad authentications for testing, this slows
downs the tests significantly. So this patch disables it.
Our tests invoke a variety of helpers, some of them are written in Go.
Today, we call "go build" (directly or indirectly via "go run"), which is
a bit wasteful and slows down the tests.
This patch makes the tests only build our Go helpers once every 10s at
most.
The solution is a bit hacky but in the context of these tests, it's
practical.
The generate_cert cache has a bug because it uses the directory's age,
which won't necessarily change, and it was always re-generating
certificates after 10m.
This patch fixes the bug by checking the age of the private key file
instead of the directory.
The RFCs are very clear that in DATA contents:
> CR and LF MUST only occur together as CRLF; they MUST NOT appear
> independently in the body.
https://www.rfc-editor.org/rfc/rfc5322#section-2.3https://www.rfc-editor.org/rfc/rfc5321#section-2.3.8
Allowing "independent" CR and LF can cause a number of problems.
In particular, there is a new "SMTP smuggling attack" published recently
that involves the server incorrectly parsing the end of DATA marker
`\r\n.\r\n`, which an attacker can exploit to impersonate a server when
email is transmitted server-to-server.
https://www.postfix.org/smtp-smuggling.htmlhttps://sec-consult.com/blog/detail/smtp-smuggling-spoofing-e-mails-worldwide/
Currently, chasquid is vulnerable to this attack, because Go's standard
libraries net/textproto and net/mail do not enforce CRLF strictly.
This patch fixes the problem by introducing a new "dot reader" function
that strictly enforces CRLF when reading dot-terminated data, used in
the DATA input processing.
When an invalid newline terminator is found, the connection is aborted
immediately because we cannot safely recover from that state.
We still keep the internal representation as LF-terminated for
convenience and simplicity.
However, the MDA courier is changed to pass CRLF-terminated lines, since
that is an external program which could be strict when receiving email
messages.
See https://github.com/albertito/chasquid/issues/47 for more details and
discussion.
This patch makes mail_diff more strict in its parsing, to ensure we
catch any encoding issues that may otherwise be masked by the default
compatibility policy.
The minor dialogs test covers some very specific SMTP exchanges, and
some of those include delivering email.
Today, we don't verify the final mailbox, we just check the SMTP
exchange. However, it can be very useful for some of the tests to do
end-to-end checking of the final mailbox.
This patch implements that ability in the test itself, and on the
(currently only) email delivering dialog.
Subsequent patches that introduce new tests will make use of this
feature.
Using an empty listening address will result in chasquid listening on a
random port, which is a dangerous misconfiguration.
That is most likely done to prevent it from listening at all.
To prevent this misconfiguration, explicitly reject empty listening
addresses early and with a warning, so there is no ambiguity.
Users can still prevent chasquid from listening by just commenting out
the entry in the config (and not passing any systemd file descriptors).
See https://github.com/albertito/chasquid/issues/45 for more details and
discussion, including alternatives considered.
Thanks to xavierg who reported this via IRC.
Today, when a user sets an alias with drop characters and/or suffixes,
those go unused, since we always "clean" addresses before alias
resolution.
This results in unexpected and surprising behaviour, and it's not
properly documented either.
This patch resolves this unexpected behaviour as follows:
- Drop characters are ignored, both at parsing time and at lookup time.
- Lookups are done including the suffixes first, and if that results in
no matches, they are retried without suffixes.
This results in aliases working more intuitively for the most common use
cases: of users wanting to have different aliases for specific suffixes,
and not having to care for drop characters.
Hooks can be used to get different behaviour if needed, since the first
lookup is done with the address as-is.
Thanks to znerol@ (lo+github@znerol.ch) for reporting this, and the
discussion on how to fix it, in
https://github.com/albertito/chasquid/issues/41.
Today we check the aliases deliver mail to the expected locations, but
we don't fail if there are unexpected deliveries.
Doing so can help catch bugs (including test bugs), so this patch
implements that.
In addition, fix two of the tests that were printing on error, but not
causing the tests to fail (which was the original intention).
The t-20-bad_configs test can sometimes have false positives because
the last line recorded in the log isn't always the one causing the
failure, due to asynchronous logging and in particular in combination
with coverage tests (which alter the os.Exit behaviour subtly).
This patch fixes that by having the tests look at the last 4 lines of
output instead. This is not super pretty, but it should be good enough
to cover for any timing issues that arise in these particular tests.
Currently, if the `certs/` directory has a symlink inside, we skip it.
That is not really intended, it's an unfortunate side-effect of skipping
regular files.
To fix this, this patch adjusts the logic to only ignore regular files
instead. It also adds a message when a directory is skipped, to make it
easier to debug permission issues.
Thanks to @erjoalgo for reporting this in
https://github.com/albertito/chasquid/pull/39, and providing an
alternative patch!
This patch makes chasquid-util's aliases-resolve and domaininfo-remove
commands talk to the chasquid server (via the new localrpc server).
For aliases-resolve, currently has fairly hacky logic which reimplements
a bunch of the servers', and is also incomplete because it does not
support hooks.
In this patch we fix that by having it talk to the server, where we get
authoritative responses and have no issues with aliases hooks. This
resolves https://github.com/albertito/chasquid/issues/18.
For domaininfo-remove, currently its implementation is also very hacky
since it manipulates files behind the servers' back and without even
using the internal library.
In this patch we fix that by doing the operation through the server,
avoiding the need for those hacks, and also remove the need to manually
reload the server afterwards.
This patch updates tests.md to reflect the recent changes around
coverage testing, specifically we no longer have coverage issues with
fatal/non-zero exits, so remove that section from the docs.
Also while at it, add links to the software we reference, for
convenience.
Go 1.20 finally includes proper support for instrumenting binaries for
coverage. This allows us to drop quite a few hacks and workarounds that
we used for it, and we can now also test exiting cases.
The downside is that coverage tests now require Go 1.20, but it is an
acceptable price to pay for the more accurate results.
Normal integration tests are unchanged.
This patch updates the coverage testing infrastructure to make use of
the new Go 1.20 features.
We're running against the usage limits in Gitlab CI (500), and Github
Actions should have more (2000).
So this patch replaces Gitlab CI with Github actions for running
integration tests, and build and push Docker images (to Dockerhub and
Gitlab registry).
We'll see how the usage levels are in a few months.
This patch updates the shell scripts with some of the common best
practices, which should make them more resilient to unusual failures and
unexpected environments (in particular, directories with spaces).
Most of these were identified by shellcheck.
The integration tests spend a lot of time on some ancilliary actions,
which slows them down: generating certificates, adding users, and
waiting for things to happen.
To improve the performance of those actions, this patch:
- Makes (most) tests use plain passwords (-20%)
- Adds a certificate cache to reuse certs (-10%)
- Tightens the sleep loops (-5%)
In aggregate, this patch results in a speedup of the integration tests
of ~30-40%.
Note that some of the tests required adjusting the username, because
`chasquid-util user-add` would convert them to lowercase as per PRECIS
mapping.
This patch changes several internal packages to receive and pass tracing
annotations, making use of the new tracing library, so we can have
better debugging information.
ioutil package was deprecated in Go 1.16, replace all uses with their
respective replacements.
This patch was generated with a combination of `gofmt -r`, `eg`, and
manually (for `ioutil.ReadDir`).
Newer versions of msmtp now set In-Reply-To and References header,
causing t-16-spf test to fail because we expect them to be empty (for no
particular reason).
So this patch changes our expected DSN used for testing to ignore their
values.
By default, `git describe` uses only annotated tags. Since some may not
be annotated (like v1.10) this causes some scripts to pick a confusing
version identifier.
This patch fixes the issue by passing `--tags`, which means all tags
will be considered.
This patch is the result of running Go 1.19's `gofmt` on the codebase,
which automatically updates all Go doc comments to the new format.
https://tip.golang.org/doc/go1.19#go-doc
When running a diff for dkimpy's output, we expect that diff to exit with
non-zero code.
Unfortunately, the way we set that expectation (by prefixing the diff
invocation with `!` is incorrect.
Running `! diff ...` will not cause the hook to fail if diff exits with
0, instead `!` will cause the exit code to be ignored.
This patch fixes the problem by running `diff ... && exit 1` instead.
This was caught by shellcheck, https://www.shellcheck.net/wiki/SC2251.
We've accumulated a few linter issues around comments and a couple of
variable names.
While none of them is major, this patch cleans them up so it's easier to
go through the linter output, and we can start being more strict about
it.
The current generate_cert helper was originally taken from Go's source,
and is more complex than we need it to be.
This patch replaces it with our own version, rewritten from scratch
independently.
This patch moves the test helper binaries to a "one directory per
helper" layout, and also makes them to be ignored in the coverage build
instead of all builds.
With this change, "go build ./..." will build all binaries including the
test helpers, which helps make sure that module manage automation also
considers them. In particular, this makes "go mod tidy" work fine.
Dovecot's `state_dir` usually defaults to be at `/var/lib/dovecot`, or a
similar system-wide path.
Under some conditions, our test Dovecot instance can fail, because it's
wanting to write to state_dir, but it is not writeable by us in the test
environment.
This was reported by foxcpp in
https://github.com/albertito/chasquid/issues/28.
This patch fixes the problem by setting a custom state_dir to be within
our testing directory.
Thanks to foxcpp for reporting this problem and suggesting a fix.
