The loop test can be quite slow, specially on computers without
cryptography-friendly instructions.
This patch introduces a new flag for testing, so that we can bring the
threshold down to 5. The test is just as useful but now runs in a few
seconds, as opposed to a few minutes.
The queue IDs are internal to chasquid, and we don't need them to be
cryptographically secure, so this patch changes the ID generation to use
the PRNG.
This also helps avoid entropy issues.
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
This patch makes the hooks always have a complete set of environment
varuables, set to 0/1 or whatever is appropriate, to make it easier to
write the checks for them.
It is can be convenient for hooks to indicate that an error is
permanent; for example if the anti-virus found something.
This patch makes it so that if the hook exits with code 20, then it's
considered permanent. Otherwise it is considered transient, to help
prevent accidental errors cause final delivery issues.
The "ptr" mechanism should not be used, but unfortunately many domains
still do.
This patch extends our SPF implementation to support checking it.
https://tools.ietf.org/html/rfc7208#section-5.5
Calculating the next delay based on the previous delay causes daemon
restarts to start from scratch, as we don't persist it.
This can cause a few server restarts to generate many unnecessary sends.
This patch changes the next delay calculation to use the creation time
instead, and also adds a <=1m random perturbation to avoid all queued
emails to be retried at the exact same time after a restart.
TestAliases is unfortunately racy, and cause occasional failures.
This patch rewrites it to be more end-to-end, similar to TestBasic,
which should remove the races while keeping the main objectives of the
test.
The queue currently only considers failed recipients when deciding
whether to send a DSN or not. This is a bug, as recipients that time out
are not taken into account.
This patch fixes that issue by including both failed and pending
recipients in the DSN.
It also adds more comprehensive tests for this case, both in the queue
and in the dsn generation code.
The default INFO logs are more oriented towards debugging and can be
a bit too verbose when looking for high-level information.
This patch introduces a new "maillog" package, used to log messages of
particular relevance to mail transmission at a higher level.
This patch makes safeio preserve file ownership. This is specially
useful when using command-line utilities as root, but the files they
change are owned by a different user.
This patch implements a post-DATA hook, which is run after receiving the
data but before sending a reply.
It can be used to implement content filtering when receiving email, for
example for passing the email through an anti-spam or an anti-virus.
Unknown commands can fill the logs, traces and expvars with a lot of
noise; this patch sanitizes them a bit down to 6 bytes, as a compromise
to maintain some information for troubleshooting.
The submission port is expected to be used only by authenticated
clients, so this patch makes chasquid enforce this, which also helps
to reduce spam.
https://www.rfc-editor.org/rfc/rfc6409.txt
Today, we pick the domain used to send the DSN from based on what we
presented to the client at EHLO time, which itself may be based on the
TLS negotiation (which is not necessarily trusted).
This is complex, not necessarily correct, and involves passing the
domain around through the queue and persisting it in the items.
So this patch simplifies that handling by always using the main domain
as specified by the configuration.
This patch moves chasquid's Server and Conn structures to their own
smtpsrv package, to make chasquid.go a bit more readable. It also helps
clarify the relation between Server and Conn.
There are no functional changes.
Note that git can still track the history across this commit (e.g. git
gui blame shows the right data).
This patch introduces a new "domaininfo" package, which implements a
database with information about domains. In particular, it tracks
incoming and outgoing security levels.
That information is used in incoming and outgoing SMTP to prevent
downgrades.
This patch is the result of running go vet, go fmt -s and the linter,
and fixing some of the things they noted/suggested.
There shouldn't be any significant logic changes, it's mostly
readability improvements.
Errors can contain newlines, in particular this is common in messages
returned by remote SMTP servers.
This patch quotes them before logging, they don't interfere with the
log. Note the tracer itself can handle this just fine.
This patch simplifies the sending loop code:
- Move the recipient sending function from a closure to a method.
- Simplify the status update logic: we now update and write
unconditionally (as we should have been doing).
- Create a function for counting recipients in a given status.
It also adds a test for the removal of completed items from the queue,
which was not covered before and came up during development.
We should ignore the domains' case, and treat them uniformly, specially when it
comes to local domains.
This patch extends the existing normalization (IDNA, keeping domains as
UTF8 internally) to include case conversion and NFC form for
consistency.
This patch implements local username normalization using PRECIS
(https://tools.ietf.org/html/rfc7564,
https://tools.ietf.org/html/rfc7613)
It makes chasquid accept local email and authentication regardless of
the case. It covers both userdb and aliases.
Note that non-local usernames remain untouched.
The default for max events is 10, which is a bit short for a normal SMTP
exchange. Expand it to 30 which should be large enough to keep most of
the traces.
We don't want to write debug messages to the log, but having them in
traces is always useful.
The traces are volatile and self-cleaning so the additional volume
should not cause any problems, and helps troubleshooting.
While at it, fix the depth constant when logging.
The logic that picks the domain we use for EHLO does not need to live
within the TLS retry cycle, and makes it harder to understand.
This patch moves it out of the way, to improve readability.
This patch introduces expvar counters to chasquid and the queue
packages.
For now there's only a handful of counters, but they will be expanded in
future patches.
The test courier has a racy map access, and this started to manifest
after some recent changes. This patch fixes the race by implementing the
corresponding locks.
The SMTP courier was not properly closing the connection, and chasquid's
closing of incoming connections was not ideal (it was closing the
underlying one, not necessarily the active one, like in the case of a jump
to TLS).
This patch fixes both by adding the missing calls to Close.
This patch reviews various debug and informational messages, making more
uniform use of tracing, and extends the monitoring http server with
useful information like an index and a queue dump.
This patch adds a package for evaluating SPF, as defined by RFC 7208
(https://tools.ietf.org/html/rfc7208).
It doesn't implement 100% of the RFC, but it coves enough to handle the
most common cases, and will fail open on the others.
If there's an alias to forward email to a non-local domain, using the original
From is problematic, as we may not be an authorized sender for it.
Some MTAs (like Exim) will do it anyway, others (like gmail) will construct a
special address based on the original address.
This patch implements the latter approach, which is safer and allows the
receiver to properly enforce SPF.
We construct a (hopefully) reasonable From based on the local user, and
embedding the original From (but transformed for IDNA, as the receiver may not
support SMTPUTF8).
In the SMTP courier, we should always include a domain when saying hello, as
many MTAs will be pick about it.
An empty domain can happen if the envelope-from is <>. In that case, we fall
back to our hostname.
This patch makes the SMTP courier distinguish permanent errors, so the queue
can decide whether to retry or not.
It's based on the SMTP replies: 5xy is considerent permanent, anything else is
transient (including errors other than protocol issues).
This patch adds SMTPUTF8 to the SMTP courier.
It introduces a new internal/smtp package that extends Go's net/smtp with
SMTPUTF8 (in a very narrow way, not for general use), and then makes the
courier use it.
Also use an IDNA-safe version when saying hello, otherwise servers could
complain if the hostname is not ASCII, and at that stage we don't know if they
support it or not.
This patch implements the first steps of support for IDNA (Internationalized
Domain Names for Applications).
Internally, we maintain the byte-agnostic representation, including
configuration.
We support receiving IDNA mail over SMTP, which we convert to unicode for
internal handling.
Local deliveries are still kept agnostic.
For sending over SMTP, we use IDNA for DNS resolution, but there are some
corner cases pending in the SMTP courier that are tied with SMTPUTF8 and will
be fixed in subsequent patches.
Make the SMTP courier fall back to the A record when MX does not exist, as per
standard behaviour.
This is not implemented nicely, because Go's API does not give a clear signal
if the answer was that there are no MX records or something else happens.
For now, we implement it with a heuristic that should work pretty reliably,
but it's definitely not very nice.
Some servers, like postfix, will pay close attention to the domain we say as a
part of the EHLO.
By default, Go's smtp package will use "localhost", causing it to complain.
This patch fixes that by using the envelope-from's domain.
It's not clear if that's better than using what we are resolving to, but
that's much more involved so we're going to do this for now.
There are a couple of places where it's handy to print TLS constants in
human-readable form.
To do so, we need functions that take TLS constants (usually in uint16 form)
and give us friendly strings.
Go's crypto/tls package does not provide us with this, so this patch
introduces a new module with that purpose.
This patch performs some minor cleanups for things detected by "go vet":
- Remove one line of unreachable code.
- Don't leak contexts until their deadline expires, cancel them.
It's more convenient and in line with standard practice to fail RCPT TO if the
user does not exist.
This involves making the server and client aware of aliases, but it doesn't
end up being very convoluted, and simplifies other code.
Today, if the aliases file does not exist when chasquid starts up, the entire
domain will be skipped from aliases resolution.
That's a bug, as it means we don't perform character and suffix replacements
for known domains, and is also an inconvenience as it forces us to reload the
daemon when adding a file for a known domain.
This patch fixes this by adding them unconditionally, even if the file does
not exist.
When we permanently failed to deliver to one or more recipients, send delivery
status notifications back to the sender.
To do this, we need to extend a couple of internal structures, to keep track
of the original destinations (so we can include them in the message, for
reference), and the hostname we're identifying ourselves as (this is arguable
but we're going with it for now, may change later).
