The loop test can be quite slow, specially on computers without
cryptography-friendly instructions.
This patch introduces a new flag for testing, so that we can bring the
threshold down to 5. The test is just as useful but now runs in a few
seconds, as opposed to a few minutes.
The queue IDs are internal to chasquid, and we don't need them to be
cryptographically secure, so this patch changes the ID generation to use
the PRNG.
This also helps avoid entropy issues.
glog works fine and has great features, but it does not play along well
with systemd or standard log rotators (as it does the rotation itself).
So this patch replaces glog with a new logging module "log", which by
default logs to stderr, in a systemd-friendly manner.
Logging to files or syslog is still supported.
This patch introduces a Makefile to make it easier to build with version
information, and run all the tests together.
It's just for convenience, plain go commands continue to work just fine.
While greylisting has severe drawbacks, it's reasonably common.
This patch extends the default hook for greylisting, using "greylistd".
Completing authentication or passing SPF will make it skip the greylist
check, to avoid some of the most common issues with it.
This patch makes the hooks always have a complete set of environment
varuables, set to 0/1 or whatever is appropriate, to make it easier to
write the checks for them.
It is can be convenient for hooks to indicate that an error is
permanent; for example if the anti-virus found something.
This patch makes it so that if the hook exits with code 20, then it's
considered permanent. Otherwise it is considered transient, to help
prevent accidental errors cause final delivery issues.
The "ptr" mechanism should not be used, but unfortunately many domains
still do.
This patch extends our SPF implementation to support checking it.
https://tools.ietf.org/html/rfc7208#section-5.5
Calculating the next delay based on the previous delay causes daemon
restarts to start from scratch, as we don't persist it.
This can cause a few server restarts to generate many unnecessary sends.
This patch changes the next delay calculation to use the creation time
instead, and also adds a <=1m random perturbation to avoid all queued
emails to be retried at the exact same time after a restart.
TestAliases is unfortunately racy, and cause occasional failures.
This patch rewrites it to be more end-to-end, similar to TestBasic,
which should remove the races while keeping the main objectives of the
test.
The queue currently only considers failed recipients when deciding
whether to send a DSN or not. This is a bug, as recipients that time out
are not taken into account.
This patch fixes that issue by including both failed and pending
recipients in the DSN.
It also adds more comprehensive tests for this case, both in the queue
and in the dsn generation code.
This patch changes chasquid-util's subcommands and parameters to
(hopefully) make them more user friendly and intuitive by default.
The changes include defaulting the configuration to /etc/chasquid, and
using full addresses as usernames.
It also adds some shell tests to cover most of the functionality.
The default INFO logs are more oriented towards debugging and can be
a bit too verbose when looking for high-level information.
This patch introduces a new "maillog" package, used to log messages of
particular relevance to mail transmission at a higher level.
This patch makes safeio preserve file ownership. This is specially
useful when using command-line utilities as root, but the files they
change are owned by a different user.
This patch implements a post-DATA hook, which is run after receiving the
data but before sending a reply.
It can be used to implement content filtering when receiving email, for
example for passing the email through an anti-spam or an anti-virus.
Unknown commands can fill the logs, traces and expvars with a lot of
noise; this patch sanitizes them a bit down to 6 bytes, as a compromise
to maintain some information for troubleshooting.
The submission port is expected to be used only by authenticated
clients, so this patch makes chasquid enforce this, which also helps
to reduce spam.
https://www.rfc-editor.org/rfc/rfc6409.txt
Today, we pick the domain used to send the DSN from based on what we
presented to the client at EHLO time, which itself may be based on the
TLS negotiation (which is not necessarily trusted).
This is complex, not necessarily correct, and involves passing the
domain around through the queue and persisting it in the items.
So this patch simplifies that handling by always using the main domain
as specified by the configuration.
This patch moves chasquid's Server and Conn structures to their own
smtpsrv package, to make chasquid.go a bit more readable. It also helps
clarify the relation between Server and Conn.
There are no functional changes.
Note that git can still track the history across this commit (e.g. git
gui blame shows the right data).
This patch introduces a new "domaininfo" package, which implements a
database with information about domains. In particular, it tracks
incoming and outgoing security levels.
That information is used in incoming and outgoing SMTP to prevent
downgrades.
This patch is the result of running go vet, go fmt -s and the linter,
and fixing some of the things they noted/suggested.
There shouldn't be any significant logic changes, it's mostly
readability improvements.
Errors can contain newlines, in particular this is common in messages
returned by remote SMTP servers.
This patch quotes them before logging, they don't interfere with the
log. Note the tracer itself can handle this just fine.
This patch simplifies the sending loop code:
- Move the recipient sending function from a closure to a method.
- Simplify the status update logic: we now update and write
unconditionally (as we should have been doing).
- Create a function for counting recipients in a given status.
It also adds a test for the removal of completed items from the queue,
which was not covered before and came up during development.
Add some links to the monitoring HTML index, to reflect the new
additions.
Also reorder to make it more practical, and default to the expanded view
for the tracer links.
HELO and EHLO both take a mandatory parameter, which also should be used
in the Received header.
This patch tracks and enforces that parameter, and also updates the
Received header generation to use it.
https://tools.ietf.org/html/rfc5321#section-4.4
If a connection has accumulated 10 errors, it's very likely that
something has gone significantly wrong, or they're just probing/abusing
the service.
This patch makes chasquid break the connection after 10 errors.
The number is arbitrary, we may adjust it later.
We should ignore the domains' case, and treat them uniformly, specially when it
comes to local domains.
This patch extends the existing normalization (IDNA, keeping domains as
UTF8 internally) to include case conversion and NFC form for
consistency.
This patch implements local username normalization using PRECIS
(https://tools.ietf.org/html/rfc7564,
https://tools.ietf.org/html/rfc7613)
It makes chasquid accept local email and authentication regardless of
the case. It covers both userdb and aliases.
Note that non-local usernames remain untouched.
The default for max events is 10, which is a bit short for a normal SMTP
exchange. Expand it to 30 which should be large enough to keep most of
the traces.
Currently, we do SPF checks for all connections.
However, authenticated users will be sending email from different
locations, applying SPF to them will result in false positives.
So this patch makes chasquid skip SPF checking if the connection is
authenticated.
We don't want to write debug messages to the log, but having them in
traces is always useful.
The traces are volatile and self-cleaning so the additional volume
should not cause any problems, and helps troubleshooting.
While at it, fix the depth constant when logging.
The logic that picks the domain we use for EHLO does not need to live
within the TLS retry cycle, and makes it harder to understand.
This patch moves it out of the way, to improve readability.
This patch implements some measures against email loops, such as keeping
a limit on the lenght of an address, and rejecting email that has too
many Received headers.
It's not perfect (a server could be actively removing Received headers),
but it should cover the normal accidents and misconfigurations.
This patch introduces expvar counters to chasquid and the queue
packages.
For now there's only a handful of counters, but they will be expanded in
future patches.
The test courier has a racy map access, and this started to manifest
after some recent changes. This patch fixes the race by implementing the
corresponding locks.
