Email Server Configuration: The Essentials

Email server configuration is straightforward when you follow a logical order and skip the optional pieces until you need them. Most "my mail server doesn't work" problems trace to two issues: missed DNS records and incorrect authentication setup. This guide is the essentials checklist I work through on new server builds.

The configuration checklist

In order:

1. DNS records — MX, A/AAAA, reverse DNS, SPF, DKIM, DMARC
2. MTA install and config — Postfix is standard
3. IMAP install and config (if receiving) — Dovecot
4. TLS certificates — Let's Encrypt
5. SMTP authentication — SASL bridging to Dovecot or PAM
6. DKIM signing — OpenDKIM or built into stack
7. Spam filtering — Rspamd or SpamAssassin
8. Monitoring — queue, logs, blacklist, reputation

Pre-built stacks (Mailcow, Mail-in-a-Box, iRedMail) handle 2-7 in a single install. The pieces that matter most: DNS, TLS, and DKIM. Skip those or get them wrong and your mail won't reach the inbox.

DNS configuration

Every sending or receiving mail server needs:

mail.example.com.        A      203.0.113.10
mail.example.com.        AAAA   2001:db8::10
example.com.             MX  10 mail.example.com.
example.com.             TXT    "v=spf1 mx ~all"
selector1._domainkey.example.com.   TXT  "v=DKIM1; k=rsa; p=..."
_dmarc.example.com.      TXT    "v=DMARC1; p=none; rua=mailto:[email protected];"

Plus reverse DNS (PTR) at your VPS provider:

10.113.0.203.in-addr.arpa.  PTR  mail.example.com.

The PTR must match the hostname your mail server uses in HELO/EHLO. Receivers (Gmail, Microsoft) reject mail from servers whose PTR doesn't match.

See DNS records for email for the complete reference including BIMI and MTA-STS.

Postfix configuration (the MTA)

After installing Postfix, the critical /etc/postfix/main.cf settings:

# Identity
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain

# Network
inet_interfaces = all
inet_protocols = ipv4, ipv6
mydestination = $myhostname, localhost.$mydomain, localhost

# Mail flow
relay_domains = $mydestination
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

# TLS (outbound)
smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_CApath = /etc/ssl/certs

# TLS (inbound)
smtpd_tls_security_level = may
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_loglevel = 1

# SASL authentication via Dovecot
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_relay_restrictions =
    permit_mynetworks
    permit_sasl_authenticated
    reject_unauth_destination

# Message size
message_size_limit = 26214400

Then /etc/postfix/master.cf enables submission on port 587 with STARTTLS:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

Restart with systemctl reload postfix.

Full Postfix configuration is covered in Postfix Dovecot setup guide.

Dovecot configuration (the IMAP server)

Dovecot handles IMAP, POP3, and provides the SASL backend Postfix uses.

In /etc/dovecot/conf.d/10-master.conf, enable the SASL socket for Postfix:

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

In /etc/dovecot/conf.d/10-ssl.conf, enable TLS:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
ssl_min_protocol = TLSv1.2

In /etc/dovecot/conf.d/10-mail.conf, set mailbox location:

mail_location = maildir:~/Maildir

Restart with systemctl reload dovecot.

TLS certificates

Use Let's Encrypt:

apt install certbot
certbot certonly --standalone -d mail.example.com

Configure auto-renewal:

systemctl enable certbot.timer
systemctl start certbot.timer

Or run a renewal cron job:

0 2 * * * certbot renew --quiet --post-hook "systemctl reload postfix dovecot"

Certificates expire every 90 days. Auto-renewal must work — expired certs break mail flow.

DKIM signing

Install OpenDKIM:

apt install opendkim opendkim-tools

Generate a key pair:

mkdir -p /etc/opendkim/keys/example.com
opendkim-genkey -b 2048 -d example.com -s mail -D /etc/opendkim/keys/example.com
chown -R opendkim:opendkim /etc/opendkim

This creates /etc/opendkim/keys/example.com/mail.private (your private key) and mail.txt (the public key for DNS).

Configure /etc/opendkim.conf:

Domain                  example.com
KeyFile                 /etc/opendkim/keys/example.com/mail.private
Selector                mail
Socket                  inet:8891@localhost

Configure Postfix to use OpenDKIM as a milter in main.cf:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

Restart both:

systemctl restart opendkim postfix

Publish the public key (contents of mail.txt) at mail._domainkey.example.com.

See DKIM setup guide for full DKIM detail.

SMTP authentication

The recommended pattern is SASL via Dovecot (configured above). When a user connects on port 587, they authenticate with their mailbox username and password, Dovecot verifies, and Postfix permits relay.

For OAuth-based authentication (less common in self-hosted), use Dovecot's XOAUTH2 plugin.

For application/service accounts, create dedicated mailboxes and credentials. Don't use admin accounts for SMTP auth in apps.

Spam filtering (inbound)

For inbound filtering, install Rspamd:

apt install rspamd

Configure Postfix to use Rspamd as a milter:

smtpd_milters = inet:localhost:8891, inet:localhost:11332

(The first milter is OpenDKIM; the second is Rspamd.)

Rspamd integrates with Bayesian filtering, RBL checks, URL reputation, and more out of the box.

Practitioner note: Skip SpamAssassin in 2026 — Rspamd is faster, has better default rules, and is the de facto standard for modern stacks. Mailcow ships with Rspamd; Mail-in-a-Box uses SpamAssassin. Either works, but for performance and configurability, Rspamd wins.

Monitoring

What to monitor:

• Postfix queue depth — mailq or postqueue -p. Spikes indicate delivery issues.
• Mail logs — /var/log/mail.log for errors, deferrals, rejections.
• Blacklist status — daily checks of your IP against major RBLs (Spamhaus, SpamCop, Barracuda).
• DMARC aggregate reports — set rua=mailto:[email protected] and review weekly.
• Disk usage — mailboxes and logs grow.
• TLS certificate expiry — renew warnings.

Tools: Prometheus + node_exporter + postfix_exporter for metrics; Loki or syslog forwarding for logs; MXToolbox or blacklist monitoring tools for RBL checks.

Common configuration mistakes

• Missing or mismatched reverse DNS — top reason mail goes to spam from self-hosted servers.
• DKIM signing only some mail — milter not wired into both smtpd_milters and non_smtpd_milters.
• TLS cert expired without renewal hook — mail flow breaks intermittently.
• No queue monitoring — failures pile up invisibly.
• Open relay — mynetworks or relay_domains too permissive lets the world relay through you.
• No outbound rate limiting — a misbehaving local user can burn your IP reputation in an hour.

If you're configuring an email server from scratch or trying to fix one that's been limping along, book a consultation. Server-level mail configuration is a regular engagement type and I can usually catch the misconfiguration in an hour or two.

Sources

• Postfix Configuration Documentation
• Dovecot Configuration Documentation
• OpenDKIM Documentation
• Let's Encrypt Documentation
• RFC 5321: SMTP
• M3AAWG Sender Best Common Practices v3

---

v1.0 · May 2026