Self-Hosted Email Security Best Practices

The Security Checklist

Self-hosted email means you're the sysadmin. A compromised email server sends spam from your IP, gets blacklisted, and destroys your reputation. Prevention is everything. This complements our Mailcow setup guide and SMTP authentication guide.

1. TLS Everywhere

All connections must be encrypted:

• Port 25 (SMTP): STARTTLS for server-to-server
• Port 587 (Submission): STARTTLS for authenticated sending
• Port 993 (IMAPS): TLS for email retrieval
• Port 443 (HTTPS): TLS for webmail and admin

Certificate: Use Let's Encrypt (free, auto-renewed). Mailcow handles this automatically. For raw Postfix: install certbot.

Verify: Test your TLS configuration at checktls.com.

2. Prevent Open Relay

This is the #1 security failure for self-hosted email. An open relay sends email from anyone to anyone — spammers will find and abuse it within hours.

For Postfix: Verify in /etc/postfix/main.cf:

smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

For Mailcow/Postal: Closed by default. Don't modify relay settings unless you understand the implications.

Test: Use mxtoolbox.com/diagnostic.aspx to test for open relay.

3. fail2ban

Brute-force attacks against SMTP/IMAP are constant. fail2ban automatically blocks IPs that fail authentication repeatedly.

# Install
apt install fail2ban

# Configure for email (create /etc/fail2ban/jail.local)
[postfix-sasl]
enabled = true
port = smtp,465,submission,imap,imaps
filter = postfix-sasl
logpath = /var/log/mail.log
maxretry = 3
bantime = 3600

Mailcow includes Netfilter (Docker-based fail2ban equivalent). Enable it in mailcow.conf.

4. Firewall

Only open the ports you need:

# UFW example
ufw default deny incoming
ufw allow 22/tcp    # SSH
ufw allow 25/tcp    # SMTP (server to server)
ufw allow 587/tcp   # Submission (authenticated)
ufw allow 993/tcp   # IMAPS
ufw allow 443/tcp   # HTTPS (webmail/admin)
ufw enable

Block everything else. If you're only sending (not receiving), you can skip port 993.

5. Strong Authentication

• Admin panel: Strong password + 2FA if available
• Mailbox accounts: Minimum 12-character passwords
• SMTP credentials: Unique per service/integration
• SSH: Key-based authentication, disable password login

6. Automatic Updates

# Enable unattended security updates (Ubuntu)
apt install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

For Mailcow Docker updates:

cd /opt/mailcow-dockerized
./update.sh

Run monthly. Critical security patches: apply within 24 hours.

7. Backup Strategy

What to backup:

• Email data (mailboxes, DKIM keys)
• Configuration files
• Database

For Mailcow:

cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh backup

Schedule: Daily incremental, weekly full. Store offsite (different VPS or S3).

8. Monitoring

Watch for signs of compromise:

• Sudden increase in outgoing email volume
• Authentication failures in logs
• New unknown mailbox accounts
• IP appearing on blacklists
• Bounce-back from emails you didn't send

Set up alerts for these conditions. Blacklist monitoring guide.

If Your Server Gets Compromised

1. Immediately block outgoing port 25 (stop spam from leaving)
2. Change all passwords (admin, mailboxes, SMTP credentials)
3. Check for unauthorized accounts in your mail server
4. Review authentication logs for the breach point
5. Check blacklists and begin delisting
6. Patch the vulnerability that was exploited
7. Restore from backup if the system was modified
8. Re-enable sending after securing

Mailcow vs Manual Security

Security Feature	Mailcow	Raw Postfix
TLS auto-renewal	Automatic (Let's Encrypt)	Manual (certbot)
Spam filtering	Rspamd (included)	Manual (SpamAssassin, Rspamd)
Antivirus	ClamAV (included)	Manual install
Brute-force protection	Netfilter (included)	fail2ban (manual)
Open relay prevention	Configured by default	Must verify manually
Updates	update.sh script	Manual apt commands
Docker isolation	Yes	N/A

Mailcow reduces security configuration from hours to minutes. For most self-hosted setups, it's the pragmatic choice.

Practitioner note: The most common self-hosted email compromise I've seen: weak SMTP credentials (password123) on a Postfix server without fail2ban. A brute-force attack cracked the password in under an hour. The server sent 50K spam messages before the owner noticed. The IP was blacklisted on Spamhaus for 2 weeks. Use strong passwords. Install fail2ban. This is not optional.

Practitioner note: If you're running a self-hosted email server on a VPS, also harden SSH. Disable password authentication (use key-based only), change the SSH port from 22, and install fail2ban for SSH. Your email server is only as secure as the VPS it runs on.

If you want a self-hosted email server set up with proper security from the start, schedule a consultation.

Sources

• Postfix: SASL Authentication
• fail2ban: Documentation
• Let's Encrypt: certbot
• Mailcow: Security

---

v1.0 · March 2026