Build a Mail Server with OpenSMTPD, Dovecot, and Rspamd

Synopsis #

This guide combines the base-system smtpd(8) service, packaged Dovecot IMAP and LMTP delivery, and packaged Rspamd filtering into one small mail server. It uses system accounts and Maildir storage to keep the initial design reviewable.

An Internet mail server is an operational commitment. Correct DNS, abuse handling, TLS renewal, filtering, queue monitoring, backups, reputation, and timely upgrades are part of the service. Test the complete design on a non-production domain before changing live MX records.

Package configuration can change independently of the OpenBSD base release. Read every installed package message and /usr/local/share/doc/pkg-readmes/ file before enabling a package daemon. Validate the installed configuration with the installed binaries.

Example Design #

The example uses:

Replace all example names before applying the configuration.

Prepare DNS and Network Access #

Before accepting mail, establish:

• an A record and, when fully routed, an AAAA record for mail.example.net;
• an MX record for example.net that names mail.example.net;
• matching reverse DNS for each public sending address, configured by the address provider;
• an initial SPF policy that describes the actual sending hosts;
• a plan for DKIM signing and DMARC reporting before production use;
• inbound TCP port 25 and authenticated submission on TCP port 587;
• IMAPS on TCP port 993 for mailbox clients.

Do not publish an AAAA record until inbound and outbound IPv6, reverse DNS, PF, and application listeners all work. Do not use a residential or provider address that blocks server SMTP.

Install the Package Components #

OpenSMTPD is part of the base system. Install the package components with pkg_add(1) :

# pkg_add dovecot rspamd opensmtpd-filter-rspamd

Review the exact package messages and installed documentation:

# pkg_info -M dovecot
  # Display Dovecot installation messages
# pkg_info -M rspamd
  # Display Rspamd installation messages
# pkg_info -M opensmtpd-filter-rspamd
  # Display filter installation messages
# ls /usr/local/share/doc/pkg-readmes
  # Locate the installed package READMEs

Do not assume paths from another operating system or an older package revision.

Create the Mailbox Account #

Create the regular account with adduser(8) if it does not already exist:

# adduser

Create and protect its Maildir as that account:

# su - alice
  # Enter the mailbox account
$ install -d -m 700 ~/Maildir ~/Maildir/cur ~/Maildir/new ~/Maildir/tmp
  # Create the Maildir structure
$ exit
  # Return to the administrative shell

Install the TLS Certificate #

Obtain a certificate for mail.example.net with acme-client(1) or another controlled certificate process. The examples expect:

/etc/ssl/mail.example.net.fullchain.pem
/etc/ssl/private/mail.example.net.key

The private key must remain root-owned and mode 0600. Confirm that both OpenSMTPD and Dovecot can open the selected files during their documented privilege setup. Do not broaden private-key permissions as a first response to a startup failure.

Configure Dovecot #

Start from the configuration installed by the package and preserve its includes. The minimum policy is:

protocols = imap lmtp
mail_location = maildir:~/Maildir

ssl = required
ssl_cert = </etc/ssl/mail.example.net.fullchain.pem
ssl_key = </etc/ssl/private/mail.example.net.key

disable_plaintext_auth = yes
auth_mechanisms = plain

Configure an LMTP socket for OpenSMTPD in the Dovecot service configuration:

service lmtp {
  unix_listener /var/dovecot/lmtp {
    mode = 0660
    user = _smtpd
    group = _smtpd
  }
}

The socket path and ownership must match the installed OpenBSD package layout. Validate the effective configuration before starting Dovecot:

# dovecot -n

The output must show only the intended IMAP and LMTP protocols, Maildir location, certificate paths, and LMTP listener.

Configure Recipient Mapping #

Create /etc/mail/virtuals:

alice@example.net alice
postmaster@example.net alice
abuse@example.net alice

This explicit map determines which public recipients exist and which local accounts receive them. Protect it as administrative configuration.

Configure Rspamd Filtering #

Use the configuration and local override directories installed by the package. The OpenSMTPD filter connects to the local Rspamd worker; do not expose the worker or controller to the Internet.

Validate the effective Rspamd configuration with rspamadm(1) :

# rspamadm configtest

Start with the package defaults and local-only listeners. Redis, antivirus, neural filtering, and remote controller access are separate design choices, not prerequisites for the first working delivery path.

Configure OpenSMTPD #

Create /etc/mail/smtpd.conf, adjusting interface and certificate names as required:

pki mail.example.net cert "/etc/ssl/mail.example.net.fullchain.pem"
pki mail.example.net key "/etc/ssl/private/mail.example.net.key"

table virtuals file:/etc/mail/virtuals

filter "rspamd" proc-exec "filter-rspamd"

listen on lo0
listen on egress tls pki mail.example.net filter "rspamd"
listen on egress port submission tls-require pki mail.example.net auth filter "rspamd"

action "local" lmtp "/var/dovecot/lmtp" rcpt-to virtual <virtuals>
action "outbound" relay

match from any for domain "example.net" action "local"
match auth from any for any action "outbound"
match from local for any action "outbound"

The rules accept only mapped recipients for example.net, require authentication for remote submission, and allow locally generated mail to relay. They do not create an open relay.

The default authentication table for the submission listener uses system accounts. Any virtual-user or database-backed authentication model requires a separate mapping and storage design.

Validate the file with smtpd(8) :

# smtpd -n

Do not start the public listener until the configuration parses and the recipient, relay, certificate, and filter paths have been reviewed.

Add PF Rules #

Integrate the public ports into the complete /etc/pf.conf:

mail_ports = "{ 25, 587, 993 }"

pass in on egress proto tcp to (egress) port $mail_ports

Restrict administrative and monitoring interfaces separately. Parse and load the complete ruleset:

# pfctl -nf /etc/pf.conf
  # Check the policy without changing it
# pfctl -f /etc/pf.conf
  # Load the verified policy

Start and Verify the Stack #

Start dependencies before OpenSMTPD:

# rcctl enable rspamd dovecot smtpd
  # Enable all three services at boot
# rcctl start rspamd
  # Start the spam-filter service
# rcctl start dovecot
  # Create the IMAP and LMTP listeners
# test -S /var/dovecot/lmtp
  # Confirm that the LMTP socket exists
# rcctl start smtpd
  # Start SMTP only after its dependencies work

Confirm listeners and process state:

# rcctl check rspamd dovecot smtpd
  # Confirm all services are running
$ netstat -na -f inet | grep LISTEN
  # Confirm only the intended IPv4 listeners
$ netstat -na -f inet6 | grep LISTEN
  # Confirm intended IPv6 listeners when IPv6 is published

Test TLS from an external system:

$ openssl s_client -starttls smtp -connect mail.example.net:25 -servername mail.example.net
  # Inspect SMTP STARTTLS and the certificate chain
$ openssl s_client -connect mail.example.net:993 -servername mail.example.net
  # Inspect IMAPS and the certificate chain

Send one message to a mapped local recipient and one outbound message through authenticated submission. Inspect the queue and logs:

# smtpctl show queue
  # Confirm whether any delivery is delayed
# tail -n 100 /var/log/maillog
  # Review SMTP and delivery results
# doveadm auth test alice
  # Test the mailbox authentication path interactively

Do not include passwords in shell history, scripts, screenshots, or review output.

Publish Authentication Records #

After basic delivery works:

1. Configure DKIM signing with a maintained OpenSMTPD filter or Rspamd according to the installed package documentation.
2. Publish the generated public key under the selected DKIM selector.
3. Verify DKIM on a received external message.
4. Publish a DMARC record initially in reporting mode and monitor reports.
5. Tighten SPF and DMARC only after every legitimate sender is represented.

DNS records do not correct a broken hostname, reverse DNS, relay policy, or compromised account.

Back Up and Operate the Service #

Back up Maildir data, /etc/mail, relevant Dovecot and Rspamd local configuration, ACME state, and any DKIM private keys. Test a mailbox and configuration restore on a separate system.

Monitor:

• /var/log/maillog and Dovecot/Rspamd logs selected by their effective configurations;
• OpenSMTPD queue age and size;
• filesystem and inode use for mail storage;
• certificate expiry and renewal;
• DNS and reverse-DNS correctness;
• failed authentication and unusual outbound volume;
• package readmes and configuration changes during upgrades.

See OpenSMTPD , Dovecot , and Rspamd for component reference material. Installed package documentation remains authoritative for package-specific paths and syntax.