Build a Mail Server with OpenSMTPD, Dovecot, and Rspamd

Synopsis #

This guide combines the base-system smtpd(8) service, packaged Dovecot IMAP and LMTP delivery, and packaged Rspamd filtering into one small mail server. It stores mail in each system account’s Maildir.

An Internet mail server is an operational commitment. Correct DNS, abuse handling, TLS renewal, filtering, queue monitoring, backups, reputation, and timely upgrades are part of the service. Test the complete design on a non-production domain before changing live MX records.

Package configuration can change independently of the OpenBSD base release. Read every installed package message and /usr/local/share/doc/pkg-readmes/ file before enabling a package daemon. Validate the installed configuration with the installed binaries.

Example Design #

The example uses:

Replace all example names before applying the configuration.

Prepare DNS and Network Access #

Before accepting mail, set up:

• an A record and, when fully routed, an AAAA record for mail.example.net;
• an MX record for example.net that names mail.example.net;
• matching reverse DNS for each public sending address, configured by the address provider;
• an initial SPF policy that describes the actual sending hosts;
• a plan for DKIM signing and DMARC reporting before production use;
• inbound TCP port 25 and authenticated submission on TCP port 587;
• IMAPS on TCP port 993 for mailbox clients.

Do not publish an AAAA record until inbound and outbound IPv6, reverse DNS, PF, and application listeners all work. Do not use a residential or provider address that blocks server SMTP.

Install the Package Components #

OpenSMTPD is part of the base system. Install the package components with pkg_add(1) :

doas pkg_add dovecot rspamd redis opensmtpd-filter-rspamd

Review the exact package messages and installed documentation:

pkg_info -M dovecot
  # Display Dovecot installation messages
pkg_info -M rspamd
  # Display Rspamd installation messages
pkg_info -M opensmtpd-filter-rspamd
  # Display filter installation messages
doas ls /usr/local/share/doc/pkg-readmes
  # Locate the installed package READMEs

Do not assume paths from another operating system or an older package revision.

Create the Mailbox Account #

Create the regular account with adduser(8) if it does not already exist:

doas adduser

Create and protect its Maildir as that account:

doas su - alice -c 'install -d -m 700 ~/Maildir ~/Maildir/cur ~/Maildir/new ~/Maildir/tmp'
  # Create the Maildir structure as the mailbox account

Install the TLS Certificate #

Obtain a certificate for mail.example.net with acme-client(1) or another controlled certificate process. The examples expect:

/etc/ssl/mail.example.net.fullchain.pem
/etc/ssl/private/mail.example.net.key

The private key must remain root-owned and mode 0600. Confirm that both OpenSMTPD and Dovecot can open the selected files during their documented privilege setup. Do not broaden private-key permissions as a first response to a startup failure.

Configure Dovecot #

Edit the package configuration files and keep their include lines. Replace the certificate and key paths directly in /etc/dovecot/conf.d/10-ssl.conf; adding overrides elsewhere can still leave Dovecot trying to load missing example files. Set the following options in the appropriate configuration files:

protocols = imap lmtp
mail_location = maildir:~/Maildir

ssl = required
ssl_cert = </etc/ssl/mail.example.net.fullchain.pem
ssl_key = </etc/ssl/private/mail.example.net.key

disable_plaintext_auth = yes
auth_mechanisms = plain
auth_username_format = %n

Edit the existing service lmtp section in /etc/dovecot/conf.d/10-master.conf to configure the OpenSMTPD socket:

service lmtp {
  unix_listener lmtp {
    mode = 0660
    user = _smtpd
    group = _smtpd
  }
}

Use lmtp as the listener name to replace the package’s existing listener. Dovecot creates it under /var/dovecot, at /var/dovecot/lmtp. Using the full path as a separate listener name can create a duplicate. Check the directory, socket path, and owner before starting Dovecot:

doas dovecot -n

The output must show only the intended IMAP and LMTP protocols, Maildir location, certificate paths, and LMTP listener.

Configure Recipient Mapping #

Create /etc/mail/virtuals:

alice@example.net alice
postmaster@example.net alice@example.net
abuse@example.net alice@example.net
sales@example.net alice

This map lists valid recipient addresses and their local accounts. Aliases can point directly to a local account, as sales does, or to a mailbox address that maps to the account. Restrict write access to administrators.

Configure Rspamd Filtering #

Use the configuration and local override directories installed by the package. The OpenSMTPD filter connects to the local Rspamd worker; do not expose the worker or controller to the Internet.

Validate the effective Rspamd configuration with rspamadm configtest:

doas rspamadm configtest

The package’s /etc/rspamd/local.d/redis.conf uses local Redis for greylisting. Start Redis with Rspamd when using these defaults. Keep Redis bound to loopback and the Rspamd worker/controller ports off public interfaces. Antivirus, neural filtering, and remote controller access are optional.

Configure OpenSMTPD #

Create /etc/mail/smtpd.conf, adjusting interface and certificate names as required:

pki mail.example.net cert "/etc/ssl/mail.example.net.fullchain.pem"
pki mail.example.net key "/etc/ssl/private/mail.example.net.key"

table virtuals file:/etc/mail/virtuals

filter "rspamd" proc-exec "filter-rspamd"

listen on lo0
listen on egress tls pki mail.example.net filter "rspamd"
listen on egress port submission tls-require pki mail.example.net auth filter "rspamd"

action "local" lmtp "/var/dovecot/lmtp" user _smtpd virtual <virtuals>
action "outbound" relay

match from any for domain "example.net" action "local"
match auth from any for any action "outbound"
match from local for any action "outbound"

The rules accept only mapped recipients for example.net, require authentication for remote submission, and allow locally generated mail to relay. They do not create an open relay.

Run LMTP delivery as _smtpd, which has access to the socket; mailbox users do not need access. Without rcpt-to, OpenSMTPD sends the mapped local username to Dovecot. This lets both alice@example.net and sales@example.net deliver to the alice mailbox. Adding rcpt-to instead sends the expanded email address; a direct mapping such as sales@example.net alice can then fail because Dovecot looks up sales rather than alice. Dovecot’s auth_username_format = %n permits clients to log in as alice@example.net in this single-domain design. Multiple domains with virtual users require a separate address-to-account mapping.

The default authentication table for the submission listener uses system accounts. Any virtual-user or database-backed authentication model requires a separate mapping and storage design.

Validate the file with smtpd(8) :

doas smtpd -n

Do not start the public listener until the configuration parses and the recipient, relay, certificate, and filter paths have been reviewed.

Add PF Rules #

Integrate the public ports into the complete /etc/pf.conf:

mail_ports = "{ 25, 587, 993 }"

pass in on egress proto tcp to (egress) port $mail_ports

Restrict administrative and monitoring interfaces separately. Parse and load the complete ruleset:

doas pfctl -nf /etc/pf.conf
  # Check the policy without changing it
doas pfctl -f /etc/pf.conf
  # Load the verified policy

Start and Verify the Stack #

Start dependencies before OpenSMTPD:

doas rcctl enable redis rspamd dovecot smtpd
  # Enable the mail stack and its Redis dependency at boot
doas rcctl start redis
  # Start the local greylisting store
doas rcctl start rspamd
  # Start the spam-filter service
doas rcctl start dovecot
  # Create the IMAP and LMTP listeners
doas test -S /var/dovecot/lmtp
  # Confirm that the LMTP socket exists
doas rcctl start smtpd
  # Start SMTP only after its dependencies work

Confirm listeners and process state:

doas rcctl check redis rspamd dovecot smtpd
  # Confirm all services are running
netstat -na -f inet | grep LISTEN
  # Confirm only the intended IPv4 listeners
netstat -na -f inet6 | grep LISTEN
  # Confirm intended IPv6 listeners when IPv6 is published

Test TLS from an external system:

openssl s_client -starttls smtp -connect mail.example.net:25 -servername mail.example.net
  # Inspect SMTP STARTTLS and the certificate chain
openssl s_client -connect mail.example.net:993 -servername mail.example.net
  # Inspect IMAPS and the certificate chain

Send separate messages to alice@example.net, sales@example.net, and postmaster@example.net. Log in through IMAPS as alice and confirm that all three messages reach its inbox. The sales alias must work even though no system account named sales exists. Also test an outbound message through authenticated submission. A successful smtpd -n checks configuration syntax; it does not prove recipient mapping or mailbox delivery. Inspect the queue and logs:

doas smtpctl show queue
  # Confirm whether any delivery is delayed
doas tail -n 100 /var/log/maillog
  # Review SMTP and delivery results
doas doveadm auth test alice
  # Test the mailbox authentication path interactively

Keep passwords out of shell history, scripts, screenshots, and shared diagnostic output.

Publish Authentication Records #

After basic delivery works:

1. Configure DKIM signing with a maintained OpenSMTPD filter or Rspamd according to the installed package documentation.
2. Publish the generated public key under the selected DKIM selector.
3. Verify DKIM on a received external message.
4. Publish a DMARC record initially in reporting mode and monitor reports.
5. Tighten SPF and DMARC only after every legitimate sender is represented.

DNS records do not correct a broken hostname, reverse DNS, relay policy, or compromised account.

Back Up and Operate the Service #

Back up Maildir data, /etc/mail, relevant Dovecot and Rspamd local configuration, ACME state, and any DKIM private keys. Test a mailbox and configuration restore on a separate system.

Monitor:

• /var/log/maillog and Dovecot/Rspamd logs selected by their effective configurations;
• OpenSMTPD queue age and size;
• filesystem and inode use for mail storage;
• certificate expiry and renewal;
• DNS and reverse-DNS correctness;
• failed authentication and unusual outbound volume;
• package readmes and configuration changes during upgrades.

See OpenSMTPD , Dovecot , and Rspamd for component reference material. Installed package documentation remains authoritative for package-specific paths and syntax.