Network Services at Scale

Network Services at Scale

Synopsis #

This chapter describes building reliable, scalable network services on OpenBSD: authoritative DNS with nsd(8) configured via nsd.conf(5) , validating recursive DNS with unbound(8) configured via unbound.conf(5) , IPv4 address allocation with dhcpd(8) configured via dhcpd.conf(5) , and time services with ntpd(8) configured via ntpd.conf(5) . Service lifecycle management uses rcctl(8) . Packet filtering allowances live in pf.conf(5) and are applied with pfctl(8) . Use these patterns for branch offices, campus networks, and PoPs where you operate DNS, DHCP, and NTP as first-class, redundant services.

Design Considerations #

  • Separation of roles. Bind authoritative and recursive DNS to different IP addresses. A common pattern is NSD listening on a public/WAN address (or a dedicated service VIP) and Unbound listening on LAN addresses.
  • Redundancy. Use at least two servers per critical service. Combine with CARP and pfsync from the HA chapter for failover of service VIPs.
  • Security posture. Permit UDP/TCP 53 from the Internet only to authoritative servers. Restrict recursion to inside prefixes. Enable DNSSEC validation in Unbound.
  • Operational boundaries. NSD does not support dynamic updates; keep zone management explicit and version-controlled. Unbound is not authoritative; use stub zones to prefer local authoritative answers when both run on the same host.
  • IPv6. Provide AAAA records in authoritative zones and ensure recursive service answers both A and AAAA. Address assignment for IPv6 prefers RA (see the IPv6 chapter); DHCPv6 is out of scope here.
  • Time service. DNSSEC validation requires correct time. Ensure ntpd(8) is running on resolvers and that clients have a local NTP source.

Configuration #

Assumptions for examples:

  • Interfaces: em0 (WAN), em1 (LAN 10.10.10.0/24).
  • Service IPs: 203.0.113.53 (authoritative DNS on WAN), 10.10.10.1 (recursive DNS and NTP on LAN).
  • Authoritative zone: example.com. served by NSD.
  • Local recursion for LAN clients via Unbound.

1) Authoritative DNS with nsd(8) #

Bind NSD to the public address and serve example.com. from a local zone file. Control socket is enabled for safe reloads.

## /etc/nsd.conf — NSD authoritative service on WAN address

server:
    ip-address: 203.0.113.53
    hide-version: yes
    do-ip4: yes
    do-ip6: no
    server-count: 1
    zonesdir: "/var/nsd/zones"
    # control socket for nsd-control(8)
    control-enable: yes

zone:
    name: "example.com"
    zonefile: "example.com.zone"

Create the zone file (minimal example; expand as required). The SOA and NS records must be correct for production.

## /var/nsd/zones/example.com.zone — minimal zone

$ORIGIN example.com.
$TTL 300
@   IN SOA ns1.example.com. hostmaster.example.com. (
        2025010101 ; serial (YYYYMMDDNN)
        3600       ; refresh
        600        ; retry
        1209600    ; expire
        300        ; minimum
)
    IN  NS  ns1.example.com.
ns1 IN  A   203.0.113.53
www IN  A   203.0.113.80
api IN  A   203.0.113.81

First-time control key setup and service start:

# nsd-control-setup
  # Generate control keys/certs under /var/nsd/etc; one-time
# rcctl enable nsd
  # Start on boot
# rcctl start nsd
  # Launch now
# nsd-control reload
  # Validate and load the zone

2) Validating recursive DNS with unbound(8) #

Unbound will listen on LAN, serve only inside clients, and validate DNSSEC. When NSD and Unbound share a host, forward a local zone to NSD via a stub to avoid conflicts on port 53 by running NSD on WAN and Unbound on LAN.

## /var/unbound/etc/unbound.conf — LAN recursion with validation

server:
    interface: 127.0.0.1
    interface: 10.10.10.1
    access-control: 10.10.10.0/24 allow
    access-control: 127.0.0.0/8 allow

    # Hardening and privacy
    hide-identity: yes
    hide-version: yes
    qname-minimisation: yes
    harden-referral-path: yes
    harden-glue: yes
    prefetch: yes

    # DNSSEC
    auto-trust-anchor-file: "/var/unbound/db/root.key"

    # Root priming; Unbound can fetch root NS automatically if needed
    root-hints: "/var/unbound/db/root.hints"

# Prefer local authoritative answers (optional internal zone served by NSD)
stub-zone:
    name: "corp.example."
    stub-addr: 127.0.0.1@5353

If you need Unbound to prefer a local zone while NSD keeps port 53 on WAN, bind NSD additionally to 127.0.0.1 port 5353 and serve corp.example. there (example below). Otherwise omit the stub-zone.

## Excerpt — additional NSD listener for an internal zone (loopback)

server:
    # ... existing server block ...
    ip-address: 127.0.0.1@5353

zone:
    name: "corp.example"
    zonefile: "corp.example.zone"

Initialize Unbound control and start the resolver:

# unbound-control-setup
  # Generate control keys/certs under /var/unbound/etc; one-time
# rcctl enable unbound
# rcctl start unbound
# unbound-control status
  # Confirm validator is ready and loaded

3) DHCP for IPv4 with dhcpd(8) #

Serve 10.10.10.0/24, advertise the local resolver and default gateway, and provide a static mapping example.

## /etc/dhcpd.conf — IPv4 DHCP on LAN

option domain-name "corp.example";
option domain-name-servers 10.10.10.1;
option routers 10.10.10.1;
option subnet-mask 255.255.255.0;
default-lease-time 3600;
max-lease-time 7200;
authoritative;

subnet 10.10.10.0 netmask 255.255.255.0 {
    range 10.10.10.100 10.10.10.200;

    # Static reservation example
    host printer-01 {
        hardware ethernet 00:11:22:33:44:55;
        fixed-address 10.10.10.50;
    }
}

Enable the daemon and bind it to the LAN interface:

# rcctl set dhcpd flags em1
  # Specify served interfaces (required)
# rcctl enable dhcpd
# rcctl start dhcpd
# tail -n 20 /var/db/dhcpd.leases
  # Inspect active leases

For IPv6 address assignment, prefer Router Advertisements with rad(8) as shown in the IPv6 chapter.

4) Time service with ntpd(8) #

Provide an internal NTP source on the LAN and maintain correct time on the server itself.

## /etc/ntpd.conf — serve LAN and sync from upstream pool

listen on 10.10.10.1
servers pool.ntp.org

# rcctl enable ntpd
# rcctl start ntpd
# ntpctl -s status
  # Report synchronization state

5) PF allowances for DNS, DHCP, and NTP #

Permit external queries to NSD on WAN, recursive service on LAN, DHCP on LAN, and NTP for clients.

## /etc/pf.conf — service allowances (merge into your policy)

set skip on lo

wan = "em0"
lan = "em1"

# Base policy
block all

# Authoritative DNS on WAN (NSD)
pass in on $wan proto { udp tcp } to 203.0.113.53 port 53 keep state

# Recursive DNS on LAN (Unbound)
pass in on $lan proto { udp tcp } to 10.10.10.1 port 53 keep state

# DHCP on LAN (server replies from 67 to client port 68)
pass in on $lan proto udp from port 68 to port 67 keep state
pass out on $lan proto udp from port 67 to port 68 keep state

# NTP on LAN
pass in on $lan proto udp to 10.10.10.1 port 123 keep state

# Resolver and NSD outbound as needed
pass out on $wan proto { udp tcp } to any port { 53, 123 } keep state

Reload after editing:

# pfctl -f /etc/pf.conf
  # Load rules atomically

Verification #

  • Authoritative zone served by NSD (from a remote host or the firewall itself), using drill(1) :
$ drill @203.0.113.53 example.com SOA
  # Expect the SOA you configured
$ drill @203.0.113.53 www.example.com A
  # Expect 203.0.113.80
  • Recursive resolution with DNSSEC validation (from a LAN client):
$ drill @10.10.10.1 cloudflare.com A
  # General recursion works
$ drill -S @10.10.10.1 dnssec-failed.org
  # Should return a validation failure (bogus), proving DNSSEC is active
  • Unbound and NSD control:
# unbound-control list_stubs
  # Confirm any stub-zones are loaded
# nsd-control stats
  # Basic qps/counters for authoritative service
  • DHCP leases and client config:
$ grep dhcp /var/log/messages | tail -n 20
  # Server activity
$ cat /var/db/dhcpd.leases | tail -n 10
  # Lease database updates
  • NTP status:
$ ntpctl -s peers
  # Upstream peers and offsets
$ ntpctl -s status
  # Overall sync status

Troubleshooting #

  • Port conflict on 53. NSD and Unbound cannot both bind the same IP:port. Bind NSD to the WAN address and Unbound to LAN addresses, or move one to 127.0.0.1@5353 and use Unbound stub-zone for the internal domain.
  • Open resolver exposed. If outside hosts can recurse, restrict Unbound with access-control to inside prefixes only and confirm PF denies WAN access to the recursive listener.
  • DNSSEC failures. Validation requires correct system time and a current trust anchor. Ensure ntpd(8) is synchronized and that Unbound has a valid auto-trust-anchor-file.
  • Authoritative zone not loading. Check nsd-control reload output and syslog for syntax errors. Validate SOA/NS correctness and serial increments.
  • DHCP not serving. Confirm rcctl set dhcpd flags <lan-ifaces> and that PF allows UDP 67/68 on the LAN. Inspect /var/db/dhcpd.leases and logs for pool exhaustion or MAC mismatches.
  • Clients ignore NTP. Ensure LAN clients point to 10.10.10.1 for NTP and that PF allows UDP 123 from LAN to the server.

See Also #