Reference Architectures

Synopsis #

This chapter assembles end-to-end reference designs built from components covered earlier. It presents three canonical patterns:

Redundant Internet Edge with carp(4), pfsync(4), per-uplink NAT and policy routing in pf.conf(5), managed by pfctl(8).
Campus/Branch Routed Access using VLAN SVIs, Router Advertisements with rad(8), internal recursion via unbound(8), authoritative zones with nsd(8), and DHCP with dhcpd(8).
Regional POP with either an MPLS LDP core via ldpd(8) and mpe(4), or an encrypted hub-and-spoke using iked(8) or wg(4). BGP policy lives in the OpenBGPD chapter.

Each pattern includes minimal, copy-pasteable configuration and verification guidance.

Design Considerations #

Failure domains. Keep state sync and control traffic on a private link. Avoid bridging across routers unless strictly required.
Configuration parity. Maintain identical PF and service configs on HA pairs; only device-specific interface addressing should differ.
Change management. Use atomic PF reloads and a rollback timer during first deployment. Keep a failsafe anchor that preserves management access.
Addressing plan. Allocate deterministic per-VLAN /24 (IPv4) and /64 (IPv6) with structured mapping to sites and roles.
Observability. Enable pflogd(8) and (optionally) pflow(4) toward a collector; keep clocks accurate via ntpd(8).

Pattern 1 — Redundant Internet Edge (Two Firewalls, Two ISPs) #

Scope. Two OpenBSD firewalls in HA; two WAN uplinks (ISP-A and ISP-B); deterministic outbound steering and symmetric inbound using route-to and reply-to.

Topology and Interface Assumptions #

em0 = ISP-A, em3 = ISP-B, em1 = LAN, em2 = SYNC
CARP VIPs: WAN-A 198.51.100.2/29 (vhid 10), WAN-B 203.0.113.2/29 (vhid 20), LAN 10.10.10.1/24 (vhid 30)
Node A preferred (advskew 0), Node B backup (advskew 100)

System and Interfaces (Node A; mirror with advskew 100 on Node B) #

# printf '%s\n' 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
# sysctl net.inet.carp.preempt=1
  # Prefer the designated MASTER after repair

# cat > /etc/hostname.em2 <<'EOF'
inet 10.255.255.2 255.255.255.252
EOF
  # State-sync /30

# cat > /etc/hostname.pfsync0 <<'EOF'
up syncdev em2
EOF
  # pfsync over SYNC

# cat > /etc/hostname.em0 <<'EOF'
inet 198.51.100.3 255.255.255.248
EOF
# cat > /etc/hostname.em3 <<'EOF'
inet 203.0.113.3 255.255.255.248
EOF
# cat > /etc/hostname.em1 <<'EOF'
inet 10.10.10.2 255.255.255.0
EOF

# cat > /etc/hostname.carp0 <<'EOF'
inet 198.51.100.2 255.255.255.248 vhid 10 carpdev em0 pass sharedsecret advskew 0
EOF
# cat > /etc/hostname.carp1 <<'EOF'
inet 203.0.113.2 255.255.255.248 vhid 20 carpdev em3 pass sharedsecret advskew 0
EOF
# cat > /etc/hostname.carp2 <<'EOF'
inet 10.10.10.1 255.255.255.0 vhid 30 carpdev em1 pass sharedsecret advskew 0
EOF

# sh /etc/netstart em0 em1 em2 em3 carp0 carp1 carp2 pfsync0
  # Activate without reboot

PF Policy (both nodes) #

## /etc/pf.conf — Edge HA + Multi-WAN

set skip on { lo, pfsync0 }

# Interfaces and macros
lan   = "em1"
wan_a = "em0"
wan_b = "em3"
lan_net = "10.10.10.0/24"
gw_a = "198.51.100.1"
gw_b = "203.0.113.1"
web_svc = "10.10.10.50"

# Normalization
scrub in all max-mss 1452

# Allow HA control-plane
pass on $lan proto carp
pass on $wan_a proto carp
pass on $wan_b proto carp
pass on em2 proto pfsync

# NAT per uplink
match out on $wan_a inet from $lan_net to any nat-to ($wan_a)
match out on $wan_b inet from $lan_net to any nat-to ($wan_b)

# Base policy
block all

# Inbound to firewall itself with symmetric reply
pass in on $wan_a proto { tcp udp icmp } to ($wan_a) reply-to ($wan_a $gw_a) keep state
pass in on $wan_b proto { tcp udp icmp } to ($wan_b) reply-to ($wan_b $gw_b) keep state

# Outbound policy from LAN:
# - Bulk to ISP-B
# - Default to ISP-A
pass in on $lan proto { tcp udp } from $lan_net to any port { 873, 6881:6999 } \
    route-to ($wan_b $gw_b) keep state
pass in on $lan from $lan_net to any \
    route-to ($wan_a $gw_a) keep state

# Inbound service on both ISPs (HTTPS)
rdr on $wan_a proto tcp to ($wan_a) port 443 -> $web_svc
pass in on $wan_a proto tcp to $web_svc port 443 reply-to ($wan_a $gw_a) keep state
rdr on $wan_b proto tcp to ($wan_b) port 443 -> $web_svc
pass in on $wan_b proto tcp to $web_svc port 443 reply-to ($wan_b $gw_b) keep state

# LAN to firewall services (SSH/HTTPS example)
pass in on $lan proto tcp from $lan_net to ($lan) port { 22, 443 } keep state

# Outbound from the firewall
pass out on { $wan_a, $wan_b } proto { tcp udp icmp } from (self) to any keep state

Verification (either node):

$ ifconfig carp0 carp1 carp2 | egrep 'vhid|status'
  # MASTER/BACKUP per VIP
$ pfctl -vvsr | egrep 'route-to|reply-to|proto carp|pfsync'
  # Policy pins and HA allowances
$ pfctl -vvss | head
  # Replicated states present on both nodes
$ tcpdump -ni pfsync0
  # Flow updates on SYNC

Pattern 2 — Campus/Branch Routed Access with Local Services #

Scope. Two routers provide VLAN gateways with CARP, advertise IPv6 via rad(8), and offer internal recursion (Unbound), authoritative zones (NSD), DHCP, and anycast resolver placement.

Gateways and CARP (R1/R2) #

Assume trunk em1, VLAN 10 Users 10.10.10.0/24, VLAN 20 Servers 10.20.20.0/24, VIPs .1, router unicast .2 (R1) / .3 (R2).

# /etc/hostname.vlan10 (R1)
vlandev em1
vlan 10
inet 10.10.10.2 255.255.255.0
up
# /etc/hostname.vlan20 (R1)
vlandev em1
vlan 20
inet 10.20.20.2 255.255.255.0
up
# /etc/hostname.carp0 (R1)
inet 10.10.10.1 255.255.255.0 vhid 10 carpdev vlan10 pass shared advskew 0
# /etc/hostname.carp1 (R1)
inet 10.20.20.1 255.255.255.0 vhid 20 carpdev vlan20 pass shared advskew 0

Mirror on R2 with .3 and advskew 100.

Router Advertisements and DNS on a Services Node #

rad(8) advertises /64s; Unbound serves recursion; NSD serves your zone; DHCP serves IPv4.

## /etc/rad.conf
interface vlan10
    prefix 2001:db8:10:10::/64
    rdnss 2001:db8:10:10::53
    dnssl corp.example
interface vlan20
    prefix 2001:db8:10:20::/64
    rdnss 2001:db8:10:20::53
    dnssl corp.example

## /var/unbound/etc/unbound.conf (excerpt)
server:
    interface: 10.10.10.53
    interface: 10.20.20.53
    access-control: 10.10.10.0/24 allow
    access-control: 10.20.20.0/24 allow
    auto-trust-anchor-file: "/var/unbound/db/root.key"

## /etc/nsd.conf (excerpt)
server:
    ip-address: 10.20.20.53
    zonesdir: "/var/nsd/zones"
zone:
    name: "corp.example"
    zonefile: "corp.example.zone"

## /etc/dhcpd.conf (excerpt)
subnet 10.10.10.0 netmask 255.255.255.0 {
    range 10.10.10.100 10.10.10.200;
    option routers 10.10.10.1;
    option domain-name-servers 10.10.10.53;
}
subnet 10.20.20.0 netmask 255.255.255.0 {
    range 10.20.20.100 10.20.20.200;
    option routers 10.20.20.1;
    option domain-name-servers 10.20.20.53;
}

Anycast Resolver Address #

Publish 10.10.255.53/32 from two resolvers and ECMP it on R1/R2.

# ifconfig lo0 alias 10.10.255.53/32
  # On each resolver

# route add -mpath -host 10.10.255.53 10.20.20.11
# route add -mpath -host 10.10.255.53 10.20.20.12
  # On each router; confirm with 'route -n show'

Verification:

$ ifconfig vlan10 vlan20 carp0 carp1 | egrep 'status|vhid'
$ drill @10.10.10.53 openbsd.org A
$ ping -n -c 3 10.10.255.53
$ ndp -an | head

Pattern 3 — Regional POP #

Two options are common:

A) MPLS LDP Core with L3 VPNs #

Enable MPLS on core links, run ldpd(8), and connect customer VRFs via mpe(4). BGP VPN policy is defined in OpenBGPD.

# ifconfig em0 mpls
# ifconfig em1 mpls
# rcctl enable ldpd ; rcctl start ldpd

## /etc/ldpd.conf (core interfaces)
router-id 192.0.2.1
address-family ipv4 {
    interface em0
    interface em1
}

Bind a customer VRF (rdomain 10) to MPLS with mpe0:

# ifconfig em2 rdomain 10 inet 10.10.10.1/24
# ifconfig mpe0 create
# ifconfig mpe0 rdomain 10 mplslabel 2000 inet 192.198.0.1/32 up

Announce customer prefixes via OpenBGPD VPN AFI/SAFI (see the OpenBGPD chapter for bgpd.conf vpn stanza and iBGP toward other PEs).