Reference Architectures

Synopsis #

The following three reference designs combine the components from earlier chapters:

Redundant Internet Edge with carp(4) , pfsync(4) , per-uplink NAT and policy routing in pf.conf(5) , managed by pfctl(8) .
Campus/Branch Routed Access using VLAN SVIs, Router Advertisements with rad(8) , internal recursion via unbound(8) , authoritative zones with nsd(8) , and DHCP with dhcpd(8) .
Regional POP with either an MPLS LDP core via ldpd(8) and mpe(4) , or an encrypted hub-and-spoke using iked(8) or wg(4) . BGP policy lives in the OpenBGPD chapter.

Each pattern includes configuration excerpts that require integration with the linked chapters and verification guidance.

Design Considerations #

Failure domains. Keep state sync and control traffic on a private link. Avoid bridging across routers unless strictly required.
Configuration parity. Maintain identical PF and service configs on HA pairs; only device-specific interface addressing should differ.
Change management. Use atomic PF reloads and a rollback timer during first deployment. Keep a failsafe anchor that preserves management access.
Addressing plan. Allocate deterministic per-VLAN /24 (IPv4) and /64 (IPv6) with structured mapping to sites and roles.
Observability. Enable pflogd(8) and (optionally) pflow(4) toward a collector; keep clocks accurate via ntpd(8) .

Pattern 1 — Redundant Internet Edge (Two Firewalls, Two ISPs) #

Scope. Two OpenBSD firewalls in HA; two WAN uplinks (ISP-A and ISP-B); deterministic outbound steering and symmetric inbound using route-to and reply-to.

Topology and Interface Assumptions #

em0 = ISP-A, em3 = ISP-B, em1 = LAN, em2 = SYNC
CARP VIPs: WAN-A 198.51.100.2/29 (vhid 10), WAN-B 203.0.113.2/29 (vhid 20), LAN 10.10.10.1/24 (vhid 30)
Node A preferred (advskew 0), Node B backup (advskew 100)

System and Interfaces (Node A; mirror with advskew 100 on Node B) #

doas sh <<'ROOT'
printf '%s\n' 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
sysctl net.inet.carp.preempt=1
  # Prefer the designated MASTER after repair

cat > /etc/hostname.em2 <<'EOF'
inet 10.255.255.2 255.255.255.252
EOF
  # State-sync /30

cat > /etc/hostname.pfsync0 <<'EOF'
syncdev em2
up
EOF
  # pfsync over SYNC

cat > /etc/hostname.em0 <<'EOF'
inet 198.51.100.3 255.255.255.248
EOF
cat > /etc/hostname.em3 <<'EOF'
inet 203.0.113.3 255.255.255.248
EOF
cat > /etc/hostname.em1 <<'EOF'
inet 10.10.10.2 255.255.255.0
EOF

cat > /etc/hostname.carp0 <<'EOF'
inet 198.51.100.2 255.255.255.248 NONE vhid 10 carpdev em0 pass sharedsecret advskew 0
EOF
cat > /etc/hostname.carp1 <<'EOF'
inet 203.0.113.2 255.255.255.248 NONE vhid 20 carpdev em3 pass sharedsecret advskew 0
EOF
cat > /etc/hostname.carp2 <<'EOF'
inet 10.10.10.1 255.255.255.0 NONE vhid 30 carpdev em1 pass sharedsecret advskew 0
EOF

sh /etc/netstart em0 em1 em2 em3 carp0 carp1 carp2 pfsync0
  # Activate without reboot
ROOT

Node B needs distinct physical addresses: WAN-A .4, WAN-B .4, LAN .3, and SYNC 10.255.255.1/30. Keep the VIPs identical. Enable IPv4 forwarding and protect both nodes’ CARP secrets with mode 600.

PF Policy (both nodes) #

## /etc/pf.conf — Edge HA + Multi-WAN

set skip on lo

# Interfaces and macros
lan   = "em1"
wan_a = "em0"
wan_b = "em3"
lan_net = "10.10.10.0/24"
gw_a = "198.51.100.1"
gw_b = "203.0.113.1"
web_svc = "10.10.10.50"

# Normalization
match in all scrub (max-mss 1452)

# Base policy
block all

# Allow HA control-plane
pass on $lan proto carp
pass on $wan_a proto carp
pass on $wan_b proto carp
pass on em2 proto pfsync

# NAT per uplink
match out on $wan_a inet from $lan_net to any nat-to 198.51.100.2
match out on $wan_b inet from $lan_net to any nat-to 203.0.113.2

# Inbound to firewall itself with symmetric reply
pass in on $wan_a proto icmp to ($wan_a) reply-to $gw_a keep state
pass in on $wan_b proto icmp to ($wan_b) reply-to $gw_b keep state

# Outbound policy from LAN:
# - Bulk to ISP-B
# - Default to ISP-A
pass in quick on $lan proto { tcp udp } from $lan_net to any port { 873, 6881:6999 } \
    route-to $gw_b keep state
pass in on $lan from $lan_net to any \
    route-to $gw_a keep state

# Inbound service on both ISPs (HTTPS)
pass in on $wan_a proto tcp to 198.51.100.2 port 443 \
    reply-to $gw_a rdr-to $web_svc
pass in on $wan_b proto tcp to 203.0.113.2 port 443 \
    reply-to $gw_b rdr-to $web_svc

# LAN to firewall services (SSH/HTTPS example)
pass in on $lan proto tcp from $lan_net to ($lan) port { 22, 443 } keep state

# Outbound from the firewall
pass out on { $wan_a, $wan_b } proto { tcp udp icmp } from (self) to any keep state

Verification (either node):

ifconfig carp0
ifconfig carp1
ifconfig carp2 | egrep 'vhid|status'
  # MASTER/BACKUP per VIP
doas pfctl -vvsr | egrep 'route-to|reply-to|proto carp|pfsync'
  # Policy pins and HA allowances
doas pfctl -vvss | head
  # Replicated states present on both nodes
doas tcpdump -ni pfsync0
  # Flow updates on SYNC

Pattern 2 — Campus/Branch Routed Access with Local Services #

Scope. Two routers provide VLAN gateways with CARP, advertise IPv6 via rad(8), and offer internal recursion (Unbound), authoritative zones (NSD), DHCP, and anycast resolver placement.

Gateways and CARP (R1/R2) #

Assume trunk em1, VLAN 10 Users 10.10.10.0/24, VLAN 20 Servers 10.20.20.0/24, VIPs .1, router unicast .2 (R1) / .3 (R2).

# /etc/hostname.vlan10 (R1)
parent em1
vnetid 10
inet 10.10.10.2 255.255.255.0
up
# /etc/hostname.vlan20 (R1)
parent em1
vnetid 20
inet 10.20.20.2 255.255.255.0
up
# /etc/hostname.carp0 (R1)
inet 10.10.10.1 255.255.255.0 NONE vhid 10 carpdev vlan10 pass shared advskew 0
# /etc/hostname.carp1 (R1)
inet 10.20.20.1 255.255.255.0 NONE vhid 20 carpdev vlan20 pass shared advskew 0

Mirror on R2 with .3 and advskew 100. Bring up the physical VLAN parent and permit CARP and the required client traffic in PF.

Router Advertisements and DNS on a Services Node #

rad(8) advertises /64s; Unbound serves recursion; NSD serves your zone; DHCP serves IPv4.

## /etc/rad.conf
interface vlan10 {
    prefix 2001:db8:10:10::/64
    dns { nameserver 2001:db8:10:10::53
          search corp.example
    }
}
interface vlan20 {
    prefix 2001:db8:10:20::/64
    dns { nameserver 2001:db8:10:20::53
          search corp.example
    }
}

## /var/unbound/etc/unbound.conf (excerpt)
server:
    interface: 10.10.10.53
    interface: 10.20.20.53
    access-control: 10.10.10.0/24 allow
    access-control: 10.20.20.0/24 allow
    auto-trust-anchor-file: "/var/unbound/db/root.key"

## /var/nsd/etc/nsd.conf (excerpt)
server:
    ip-address: 10.20.20.54
    zonesdir: "/var/nsd/zones"
zone:
    name: "corp.example"
    zonefile: "corp.example.zone"

## /etc/dhcpd.conf (excerpt)
subnet 10.10.10.0 netmask 255.255.255.0 {
    range 10.10.10.100 10.10.10.200;
    option routers 10.10.10.1;
    option domain-name-servers 10.10.10.53;
}
subnet 10.20.20.0 netmask 255.255.255.0 {
    range 10.20.20.100 10.20.20.200;
    option routers 10.20.20.1;
    option domain-name-servers 10.20.20.53;
}

Anycast Resolver Address #

Publish 10.10.255.53/32 from two resolvers and ECMP it on R1/R2.

Follow the anycast resolver setup : use IPv4 receive mode 2 on standalone resolvers, keep forwarding mode 1 on routers, enable router multipath, and limit DNS access to approved clients.

doas ifconfig lo0 alias 10.10.255.53/32
  # On each resolver

doas route add -mpath -host 10.10.255.53 10.20.20.11
doas route add -mpath -host 10.10.255.53 10.20.20.12
  # On each router; confirm with 'route -n show'

Verification:

ifconfig vlan10
ifconfig vlan20
ifconfig carp0
ifconfig carp1 | egrep 'status|vhid'
drill @10.10.10.53 openbsd.org A
ping -n -c 3 10.10.255.53
ndp -an | head

Pattern 3 — Regional POP #

Two options are common:

A) MPLS LDP Core with L3 VPNs #

Enable MPLS on core links, run ldpd(8), and connect customer VRFs via mpe(4) . BGP VPN policy is defined in OpenBGPD.

doas ifconfig em0 mpls
doas ifconfig em1 mpls

## /etc/ldpd.conf (core interfaces)
router-id 192.0.2.1
address-family ipv4 {
    interface em0
    interface em1
}

After writing /etc/ldpd.conf, check it before enabling and starting LDP:

doas ldpd -n -f /etc/ldpd.conf
doas rcctl enable ldpd
doas rcctl start ldpd

Bind a customer VRF (rdomain 10) to MPLS with mpe0:

doas ifconfig em2 rdomain 10
doas ifconfig em2 inet 10.10.10.1/24
doas ifconfig mpe0 create
doas ifconfig mpe0 rdomain 10 mplslabel 2000
doas ifconfig mpe0 inet 192.0.2.100/32 up

Announce customer prefixes via OpenBGPD VPN AFI/SAFI (see the OpenBGPD chapter for bgpd.conf vpn stanza and iBGP toward other PEs).