High Availability and State Replication

Synopsis #

Build a two-node OpenBSD firewall or gateway with carp(4) virtual IPs, pfsync(4) state replication, and optional service failover using relayd(8) . Configure inside and outside virtual IPs (VIPs) and a dedicated state-sync network for failover. Use this pattern whenever an outage of a single gateway or firewall would impact production traffic. Packet filtering is configured in pf.conf(5) and managed at runtime with pfctl(8) .

Design Considerations #

  • Topology. Use three physical interfaces per node: WAN, LAN, and SYNC. Place pfsync(4) on a private L2 segment with no other traffic.
  • Addressing. Assign each node a fixed unicast address on WAN and LAN, and create per-segment CARP VIPs with unique VHIDs. Example:
    • WAN: node A 203.0.113.2/29, node B 203.0.113.3/29, VIP 203.0.113.1/29, VHID 10
    • LAN: node A 10.10.10.2/24, node B 10.10.10.3/24, VIP 10.10.10.1/24, VHID 20
  • Priority and preemption. Lower advskew wins. Enable net.inet.carp.preempt=1 so the preferred node retakes MASTER after repair.
  • Isolation and filtering. Allow proto carp on member interfaces and proto pfsync only on the SYNC interface. Filter pfsync on its physical SYNC interface.
  • Failure domains. A SYNC link failure must trigger CARP demotion so a healthy peer becomes MASTER. Monitor upstream reachability if asymmetric failures are possible.
  • Configuration parity. Keep identical PF and service configuration on both nodes. States replicate; rules do not.
  • Operational guardrails. Test failover during maintenance windows. Log CARP events at an elevated level during rollout.

Configuration #

The examples assume interface names em0 (WAN), em1 (LAN), and em2 (SYNC). Adjust to your hardware. Interface files are described in hostname.if(5) . Interface configuration and CARP parameters are managed with ifconfig(8) . System controls are set with sysctl(8) and persisted via sysctl.conf(5) . Services are managed with rcctl(8) .

System settings (both nodes) #

doas sh <<'ROOT'
printf '%s\n' \
    'net.inet.carp.preempt=1' \
    'net.inet.carp.log=2' \
    >> /etc/sysctl.conf
  # Enable CARP preemption and retain state-change logging; persists across reboots

sysctl net.inet.carp.preempt=1 net.inet.carp.log=2
  # Apply immediately at runtime
ROOT

Enable IP forwarding for the routed address families and configure the upstream route on both nodes. Replace sharedsecret with the same strong secret on both peers and protect the CARP hostname files with mode 600. NAT, if required, must use the shared WAN VIP so it remains valid after failover.

Put NONE between the netmask and vhid in each IPv4 hostname-file entry so the system calculates the broadcast address. Without it, vhid is read as an address.

Node A (preferred MASTER) #

doas sh <<'ROOT'
cat > /etc/hostname.em0 <<'EOF'
inet 203.0.113.2 255.255.255.248
EOF
  # WAN unicast on em0

cat > /etc/hostname.em1 <<'EOF'
inet 10.10.10.2 255.255.255.0
EOF
  # LAN unicast on em1

cat > /etc/hostname.em2 <<'EOF'
inet 10.255.255.1 255.255.255.252
EOF
  # SYNC network on em2 (private /30)

cat > /etc/hostname.carp0 <<'EOF'
inet 203.0.113.1 255.255.255.248 NONE vhid 10 carpdev em0 pass sharedsecret advskew 0
EOF
  # WAN VIP: VHID 10, lowest advskew to prefer MASTER

cat > /etc/hostname.carp1 <<'EOF'
inet 10.10.10.1 255.255.255.0 NONE vhid 20 carpdev em1 pass sharedsecret advskew 0
EOF
  # LAN VIP: VHID 20

cat > /etc/hostname.pfsync0 <<'EOF'
syncdev em2
up
EOF
  # pfsync over em2; multicast by default on the SYNC segment

sh /etc/netstart em0 em1 em2 carp0 carp1 pfsync0
  # Bring interfaces up now without reboot
ROOT

Node B (BACKUP) #

doas sh <<'ROOT'
cat > /etc/hostname.em0 <<'EOF'
inet 203.0.113.3 255.255.255.248
EOF

cat > /etc/hostname.em1 <<'EOF'
inet 10.10.10.3 255.255.255.0
EOF

cat > /etc/hostname.em2 <<'EOF'
inet 10.255.255.2 255.255.255.252
EOF

cat > /etc/hostname.carp0 <<'EOF'
inet 203.0.113.1 255.255.255.248 NONE vhid 10 carpdev em0 pass sharedsecret advskew 100
EOF
  # Higher advskew so this node remains BACKUP while A is healthy

cat > /etc/hostname.carp1 <<'EOF'
inet 10.10.10.1 255.255.255.0 NONE vhid 20 carpdev em1 pass sharedsecret advskew 100
EOF

cat > /etc/hostname.pfsync0 <<'EOF'
syncdev em2
up
EOF

sh /etc/netstart em0 em1 em2 carp0 carp1 pfsync0
ROOT

PF policy (both nodes) #

The following minimal policy enables pfsync and CARP traffic, filters the physical SYNC interface, and allows all on LAN while blocking by default on WAN. Edit to fit your security model. Syntax is defined in pf.conf(5) . Load and inspect the rules with pfctl(8) .

## /etc/pf.conf (minimal HA skeleton)

set skip on lo

# Interfaces
wan = "em0"
lan = "em1"
sync = "em2"

# Default deny, followed by control-plane exceptions
block all

# Allow control-plane for HA
pass on $wan proto carp
pass on $lan proto carp
pass on $sync proto pfsync

# Example policy (tighten for production)
pass on $lan
pass out on $wan proto { tcp udp icmp } received-on $lan keep state
pass out on $wan proto { tcp udp icmp } from ($wan) to any keep state

The received-on $lan rule allows LAN traffic to leave through the WAN. The from ($wan) rule allows the router’s own traffic, not forwarded LAN traffic.

doas pfctl -f /etc/pf.conf
  # Atomic reload of rules
doas pfctl -sr
  # Verify rules include proto carp and proto pfsync

Optional: L4 VIP for a service using relayd (both nodes) #

The following is a TCP proxy: relayd accepts the client connection and opens a separate backend connection. pfsync does not replicate these sockets. For complete scenarios and their different connection-failover behavior, see NAT and DSR , HTTPS proxying , and gateway sharing .

Use relayd.conf(5) to front-end a service VIP on the LAN side that fails over with CARP. Manage the daemon with rcctl(8) and runtime with relayctl(8).

## /etc/relayd.conf (simple TCP health-checked relay)
table <web_backends> { 10.10.10.11, 10.10.10.12 }

relay "www" {
    listen on 10.10.10.1 port 80
    forward to <web_backends> port 80 check tcp
}

doas rcctl enable relayd
  # Start on boot
doas rcctl start relayd
  # Launch now

Optional: event-driven demotion on SYNC or upstream loss #

In addition to automatic kernel demotion on certain failures, operators often demote CARP when SYNC breaks or when upstream reachability fails. The simplest operational control is to adjust advskew on all CARP VIPs of the affected node. The following example lowers a node’s preference or restores its preferred priority. A healthy reachable peer and compatible preemption policy are required for takeover:

doas ifconfig carp0 advskew 240
  # Heavily demote WAN VIP on this node

doas ifconfig carp1 advskew 240
  # Heavily demote LAN VIP on this node

doas ifconfig carp0 advskew 0
  # Restore preferred priority for WAN VIP

doas ifconfig carp1 advskew 0
  # Restore preferred priority for LAN VIP

Automating this decision can be done with policy tooling such as ifstated(8) (not shown here; refer to its manual for condition syntax) that triggers demotion on link or reachability failure.

Verification #

  • Interface roles. Confirm MASTER/BACKUP on each CARP VIP:
ifconfig carp0
  # Shows "MASTER" or "BACKUP" and VHID/advskew
ifconfig carp1
ifconfig pfsync0
  # Confirms syncdev em2 and up status
  • State replication. Start a connection through the cluster, then verify states exist on both nodes with pfctl(8) :
doas pfctl -vvss | head
  # Show replicated states with creators and age/counters
  • Wire-level sync. Observe pfsync traffic on the SYNC interface with tcpdump(8) :
doas tcpdump -n -e -ttt -i pfsync0
  # Expect steady state updates; bursts during new flows and failover
  • CARP health. Inspect CARP sysctls with sysctl(8) :
sysctl net.inet.carp
  # Shows allow, preempt, log level, and log level; use `ifconfig -g carp` for group demotion
  • Runtime dashboards. Use systat(1) pf view for live counters, and check /var/log/messages for CARP role changes.

Troubleshooting #

  • MASTER/MASTER condition. Two nodes both claim MASTER on a VLAN. Check for VHID collisions, mismatched pass, or isolated broadcast domains. Inspect ifconfig carpX and correct VHID/password.
  • No state replication. Verify pfsync0 is up with the correct syncdev, PF is enabled, and proto pfsync is permitted only on the SYNC interface. Confirm with tcpdump -i pfsync0.
  • Flapping roles. Upstream instability or asymmetric paths can cause preemption loops. Temporarily raise advskew on the unstable node, investigate physical links, and consider disabling preemption until repaired.
  • Failover works, but connections drop. Ensure rulesets are identical on both nodes and NAT settings match. Confirm states replicate (pfctl -vvss) and that hosts on LAN receive gratuitous ARP from CARP during role change.
  • Relayd not serving after failover. Ensure relayd runs on both nodes, listens on the CARP VIP, and that the health checks match the service. Review /var/log/daemon and relayctl show relays.
  • Demotion stuck high. Inspect ifconfig -g carp. Find and resolve the underlying cause (for example, SYNC down). Clear the underlying fault; lowering advskew does not clear a group demotion counter.

See Also #

Report a bug