Multi-WAN and Policy-Based Routing

Synopsis #

This chapter shows how to operate an OpenBSD edge with two or more upstream providers. It covers outbound policy using route-to, symmetric return paths with reply-to, per-interface NAT, and inbound publishing of services on multiple providers. Configuration lives in pf.conf(5) and is managed with pfctl(8) . Interface attributes are managed with ifconfig(8) . Where automated failover is required, integrate link and reachability checks with ifstated(8) . Use this pattern when you must split or steer traffic across providers, or need deterministic return paths for inbound NAT without using BGP.

Design Considerations #

  • Topology. Use distinct physical interfaces per WAN. Example: em0 for ISP-A and em3 for ISP-B. Keep a clear LAN interface for policy classification.
  • Addressing and gateways. Each WAN has its own gateway. Example: ISP-A 198.51.100.1, ISP-B 203.0.113.5.
  • Asymmetry. Stateful filtering and NAT tie flows to the egress that created the state. Changing egress requires new states. Plan to clear affected states during failover.
  • NAT alignment. Private IPv4 source addresses need translation appropriate to the selected WAN; globally routed space may not need NAT. Avoid a single egress NAT when you intend to steer flows across multiple egresses.
  • Policy granularity. Classify by source networks, application ports, or destinations. Simpler is better. Start with per-subnet policies.
  • Inbound services. Without BGP or external DNS steering, inbound presence is per-provider. Use reply-to to force responses back out the interface that received the connection.
  • Monitoring and automation. Detect failure of a specific provider before changing policy. Use ifstated(8) to trigger clean policy reloads and optional state resets.
  • MTU. Mixed-access technologies (for example, PPPoE vs. Ethernet) can introduce fragmentation. Consider a conservative scrub with MSS clamping if you observe PMTUD problems.

Configuration #

The example below implements two uplinks with deterministic outbound and inbound behavior. Adjust interface names and addresses to match your environment.

Interface assumptions #

  • em0 — ISP-A, address 198.51.100.2/29, gateway 198.51.100.1
  • em3 — ISP-B, address 203.0.113.6/29, gateway 203.0.113.5
  • em1 — LAN, 10.10.10.0/24 users and servers
  • Optional server to publish inbound: 10.10.10.50 (HTTPS)

PF policy with per-WAN NAT, route-to, and reply-to #

Place the following in /etc/pf.conf. Load and inspect with pfctl(8) . Syntax is defined in pf.conf(5) .

## /etc/pf.conf — Multi-WAN and policy-based routing (example)

set skip on lo

# Interfaces and networks
lan      = "em1"
wan_a    = "em0"
wan_b    = "em3"
lan_net  = "10.10.10.0/24"

# Upstream gateways
gw_a = "198.51.100.1"
gw_b = "203.0.113.5"

# Optional convenience macros
web_svc = "10.10.10.50"
tcp_services = "{ 22, 80, 443 }"

# Conservative scrub; adjust as required
match in all scrub (max-mss 1452)

# Per-interface NAT for each uplink
match out on $wan_a inet from $lan_net to any nat-to ($wan_a)
match out on $wan_b inet from $lan_net to any nat-to ($wan_b)

# Base policy
block all

# Inbound control-plane to the firewall itself (per-WAN reply-to for symmetry)
pass in on $wan_a proto icmp to ($wan_a) \
    reply-to $gw_a keep state
pass in on $wan_b proto icmp to ($wan_b) \
    reply-to $gw_b keep state

# Outbound policy from LAN:
# - Default via ISP-A
# - Specific subnet or application class via ISP-B
# Order matters: specific policies first, then the default.

# Example: send a source subnet (or specific hosts) via ISP-B
# Replace 10.10.20.0/24 with a real subset if applicable.
# pass in quick on $lan from 10.10.20.0/24 to any \
#     route-to $gw_b keep state

# Example: send bulk/backup traffic via ISP-B (by destination ports)
pass in quick on $lan proto { tcp, udp } from $lan_net to any port { 6881:6999, 873 } \
    route-to $gw_b keep state

# Default: everything else goes via ISP-A
pass in on $lan from $lan_net to any \
    route-to $gw_a keep state

# Inbound publishing of a HTTPS service on both providers
# rdr occurs before filtering; provide a pass rule that includes reply-to
# so return traffic follows the same ISP it arrived on.

# ISP-A publication
pass in on $wan_a proto tcp to ($wan_a) port 443 \
    reply-to $gw_a rdr-to $web_svc

# ISP-B publication
pass in on $wan_b proto tcp to ($wan_b) port 443 \
    reply-to $gw_b rdr-to $web_svc

# Allow LAN to firewall services as needed (SSH/HTTPS example)
pass in on $lan proto tcp from $lan_net to ($lan) port $tcp_services keep state

# Outbound from the firewall itself (administration, package fetches)
pass out on $wan_a inet to any keep state
pass out on $wan_b inet to any keep state

Notes #

  • route-to applies where the packet enters PF. For traffic originating on the LAN, match it with pass in on $lan ... route-to $gateway.
  • reply-to on the WAN-side pass rules pins return traffic for inbound sessions to the same interface and gateway that received the connection.
  • NAT rules are written per uplink so that translation follows the egress selected by policy.

Operational toggles and state hygiene #

These rules do not implement automatic failover. Prepare and validate an alternate policy that selects the surviving gateway before resetting affected states. Enable IPv4 forwarding and configure reachable upstream gateways. During planned failover or while testing, it is useful to adjust policy and reset only the flows that must move.

doas pfctl -f /etc/pf.conf
  # Atomic reload after any change

doas ifconfig em0 down
  # Simulate ISP-A failure (bring the interface back up after testing with: ifconfig em0 up)

doas pfctl -k 0.0.0.0/0 -k 0.0.0.0/0
  # Flush all IPv4 states so new paths can be chosen
doas pfctl -k 10.10.10.0/24
  # Or selectively kill states for the LAN prefix only

doas tcpdump -ni em3 icmp or port 443
  # Observe traffic moving to ISP-B during a test

If you automate uplink health detection with ifstated(8) , make sure the actions include a clean pfctl -f of a prepared policy and a targeted state reset for affected prefixes or applications.

Verification #

Use PF counters, live captures, and system routes to verify that selection and symmetry work as intended.

  • Rule counters and states with pfctl(8) :
doas pfctl -vvsr
  # Verify that specific "route-to" rules are matching and increasing counters
doas pfctl -vvss | egrep 'em0|em3'
  # Inspect states; egress interface and gateway selection appear in state details

traceroute -n 1.1.1.1
  # From a LAN host through the firewall; expect ISP-A by default
traceroute -n 9.9.9.9
  # For a destination matched by a policy to ISP-B, expect the ISP-B path
ping -n -c 3 8.8.8.8
  # Basic reachability; repeat while toggling policies

doas tcpdump -ni em0 host 8.8.8.8 or port 443
  # Confirm flows selected for ISP-A egress
doas tcpdump -ni em3 host 9.9.9.9 or port 443
  # Confirm flows selected for ISP-B egress

route -n show -inet
  # The system default route is not authoritative for PF "route-to" decisions,
  # but it matters for traffic sourced by the firewall itself

Troubleshooting #

  • Asymmetric return paths. Missing or incorrect reply-to on WAN pass rules for inbound services causes replies to leave via the wrong ISP. Add reply-to on each WAN.
  • NAT mismatch. If clients on LAN lose connectivity when a policy switches egress, ensure per-interface match out on $wan_* nat-to ($wan_*) rules exist for each uplink.
  • Flows stick to old egress. Existing states pin to the original path. Kill only the affected states using pfctl -k <subnet> or perform a targeted application restart.
  • Performance issues after path change. MTU or MSS differences between providers can break PMTUD. Keep a conservative scrub ... max-mss during testing and refine later.
  • Policy does not match. Rule ordering matters. Place specific selectors before the default LAN routing rule. Confirm with pfctl -vvsr.
  • Firewall-originated traffic uses the wrong ISP. The illustrated ingress route-to rules do not match traffic sourced by the firewall. Outbound rules can apply route-to to that traffic. Adjust the system default route with route(8) or add application-specific source addresses.

See Also #

Report a bug