Synopsis #
Configure two IPv4 gateways with shared addresses on the client and backend networks. Start with one active gateway and one standby. After routing and failover work, add CARP IP balancing if both gateways need to carry traffic.
CARP IP balancing distributes traffic across gateways based on the source and destination IP addresses. It leaves the destination server unchanged. To select servers from a backend pool, use NAT or DSR . To route by HTTP hostname or path, use relayd proxying .
Use console access while changing interfaces and filtering. Replace the example interfaces and addresses with those for the intended network. The sync network must be private to the pair or carried inside a protected transport.
Assign Addresses #
| Role | Frontend, vio0 | Backend, vio1 | Sync, vio2 |
|---|---|---|---|
| Gateway A | 198.51.100.2/24 | 10.10.10.2/24 | 10.255.0.1/30 |
| Gateway B | 198.51.100.3/24 | 10.10.10.3/24 | 10.255.0.2/30 |
| Shared address | 198.51.100.100 | 10.10.10.1 | No shared address |
The client is 198.51.100.10/24. Its route to 10.10.10.0/24 uses 198.51.100.100. Backends 10.10.10.11 and 10.10.10.12 send replies to the client network through 10.10.10.1. The frontend shared address also serves the linked NAT, DSR and HTTPS examples.
Configure the three physical interfaces persistently with hostname.if(5)
. Give each gateway its own physical interface addresses. Add or update these entries in /etc/sysctl.conf on both gateways:
net.inet.ip.forwarding=1
net.inet.carp.preempt=1
Apply them now:
doas sysctl net.inet.ip.forwarding=1 net.inet.carp.preempt=1
Configure Active and Standby #
On A, create /etc/hostname.carp0:
inet 198.51.100.100/24 vhid 101 carpdev vio0 pass lb-example advskew 0
Create /etc/hostname.carp1:
inet 10.10.10.1/24 vhid 102 carpdev vio1 pass lb-example advskew 0
On B, use the same addresses, VHIDs and passphrase, with advskew 100 in both files. Replace lb-example with a private passphrase, using the same value on both gateways. Protect the files:
doas chmod 600 /etc/hostname.carp0 /etc/hostname.carp1
On A, /etc/hostname.pfsync0 contains:
syncdev vio2 syncpeer 10.255.0.2 defer
On B, use syncpeer 10.255.0.1. The defer option briefly queues an initial packet while its new state is synchronized, which is useful when either firewall may receive traffic.
Bring up these interfaces on each node:
doas sh /etc/netstart carp0 carp1 pfsync0
Permit Control and Routed Traffic #
Merge the following into the pair’s /etc/pf.conf. This example allows traffic between the client and backend networks. Restrict the rules to the required services before using them on a live network. The relayd/* anchor is used by the linked NAT and DSR guide.
set skip on lo
block all
pass quick on vio2 proto pfsync keep state (no-sync)
pass quick on { vio0 vio1 } proto carp keep state (no-sync)
anchor "relayd/*" quick
pass in on vio0 inet proto tcp to 198.51.100.100 port { 80 443 }
pass in on vio0 inet from 198.51.100.0/24 to 10.10.10.0/24
pass in on vio1 inet from 10.10.10.0/24 to 198.51.100.0/24
pass out on { vio0 vio1 } inet
Validate and load the rules on both nodes:
doas pfctl -nf /etc/pf.conf
doas pfctl -f /etc/pf.conf
The quick anchor prevents later general pass rules from overriding the forwarding decision in a matching relayd rule. DSR needs this to preserve its route-to action.
PF must be enabled. Existing gateways normally already have it enabled; confirm with doas pfctl -si. Keep the pair’s rules and interface names consistent. pfsync synchronizes states, not configuration files.
Verify Routing and Takeover #
Check ifconfig carp0 and ifconfig carp1 on both nodes. A should be MASTER for both shared addresses and B should be BACKUP. Fetch a known resource from 10.10.10.11 on the client, then inspect doas pfctl -ss on both gateways. Look for the same client/service connection, not merely a nonempty state table.
Keep a file transfer or service connection open from the client, then temporarily lower A’s CARP priority:
doas ifconfig -g carp carpdemote 128
Check that B takes over both shared addresses and that new requests work. Check that the original connection continues without reconnecting. Restore only the demotion increment added for this test:
doas ifconfig -g carp -carpdemote 128
Also test a shutdown of the active gateway. Check client traffic as well as the CARP roles. If a gateway loses access to the backend network while CARP remains active, monitoring may need to lower its CARP priority; see the HA reference .
Share Forwarding with CARP IP Balancing #
After the active/standby pair works, CARP can distribute address pairs across both nodes. Replace the CARP configuration with matching balancing groups on both sides. Use VHIDs that do not conflict with another CARP service on either segment.
On A, the frontend file becomes:
inet 198.51.100.100/24 carpdev vio0 pass lb-example carpnodes 201:0,202:100 balancing ip
The backend file uses the same carpnodes list with address 10.10.10.1/24 and parent vio1. On B, use carpnodes 201:100,202:0 in both files. Restart the CARP interfaces from the console during a maintenance window to apply the change.
Changing the balancing mode also changes the virtual MAC address. Check the adjacent client and backend gateway ARP entries after the transition. If they still contain the old MAC address, refresh the affected gateway entries or wait for them to expire. Restarting CARP may leave stale entries on neighboring machines.
The source and destination IP addresses determine which gateway accepts a packet. Test with several client addresses: connections from one client to one server use the same gateway. Capture the same routed service traffic on both gateways and verify the reverse traffic uses the corresponding node.
The switches must deliver frames for the shared multicast MAC address to both gateways. Check multicast handling and filtering on the switches that will carry the traffic. IP balancing does not increase a single connection’s bandwidth or check application health, and traffic volume may differ between gateways. Use active/standby when one gateway can handle the load.
Troubleshooting #
- Both nodes claim the same role: inspect VHIDs, passphrases, CARP advertisements and the L2 path.
- Only one direction works: inspect client and backend routes before changing PF state policy.
- No replicated connection: check the dedicated sync link,
syncpeer, filtering and ruleset compatibility. - Takeover succeeds but the service fails: inspect the destination service and both data paths. pfsync cannot repair a failed backend or copy application state.
- Only one gateway receives traffic in IP balancing mode: check that the switches deliver frames for the shared multicast MAC address to both gateways.