Test OpenBGPD with Two Providers

Synopsis #

A BGP session can be established while traffic still fails because of forwarding, firewall policy, or a missing return route. This OpenBSD 8.0 lab connects a client to an HTTP server through two simulated providers. It separates route selection, actual packet delivery, export filtering, and recovery into observable steps.

Use five disposable OpenBSD machines with console access. Connect them only to the four isolated Ethernet segments below. Do not attach these segments to a physical network, host gateway, NAT service, or Internet uplink. All addresses and autonomous system numbers are examples for this closed lab.

The OpenBGPD reference explains broader deployment policy. This exercise uses IPv4 and fresh connections; it does not demonstrate production Internet peering or uninterrupted application sessions.

Example Topology #

Create four separate virtual switches: client LAN, edge–A, edge–B, and the remote segment. The remote segment connects both providers and the server. The customer edge needs three interfaces, each provider needs two, and the client and server need one each.

The server also runs BGP. A static return route through A would prevent replies from following B when A fails. Only the client has a static default route.

Prepare the Lab Machines #

Perform configuration through each machine’s console. The following steps replace network and firewall settings and belong only on disposable lab machines. Interface names must match the assigned virtual hardware.

Stop any inherited BGP service on the edge, both providers, and the server before configuring the lab:

doas rcctl stop bgpd

For each interface in the table, create its corresponding hostname file with the assigned address. For example, the edge’s /etc/hostname.vio1 contains:

inet 10.0.0.2/30

Disable DHCP and remove inherited default routes on all five machines:

doas rcctl stop dhcpleased
doas rcctl disable dhcpleased
doas sh -c ': > /etc/mygate'

If a default route exists, delete it:

doas route -n delete default

For each interface, remove IPv4 autoconfiguration and apply its static address. For example, on the edge:

doas ifconfig vio1 inet -autoconf
doas ifconfig vio1 inet 10.0.0.2/30 up

Repeat for every assigned interface. Remove any unrelated inherited addresses. Set the client’s /etc/mygate to 198.51.100.1 and add its route:

doas route add default 198.51.100.1

Keep /etc/mygate empty on the other four machines. Inspect route -n show -inet: neither the edge, providers, nor server should have a default route or an extra static route to the opposite endpoint.

Enable IPv4 forwarding on the edge and both providers:

doas sysctl net.inet.ip.forwarding=1

Persist net.inet.ip.forwarding=1 in their /etc/sysctl.conf. Leave forwarding disabled on the client and server.

Lab Firewall Policy #

On the edge, use this /etc/pf.conf:

set skip on lo
block all
pass on { vio0, vio1, vio2 } inet

On each provider, limit the pass rule to { vio0, vio1 }. On the client and server, use pass on vio0 inet. Parse and load each machine’s rules:

doas pfctl -nf /etc/pf.conf
doas pfctl -f /etc/pf.conf
doas pfctl -e

If PF is already enabled, the last command reports that state. These permissive IPv4 rules isolate BGP and forwarding behavior within the disconnected lab. There is no NAT. A real deployment needs service-specific firewall rules.

Configure BGP #

Install the following /etc/bgpd.conf on each named machine. The 15-second holdtime and 5-second connection retry shorten this exercise; they are lab settings, not general production recommendations.

OpenBGPD filters deny updates unless an allow rule matches. The edge accepts only the remote network and exports only its client network. Local preference 200 favors A over B at 150. The server applies the same preference to the return path. See bgpd.conf(5) for filter and path-selection rules.

Customer Edge #

AS 64512
router-id 10.0.0.2
holdtime 15
connect-retry 5
listen on 10.0.0.2
listen on 10.0.0.6
network 198.51.100.0/24
neighbor 10.0.0.1 {
 remote-as 64500
 descr "ISP-A"
 local-address 10.0.0.2
 announce IPv4 unicast
 max-prefix 10
}
neighbor 10.0.0.5 {
 remote-as 64496
 descr "ISP-B"
 local-address 10.0.0.6
 announce IPv4 unicast
 max-prefix 10
}
allow from 10.0.0.1 prefix 203.0.113.0/24 set localpref 200
allow from 10.0.0.5 prefix 203.0.113.0/24 set localpref 150
allow to 10.0.0.1 prefix 198.51.100.0/24
allow to 10.0.0.5 prefix 198.51.100.0/24

Provider A #

AS 64500
router-id 10.0.0.1
holdtime 15
connect-retry 5
listen on 10.0.0.1
listen on 203.0.113.1
network 203.0.113.0/24
neighbor 10.0.0.2 {
 remote-as 64512
 descr "Customer"
 local-address 10.0.0.1
 announce IPv4 unicast
 max-prefix 10
}
neighbor 203.0.113.10 {
 remote-as 64513
 descr "Remote-server"
 local-address 203.0.113.1
 announce IPv4 unicast
 max-prefix 10
}
# Lab-only permissive import exposes customer export leaks.
allow from 10.0.0.2
allow to 10.0.0.2 prefix 203.0.113.0/24
allow to 203.0.113.10 prefix 198.51.100.0/24

Provider B #

AS 64496
router-id 10.0.0.5
holdtime 15
connect-retry 5
listen on 10.0.0.5
listen on 203.0.113.2
network 203.0.113.0/24
neighbor 10.0.0.6 {
 remote-as 64512
 descr "Customer"
 local-address 10.0.0.5
 announce IPv4 unicast
 max-prefix 10
}
neighbor 203.0.113.10 {
 remote-as 64513
 descr "Remote-server"
 local-address 203.0.113.2
 announce IPv4 unicast
 max-prefix 10
}
# Lab-only permissive import exposes customer export leaks.
allow from 10.0.0.6
allow to 10.0.0.6 prefix 203.0.113.0/24
allow to 203.0.113.10 prefix 198.51.100.0/24

Remote Server #

AS 64513
router-id 203.0.113.10
holdtime 15
connect-retry 5
listen on 203.0.113.10
neighbor 203.0.113.1 {
 remote-as 64500
 descr "ISP-A"
 local-address 203.0.113.10
 announce IPv4 unicast
 max-prefix 10
}
neighbor 203.0.113.2 {
 remote-as 64496
 descr "ISP-B"
 local-address 203.0.113.10
 announce IPv4 unicast
 max-prefix 10
}
allow from 203.0.113.1 prefix 198.51.100.0/24 set localpref 200
allow from 203.0.113.2 prefix 198.51.100.0/24 set localpref 150

The providers deliberately accept all customer prefixes in this lab. This makes an edge export leak visible at the receiving neighbor. In a real provider configuration, restrict customer imports to the agreed prefixes as well. The providers export only the client network toward the server and the remote network toward the edge.

On each BGP machine, parse the configuration and enable the service:

doas bgpd -n -f /etc/bgpd.conf
doas rcctl enable bgpd

Start an HTTP Endpoint #

On the remote server, use this /etc/httpd.conf:

server "bgp-lab" {
 listen on 203.0.113.10 port 8080
 root "/htdocs/bgp-lab"
}

Create the response and start the service:

doas install -d /var/www/htdocs/bgp-lab
printf '%s\n' 'OpenBGPD lab endpoint' | doas tee /var/www/htdocs/bgp-lab/health.txt
doas httpd -n
doas rcctl enable httpd
doas rcctl restart httpd
doas rcctl check httpd

Verify One Provider, Then Two #

Start BGP on A, the server, and the edge. Leave B stopped initially.

doas rcctl start bgpd
doas rcctl check bgpd
doas bgpctl show summary

Wait for the relevant neighbors to reach Established. Inspect the selected kernel route on each endpoint router:

On the client, test ICMP and retrieve a fresh HTTP response:

ping -n -c 3 -w 5 203.0.113.10
ftp -4 -V -w 5 -o /tmp/bgp-health.txt http://203.0.113.10:8080/health.txt
cat /tmp/bgp-health.txt

The response body should be OpenBGPD lab endpoint. On the server, also test traffic toward the client:

ping -n -c 3 -w 5 198.51.100.10

Start BGP on B. Confirm both providers’ sessions are established on the edge and server. Both routes should still select A. Inspect the alternatives:

doas bgpctl show rib detail

To see which provider carries a fresh HTTP request, run this on each provider in a separate console before fetching the URL again:

doas tcpdump -nli vio0 'tcp port 8080 and host 198.51.100.10 and host 203.0.113.10'

A should show both request and reply packets; B should show neither for that request. Stop the captures with Control-C. An established backup session alone does not prove that the backup carries traffic.

For a separate policy test, change A’s set localpref 200 to set localpref 100 on both the edge and server. Parse and reload both configurations:

doas bgpd -n -f /etc/bgpd.conf
doas bgpctl reload

Both routes should now select B. Repeat ICMP, HTTP, and packet captures. Restore A’s preference to 200 on both machines, parse and reload, and confirm that both directions return to A.

Prove Export Filtering #

An empty outgoing route display proves little if there was no unwanted route to export. Add a deliberate test origin to the edge’s configuration, immediately after the existing network line:

network 192.0.2.0/24

Keep the export rules unchanged. Parse and reload the edge, then confirm that the extra prefix exists locally:

doas bgpd -n -f /etc/bgpd.conf
doas bgpctl reload
doas bgpctl network show inet
doas bgpctl show rib 192.0.2.0/24

Inspect both outgoing views on the edge:

doas bgpctl show rib out neighbor 10.0.0.1
doas bgpctl show rib out neighbor 10.0.0.5

Inspect the receiving side on A:

doas bgpctl show rib in neighbor 10.0.0.2

Inspect the receiving side on B:

doas bgpctl show rib in neighbor 10.0.0.6

Each view should contain exactly 198.51.100.0/24. Neither the injected 192.0.2.0/24 nor the learned 203.0.113.0/24 should be exported by the edge.

Positive Control and Withdrawal #

Only inside this isolated lab, append these two temporary rules to the edge:

allow to 10.0.0.1 prefix 192.0.2.0/24
allow to 10.0.0.5 prefix 192.0.2.0/24

Parse and reload. Both edge outgoing views and both provider incoming views should now contain the client prefix and the test prefix. This confirms that the receiving observations can detect an actual leak.

Remove only the two temporary allow rules, retaining the test origin. Parse and reload again. Wait for the test prefix to disappear from all four views while it remains in the edge’s local RIB. Finally remove the test network line and reload the baseline configuration.

Test a Silent Provider Failure #

Restore A as the preferred provider before this test. Pausing A’s entire virtual machine stops both its control plane and forwarding without sending a graceful BGP shutdown. With OpenBSD VMM, run the following on the hypervisor, substituting the actual lab VM name:

vmctl pause bgp-isp-a

Keep a second hypervisor console available. Confirm that vmctl status bgp-isp-a reports paused: the control client can remain waiting even after the VM state changes. Check B in the same way when pausing it, and confirm running after each unpause. A command timeout alone does not establish the VM state.

Do not pause the edge or server. Wait for their sessions to A to expire and inspect both kernel routes again:

Repeat both ICMP directions and the fresh HTTP request. Capture on B to confirm request and reply packets cross its customer-facing interface.

The holdtime starts from the last received BGP message, not from the command that pauses the VM. Route installation and traffic recovery take additional time. Packet loss during detection is expected; successful new requests after convergence do not imply that an existing connection survives.

Remove Both Providers #

While A remains paused, pause B on the hypervisor:

vmctl pause bgp-isp-b

After the remaining BGP sessions expire, the edge should have no route to the remote endpoint and the server should have no route to the client. ICMP and HTTP from the client should fail. Success at this stage indicates an unintended route or network connection and invalidates the isolation check.

Resume B first:

vmctl unpause bgp-isp-b

Wait for BGP and both routes to recover through B, then repeat the traffic tests. Resume A:

vmctl unpause bgp-isp-a

Wait for both routes to prefer A again. Repeat ICMP, HTTP, and the two-sided export inspection. Ensure every paused VM is resumed before shutting down the lab.

Apply the Results #

This scenario demonstrates independent forward and return path selection, exact edge export policy, traffic over each provider, silent-failure recovery, and loss of reachability when both providers disappear. It does not model the global routing table, external provider policies, RPKI, IPv6, or redundant customer routers.

For a production design, use assigned addresses and AS numbers, explicit customer and provider import filters, appropriate prefix limits, suitable session protection, and operationally chosen timers. Agree on inbound traffic policy with the actual providers; setting local preference on the lab server is possible here because the entire remote side is under local control.