Paul Rosham

OPNsense: announce your own prefix with FRR BGP

In short. Install os-frr, enable BGP with your own AS and a router ID per node, and peer with the provider from each node's own WAN address, never from the CARP VIP. Add a static route for the prefix to Null0 on each node so the network statement has something to advertise. Accept only a default route in, send only your prefix out. Then check a public looking glass, not just vtysh.

BGP session is established but my prefix is not advertised

This is the most common question, and the answer is almost always the missing blackhole route. FRR only advertises a network statement if the prefix is in the local routing table. Your /24 is split across VLANs and NAT; the whole /24 is not a route anywhere on the firewall until you add one. A session can sit in Established for a week advertising nothing.

The second most common question is "why does my session drop on failover", and the answer is that it was built from the VIP.

What you need

Steps

  1. Enable routing. Routing > General, tick Enable, save. This starts zebra and lets the GUI pages for BGP appear.

  2. Add the blackhole route on each node. The FRR plugin has a static routes section under Routing > General; add 203.0.113.0/24 with Null0 (blackhole) as the next hop. If your plugin version lacks the field, the equivalent in FRR's own language is:

ip route 203.0.113.0/24 Null0

Confirm it took with vtysh -c "show ip route 203.0.113.0/24". The route must say blackhole. A packet to an address in the /24 that nothing more specific claims now dies on the firewall instead of looping to the provider and back, which is the other reason this route exists.

  1. Configure BGP. Routing > BGP > General: enable, AS number 64500, router ID the node's own WAN address (198.51.100.2 on fw01, 198.51.100.3 on fw02). Add 203.0.113.0/24 to the networks list. Do this on both nodes; FRR settings are per node and the HA sync is not the tool for them.

  2. Build the prefix lists. Routing > BGP > Prefix Lists. Two lists on each node:

PL-IN   seq 10 permit 0.0.0.0/0          # default only
PL-OUT  seq 10 permit 203.0.113.0/24     # the prefix, exactly

Everything not listed is denied by the implicit deny at the end. The inbound list means the firewall holds a default route and nothing else from the provider, which is all a two-node edge needs. The outbound list means a fat-fingered network statement or a leaked internal route never leaves the building.

  1. Build the route maps. Routing > BGP > Route Maps: RM-IN permit, match prefix list PL-IN; RM-OUT permit, match prefix list PL-OUT. Route maps are what you attach to the neighbour; prefix lists on their own do nothing.

  2. Add the neighbour on each node. Routing > BGP > Neighbors: peer address 198.51.100.1, remote AS 64501, update source the node's own WAN address, route map in RM-IN, route map out RM-OUT. Set the password if the provider uses MD5. Leave the source as the node's interface address and never the CARP VIP: the VIP is on one node at a time, and a session bound to it would reset on every failover, which is the moment you most want the announcement to stay up. Each node keeps its own session, so during a failover the provider already has a path to the surviving node.

  3. Apply on both nodes and watch the sessions come up.

vtysh -c "show ip bgp summary"
vtysh -c "show ip bgp neighbors 198.51.100.1 advertised-routes"
vtysh -c "show ip bgp neighbors 198.51.100.1 received-routes"

The summary should show State as a prefix count (one received), not Active or Connect. Advertised routes should list exactly 203.0.113.0/24. Received routes should list exactly 0.0.0.0/0.

flowchart LR
  P["Provider AS 64501<br/>198.51.100.1"] ---|"eBGP"| A["fw01 AS 64500<br/>198.51.100.2<br/>Null0 203.0.113.0/24"]
  P ---|"eBGP"| B["fw02 AS 64500<br/>198.51.100.3<br/>Null0 203.0.113.0/24"]
  A --- V["WAN CARP VIP<br/>203.0.113.1 + aliases"]
  B --- V

Verify it worked

vtysh tells you what you are sending. Only the internet tells you what it received. Use a public looking glass from a network unrelated to your provider and query 203.0.113.0/24: you want to see your AS as the origin and the provider's AS in the path. Check a route-collector based tool as well; it shows how widely the prefix has propagated and whether the ROA validates. Then from two external vantage points, ping and curl a published address in the prefix. If the looking glass shows the route but the probes fail, the problem is on your side: usually the blackhole route is on only one node, or the CARP VIP for the public address is missing.

Finally, enter CARP maintenance mode on fw01 and repeat the probes. Both BGP sessions should stay Established throughout, and the probes should recover in a few seconds.

Gotchas

FAQ

Do I need an IGP or iBGP between the two nodes? No. Each node announces the same prefix to the same provider and holds its own default. The CARP VIP decides which node carries traffic; BGP only has to make both nodes reachable from the provider.

What if the provider only accepts the prefix from one address? Get that changed in writing before you build. Two sessions from two addresses is the whole point of the pair. A single session from the VIP is a single point of failure that also resets on failover.

Why a /24 when I could announce more specific routes? Because the global table filters longer than /24 for IPv4, so a /25 reaches your provider and nobody else. Split the /24 inside the estate with VLANs and NAT; present one prefix to the world.

Related