Paul Rosham

OPNsense: port forward to a VM behind a CARP pair

In short. Forward on the WAN CARP VIP or an IP alias on it, never on a node's own WAN address, so the forward survives failover. Let OPNsense create the associated filter rule. Do not switch on NAT reflection to make the public name work from inside; give the name an internal answer in DNS instead. Test from two places outside the estate, then confirm the backup has the same forward.

Port forward works on the master but dies on failover

The usual story: the forward was built on the interface address of fw01. Traffic arrived on fw01's own WAN address, which only fw01 ever answers for. When fw02 took over, the public address people were using was still there, but the forward was bound to an address that had gone quiet. On a pair, anything a client connects to has to be an address that moves between nodes.

One public address per tenant, and why

Two constraints shape how forwards are laid out here. Portable address space is paid for and finite, so public addresses are not handed out per service. And every CARP virtual IP is one more election the firewall pair has to run, so the number of VIPs is kept small on purpose.

The pattern that follows: one CARP VIP per VLAN on the inside, which is the gateway, and on the WAN a single CARP VIP with every other public address attached to it as an IP alias. Each tenant gets one public alias, and that one address carries both the tenant's inbound forwards and its outbound NAT, so a tenant's traffic leaves from the same address it is reached on. One per tenant is the aim; a second is the exception that needs a written reason.

That means several HTTP services for the same tenant share one address on ports 80 and 443. The forward sends all of it to the tenant's nginx ingress at .10 inside the tenant VLAN, and nginx routes by hostname to the right VM or container. The firewall never needs a rule per service, and adding a service is a change in the tenant's nginx configuration, not on the firewall.

What you need

Steps

  1. Check the public address is on the VIP, not on a node. Firewall > Virtual IPs > Settings on the master should show 203.0.113.10 with type IP Alias and the WAN CARP VIP as its parent. If it is a plain interface address on fw01, stop and fix that first; the rest of this post assumes it moves.

  2. Create the forward on the master. Firewall > NAT > Port Forward, add a rule. Interface WAN, protocol TCP, destination set to the single-host address 203.0.113.10 (pick it from the virtual IP list rather than typing it, so it stays attached to the right object), destination port 443, redirect target 10.1.101.10, redirect port 443. In the filter rule association field choose the option that adds an associated filter rule. Give it a description you will recognise in a rule list a year from now, for example tenant1 ingress https.

  3. Save and apply. The associated rule appears under Firewall > Rules > WAN with a link icon back to the forward. Do not edit that rule by hand; edit the forward and the rule follows.

  4. Decide how the inside will reach the service, and do not use NAT reflection for it. Reflection rewrites traffic from inside the estate so that connecting to 203.0.113.10 from 10.1.101.50 works; it does so by making the traffic appear to come from the firewall, and it hides the real client address from the VM. On this estate, Unbound answers the public name with the internal address for internal clients instead. See the split-horizon post linked below. Reflection stays off under Firewall > Settings > Advanced.

  5. Sync to the backup. A GUI save on the master triggers the XMLRPC push to the backup; if you created the forward through the API, run configctl filter sync on the master. Then open Firewall > NAT > Port Forward on the backup and look for the same row. The point of this step is to see it with your own eyes, because a sync can fail quietly.

flowchart LR
  C["Client on the internet"] -->|"203.0.113.10:443"| V["WAN CARP VIP<br/>plus IP aliases"]
  V --> F1["fw01 master"]
  V -.-> F2["fw02 backup"]
  F1 -->|"rdr to 10.1.101.10:443"| VM["Tenant ingress VM"]
  F2 -.->|"same forward, idle"| VM

Verify it worked

From a machine outside the estate:

curl -sv --max-time 10 https://203.0.113.10/ -o /dev/null 2>&1 | grep -E "Connected|HTTP/"
nc -vz -w 5 203.0.113.10 443

Do that from two vantage points, not one. A single external host on the same provider as the estate can succeed through a path that the rest of the internet does not share. Then do the failover test: on the master, Interfaces > Virtual IPs > Status has a button to enter CARP maintenance mode, which hands the VIPs to the backup. Run the two probes again. They should succeed. Leave maintenance mode and run them a third time.

On the firewall, pfctl -sn | grep 203.0.113.10 shows the active rdr rule, and Firewall > Log Files > Live View filtered on the destination address shows the hits.

Gotchas

FAQ

Why not just enable NAT reflection and be done? Because it hides the client address from the service, adds a hairpin through the firewall for every internal request, and makes the "does it work from outside" question impossible to answer from inside. Split-horizon DNS is three lines of configuration and keeps the internal path honest.

Should I forward to the VM's address or to an alias? To an alias containing the VM's address, if more than one forward points at the same host. When the VM moves, you change the alias once. For a single forward the address is fine.

Does the forward need its own outbound NAT rule? Not for inbound traffic. Outbound NAT matters if the VM's own outgoing connections must appear from 203.0.113.10 rather than from the WAN VIP, for example for mail. That is a separate outbound rule under Firewall > NAT > Outbound, and it also has to be on both nodes.

Related