Proxmox: SDN VLAN zones with an external IPAM
In short. Proxmox SDN on a VLAN-aware
vmbr0gives every node the same named networks, so a guest config saysbridge=t101instead of a bridge plus a tag. Two zones:opswith the built-in IPAM andtenantswith NetBox. Subnets in the tenant zone must already exist in NetBox, because NetBox is the source of record. Apply, then checkbridge vlan showon a tap device. Nothing here filters or translates traffic; that is the firewall's job.
Goal: named tenant networks on every node, backed by the real IPAM
You have a tenant VLAN scheme (VLAN 100+N, 10.<site>.<vlan>.0/24) and NetBox already holds the prefixes. You want Proxmox to use those names and those subnets without a second copy of the plan living in /etc/pve. Proxmox VE 9.2; SDN is part of the base install.
What you need
- A PVE 9.2 cluster where
vmbr0is VLAN-aware on every node and/etc/network/interfacescontainssource /etc/network/interfaces.d/*(the installer adds it; check anyway). - NetBox reachable from every node on the ops VLAN, here
https://10.1.10.30/, with an API token that can read and write prefixes and IP addresses. - The prefix for each tenant VLAN already created in NetBox by the addressing tooling.
Steps
- Confirm the bridge is VLAN-aware on every node and the include line is present.
grep -E 'bridge-vlan-aware|^source' /etc/network/interfaces
- Register NetBox as an IPAM. The token is read from a root-only file on the node, never typed inline and never in the repository that holds the playbook. Proxmox stores it in the cluster filesystem under
/etc/pve/sdn/, which only root can read.
pvesh create /cluster/sdn/ipams --ipam netbox --type netbox \
--url https://10.1.10.30/api --token "$(cat /root/.netbox-token)"
- Create the two zones on
vmbr0. Zone names are limited to eight characters.
pvesh create /cluster/sdn/zones --zone ops --type vlan --bridge vmbr0
pvesh create /cluster/sdn/zones --zone tenants --type vlan --bridge vmbr0 --ipam netbox
- One VNet per VLAN. The VNet name becomes the bridge name guests use, so keep it short and derivable:
tplus the VLAN.
pvesh create /cluster/sdn/vnets --vnet t101 --zone tenants --tag 101
pvesh create /cluster/sdn/vnets --vnet t102 --zone tenants --tag 102
pvesh create /cluster/sdn/vnets --vnet ops10 --zone ops --tag 10
- Add the subnet to each VNet. For the tenant zone the prefix must already exist in NetBox; Proxmox looks it up and will create it if missing, which you do not want, so check first.
curl -s -H "Authorization: Token $(cat /root/.netbox-token)" \
'https://10.1.10.30/api/ipam/prefixes/?prefix=10.1.101.0/24' | grep -c '"prefix"'
pvesh create /cluster/sdn/vnets/t101/subnets --subnet 10.1.101.0/24 --type subnet --gateway 10.1.101.254
- Apply. Until this runs, the configuration is staged and no node has the VNets.
pvesh set /cluster/sdn
- Put a guest on a VNet. The VNet is the bridge; the tag comes from the VNet, so there is no
tag=on the NIC any more.
qm set 111 --net0 virtio=BC:24:11:65:01:0B,bridge=t101
flowchart TB
NB["NetBox<br/>prefixes, source of record"] -- "IPAM plugin, token" --> Z2["zone: tenants<br/>VLAN on vmbr0"]
Z1["zone: ops<br/>VLAN on vmbr0, pve IPAM"] --> V0["ops10, tag 10"]
Z2 --> V1["t101, tag 101<br/>10.1.101.0/24"]
Z2 --> V2["t102, tag 102<br/>10.1.102.0/24"]
V1 --> VM["VM 111<br/>bridge=t101"]
V1 -. "policy, NAT, DHCP" .-> FW["Firewall and tenant .2<br/>not SDN"]Verify it worked
pvesh get /cluster/resources --type sdn
ip -d link show t101 | grep -o 'vlan_filtering [01]'
bridge vlan show dev tap111i0
Every node should list each VNet as ok, the VNet bridge exists on the node with VLAN filtering enabled, and the guest's tap device carries VLAN 101 as PVID. From inside the guest, ping 10.1.101.254 and 10.1.101.2; from the switch, the MAC should show on VLAN 101. If the VNet is missing on one node, that node's ifreload -a failed, and journalctl -u networking says why.
Gotchas
- SDN does not do inter-VLAN policy or NAT. A VNet is a named VLAN; whether
t101can reacht102is decided on the firewall, and the apex NAT for a tenant's public address is a firewall rule too. Expecting the zone to isolate tenants is the common mistake. - SDN's own DHCP is left off in the tenant zone. Each tenant's
.2container runs Kea for the provisioning pool, and two DHCP servers on one VLAN is a support ticket. - The NetBox token sits in
/etc/pve/sdn/ipams.cfg. That file is in the cluster filesystem, so it lands on every node; treat the file as the secret it is and keep it out of any sync to a repository. pvesh set /cluster/sdnreloads networking on every node. Run it in a quiet moment the first time; it was harmless here, but a bad/etc/network/interfaceson one node turns an apply into an outage on that node.- Renaming a VNet is not supported in place. Guests reference it as the bridge name, so pick the convention once.
FAQ
Why keep an ops zone with the built-in IPAM at all? The ops VLANs are few and static and already in the addressing repository. Putting them under NetBox would mean NetBox is a dependency of bringing up the thing that hosts NetBox.
Can SDN allocate the guest address from NetBox automatically? Only with SDN's DHCP, which the tenant zone does not use. Here NetBox allocates the address through the addressing tooling, and cloud-init sets it statically. SDN's job is to make sure the subnet Proxmox shows is the one NetBox owns.
Is the tag on the NIC still allowed?
Yes, with bridge=vmbr0,tag=101 as before. Mixing both styles works and is confusing. Once the VNets exist, new guests use them and old ones are moved during their next maintenance.