Catalyst 3850: cross-stack LACP to a Proxmox bond, both sides
In short. Give the node one port on each stack member and bundle them with
channel-group N mode active. Trunk the port-channel with native VLAN 998 and an explicit allowed list. On the node,bond0is802.3adwithlayer3+4hashing andvmbr0is a VLAN-aware bridge on top of it. Checkshow etherchannel summaryon the switch,/proc/net/bonding/bond0on the node, then ping a gateway from a VM on a tagged VLAN.
I want a Proxmox node bonded across both members of a Catalyst stack
You have a two-member Catalyst 3850 stack and a Proxmox node with two NICs. You want real LACP, not active-backup, and you want the node to survive losing either switch member. This is the configuration I push to every node at every site. The reasoning for the stack, the poison native VLAN and the per-host-class trunk rules is in the decision post linked at the end; this post is just the two halves of the configuration and how to prove they match.
What you need
- A WS-C3850-12X48U stack on IOS-XE 16.12 with the two members already stacked (
show switchlists both as Ready). - A Proxmox VE 8.4 node with two NICs of the same speed, cabled one to each stack member. In the examples the node is
pve1at site A, the NICs areenp1s0f0andenp1s0f1, and they land onTenGigabitEthernet1/0/3andTenGigabitEthernet2/0/3. - Console or SSH access to the stack, and a console or a second path to the node, because you will restart its networking.
flowchart LR
subgraph N["pve1"]
B["bond0 (802.3ad, layer3+4)<br/>vmbr0 VLAN-aware"]
E0["enp1s0f0"]
E1["enp1s0f1"]
B --- E0
B --- E1
end
subgraph ST["Catalyst stack"]
S1["Member 1, priority 15<br/>Te1/0/3"]
S2["Member 2, priority 12<br/>Te2/0/3"]
S1 === S2
end
E0 -- "Po3 member" --> S1
E1 -- "Po3 member" --> S2Steps
-
Pick the port-channel number. I use the same number as the low port, so ports 3 on each member become
Port-channel3. It is not required, but it meansshow etherchannel summaryreads like the patch panel. -
Configure the port-channel interface first, then the physical ports. The port-channel carries the trunk settings; the members inherit them when they bundle.
sw-a.site.example# configure terminal
sw-a.site.example(config)# interface Port-channel3
sw-a.site.example(config-if)# description pve1 bond0
sw-a.site.example(config-if)# switchport mode trunk
sw-a.site.example(config-if)# switchport trunk native vlan 998
sw-a.site.example(config-if)# switchport trunk allowed vlan 1,5,10,11,100-199
sw-a.site.example(config-if)# switchport nonegotiate
sw-a.site.example(config-if)# spanning-tree portfast trunk
sw-a.site.example(config-if)# exit
sw-a.site.example(config)# interface range TenGigabitEthernet1/0/3, TenGigabitEthernet2/0/3
sw-a.site.example(config-if-range)# description pve1 bond0 member
sw-a.site.example(config-if-range)# switchport mode trunk
sw-a.site.example(config-if-range)# switchport trunk native vlan 998
sw-a.site.example(config-if-range)# switchport trunk allowed vlan 1,5,10,11,100-199
sw-a.site.example(config-if-range)# switchport nonegotiate
sw-a.site.example(config-if-range)# channel-group 3 mode active
sw-a.site.example(config-if-range)# no shutdown
sw-a.site.example(config-if-range)# end
sw-a.site.example# write memory
mode active means the switch sends LACPDUs rather than waiting for the node to start, and switchport nonegotiate stops DTP from trying to be helpful. VLAN 998 is deliberately never created: an untagged frame from the node has nowhere to go. The allowed list is explicit so a tenant VLAN you have not planned for cannot leak onto the node by default.
- On the node, write
/etc/network/interfaces. The management address sits on a VLAN sub-interface of the bridge so the bond carries everything tagged. If the node has spare NICs for a separate management bond, the pattern is identical with a second bond and bridge.
auto lo
iface lo inet loopback
auto enp1s0f0
iface enp1s0f0 inet manual
auto enp1s0f1
iface enp1s0f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp1s0f0 enp1s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vmbr0.1
iface vmbr0.1 inet static
address 10.1.1.11/24
gateway 10.1.1.254
- Apply it without a reboot.
ifreload -ais from ifupdown2, which Proxmox ships, and it reconciles the running state with the file rather than tearing everything down.
ifreload -a
Do this from the console, or from a session that does not depend on the bond, the first time. If the switch side is wrong you will find out by losing the session.
- Give a VM a tagged interface. In the VM's hardware settings the network device uses bridge
vmbr0with VLAN tag 101, which on the switch side is already in the allowed list. The guest sees an untagged interface; the bridge does the tagging.
Verify it worked
On the switch, the port-channel should be SU (layer 2, in use) and both members P (bundled):
sw-a.site.example# show etherchannel summary
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
3 Po3(SU) LACP Te1/0/3(P) Te2/0/3(P)
show lacp neighbor should show the same partner system ID on both ports, which is the node's bond MAC. If the IDs differ, you have cabled two different machines into one channel.
On the node, both slaves should report the same aggregator ID and the partner MAC should be the stack's:
cat /proc/net/bonding/bond0 | grep -E 'Bonding Mode|Transmit Hash|Aggregator ID|Partner Mac|Slave Interface|MII Status'
Finally, from a VM on VLAN 101, ping its gateway at 10.1.101.254. Then pull one cable from the node and ping again. The ping should carry on with at most one lost packet, and show etherchannel summary should show the pulled port as D while the channel stays SU.
Gotchas
- A port-channel that shows
SUproves LACP agreed. It does not prove a single data frame can pass. If the ping fails while the summary looks perfect, the VLAN side is wrong: see the companion post on a port-channel that is up with no traffic. - Put the trunk configuration on the physical ports too. If the port-channel and its members disagree, IOS-XE suspends the member (
sin the summary) and you get a one-legged bond. bridge-vids 2-4094on the Proxmox bridge is the set of VLANs the bridge will forward. The switch's allowed list is the real control; keep the bridge permissive and the switch strict, so the repository holds one source of truth.layer3+4hashing balances flows, not packets. One VM talking to one backup target will use one link. That is expected and the second link is for a different flow, or for the failure case.- The two members of the bond must land on different stack members or the exercise is pointless.
show lacp neighboron both ports tells you which chassis each is on, andshow cdp neighborsfrom a node with LLDP or CDP enabled tells you the same from the other side.
FAQ
Why mode active on the switch rather than passive?
Passive on both ends never forms a bundle. Active on the switch means the node can be passive, or active, and it still works. Proxmox's 802.3ad mode is active by default, so you end up with both sides active, which is fine and converges fastest.
Do I need lacp rate fast?
Not for a server. The default 30-second LACPDU interval only affects how quickly a silent partner is noticed; a cable pull is detected by link state immediately. Fast rate adds chatter for no benefit here.
Can the same port-channel carry the storage VLAN for Ceph?
At later sites, yes: Ceph is on its own bond, bond1, tagged as bond1.5, and the switch side is a second port-channel with allowed vlan 5. At the first site storage lives on a separate switch. Either way, do not put Ceph on bond0 with the guests.