https://paul.rosham.com/ Design decisions and how-to guides from running small infrastructure estates on a very low budget: OPNsense, Proxmox, Ceph, Proxmox Backup Server, NetBird, Authentik and Vault. en-au Tue, 06 Oct 2026 00:00:00 GMT https://paul.rosham.com/blog/free-blog-cloudflare-workers-tinacms/ https://paul.rosham.com/blog/free-blog-cloudflare-workers-tinacms/ Tue, 06 Oct 2026 00:00:00 GMT This site costs a domain registration and nothing else. Posts are Markdown in a Git repository, edited in a local CMS, built to static files and served from Cloudflare Workers. Here is the design and the three things that were decided on purpose. https://paul.rosham.com/blog/opnsense-frr-bgp-announce-prefix/ https://paul.rosham.com/blog/opnsense-frr-bgp-announce-prefix/ Mon, 05 Oct 2026 00:00:00 GMT The FRR plugin on an OPNsense pair is enough to announce a portable /24 to a transit provider. The settings that matter, the blackhole route without which nothing is advertised, strict filters both ways, and how to confirm the announcement from outside. https://paul.rosham.com/blog/opnsense-bgp-own-address-space/ https://paul.rosham.com/blog/opnsense-bgp-own-address-space/ Sat, 03 Oct 2026 00:00:00 GMT Provider-assigned addresses lock you to a contract and turn every service IP into a migration project. A portable prefix, an ASN and FRR on OPNsense are enough. The decisions that made the first announcement boring. https://paul.rosham.com/blog/opnsense-retire-public-ip-safely/ https://paul.rosham.com/blog/opnsense-retire-public-ip-safely/ Fri, 02 Oct 2026 00:00:00 GMT An alias nothing referenced was deleted during a cleanup and a routed /28 went dark for about 26 hours. The procedure that came out of it, from the inventory of references to the provider's written confirmation, dual-running the replacement, deleting on both nodes and probing from outside. https://paul.rosham.com/blog/opnsense-netbird-routing-peer/ https://paul.rosham.com/blog/opnsense-netbird-routing-peer/ Wed, 30 Sep 2026 00:00:00 GMT Install the NetBird agent on both firewalls of an HA pair, advertise one route per VLAN with the pair as routing peers, add the outbound NAT the agent's masquerade does not cover, filter on the overlay interface, and avoid the endpoint trap that drops large packets. https://paul.rosham.com/blog/catalyst-3850-console-linux-jumphost-config-backup/ https://paul.rosham.com/blog/catalyst-3850-console-linux-jumphost-config-backup/ Tue, 29 Sep 2026 00:00:00 GMT A USB serial adapter on the stack's console port, the operator in the dialout group, screen at 9600 8N1, and a small expect script that captures the running configuration into the repository. Plus the break-glass path and why management egress never goes over anyone else's WiFi. https://paul.rosham.com/blog/self-hosted-netbird-operator-access/ https://paul.rosham.com/blog/self-hosted-netbird-operator-access/ Mon, 28 Sep 2026 00:00:00 GMT One small VM runs the control plane, with its database shipped to a warm standby every thirty seconds. The firewalls are the routing peers. NetBird policies are treated as transport; real authorisation lives at the service. https://paul.rosham.com/blog/catalyst-3850-syslog-to-wazuh/ https://paul.rosham.com/blog/catalyst-3850-syslog-to-wazuh/ Fri, 25 Sep 2026 00:00:00 GMT Point the switch at the SIEM with logging host, raise the trap level to informational, timestamp with milliseconds, log configuration changes with archive, and let Wazuh decode the rest. Port flaps, LACP changes and failed logins become events next to everything else. https://paul.rosham.com/blog/webshell-incident-lessons-small-hosting/ https://paul.rosham.com/blog/webshell-incident-lessons-small-hosting/ Wed, 16 Sep 2026 00:00:00 GMT A pre-authentication flaw, exploited days after the fix shipped. A read-only document root that pinned the malware in place. A rebuild reinfected within hours. The timeline, anonymised, and the controls that came out of it. https://paul.rosham.com/blog/opnsense-acme-certificates-ha-pair/ https://paul.rosham.com/blog/opnsense-acme-certificates-ha-pair/ Sat, 12 Sep 2026 00:00:00 GMT The ACME client renews on the node it runs on. The backup keeps serving whatever it had, which after ninety days is an expired certificate at the worst possible moment. A daily copy, an import on the backup, and a fingerprint check that tells you when they differ. https://paul.rosham.com/blog/opnsense-ha-config-sync-not-working/ https://paul.rosham.com/blog/opnsense-ha-config-sync-not-working/ Thu, 10 Sep 2026 00:00:00 GMT The backup looks healthy and is a month behind. How to tell, what actually triggers the XMLRPC sync, what never syncs at all, and a drift check you can run from a laptop every fifteen minutes. https://paul.rosham.com/blog/opnsense-carp-vhid-config-sync/ https://paul.rosham.com/blog/opnsense-carp-vhid-config-sync/ Mon, 07 Sep 2026 00:00:00 GMT Two firewalls are only a pair if the backup holds the master's config. The VHID numbering that scales past VLAN 255, the config sync that only runs when asked, and the drift audit that caught rules living on one node. https://paul.rosham.com/blog/pbs-verify-failed-protect-latest/ https://paul.rosham.com/blog/pbs-verify-failed-protect-latest/ Fri, 04 Sep 2026 00:00:00 GMT When a verify job lights up dozens of snapshots at once, the first job is to stop prune from making it worse. Protect the newest snapshot in every group, read the log for the one chunk that is shared, then decide between re-uploading from the source, re-syncing from the other site, and fixing the disk. https://paul.rosham.com/blog/pbs-fingerprint-mismatch-after-rename/ https://paul.rosham.com/blog/pbs-fingerprint-mismatch-after-rename/ Fri, 28 Aug 2026 00:00:00 GMT Rename or reinstall a PBS host and it presents a new self-signed certificate. Every PVE storage entry and every remote on the other PBS has the old fingerprint pinned and refuses to connect. Where the pins live, how to read the new fingerprint, and the order to update them in. https://paul.rosham.com/blog/proxmox-migrate-vms-between-clusters/ https://paul.rosham.com/blog/proxmox-migrate-vms-between-clusters/ Fri, 21 Aug 2026 00:00:00 GMT qm remote-migrate exists and is fussy. The path that works every time is a Proxmox Backup Server backup, already replicated to the other site, restored on the other cluster from a read-only storage entry with a new guest ID. https://paul.rosham.com/blog/authentik-vault-identity-small-estate/ https://paul.rosham.com/blog/authentik-vault-identity-small-estate/ Fri, 14 Aug 2026 00:00:00 GMT People and machines have different lifecycles, so they get different roots of trust. The protocol ladder, the login policy, the credential shapes, and the one rule that neither root may depend on something it protects. https://paul.rosham.com/blog/pbs-restore-drill-file-guest-site/ https://paul.rosham.com/blog/pbs-restore-drill-file-guest-site/ Fri, 14 Aug 2026 00:00:00 GMT A backup you have not restored from is a hope. Three rehearsed restores, each with its commands and its evidence: one file from a pxar archive, one guest to a new VMID on the local PBS, and one guest on the other cluster from the replica namespace. What the first drill found, how often to run it, and what to write down. https://paul.rosham.com/blog/proxmox-sdn-vlan-zones-netbox-ipam/ https://paul.rosham.com/blog/proxmox-sdn-vlan-zones-netbox-ipam/ Fri, 07 Aug 2026 00:00:00 GMT Two VLAN zones on the VLAN-aware bridge, a VNet per VLAN, and NetBox as the IPAM behind the tenant zone so the subnet Proxmox knows about is the one the address plan allocated. SDN does the plumbing; the firewall still does policy and NAT. https://paul.rosham.com/blog/pbs-backup-directory-proxmox-backup-client-pxar/ https://paul.rosham.com/blog/pbs-backup-directory-proxmox-backup-client-pxar/ Fri, 31 Jul 2026 00:00:00 GMT vzdump skips bind mounts, so the files a container serves from CephFS never reach the backup server. A nightly proxmox-backup-client run takes a pxar archive of the latest CephFS snapshot into the tenant's namespace. The token, the command, the timer, and the one-file restore that proves it. https://paul.rosham.com/blog/proxmox-lxc-bind-mount-cephfs-unprivileged/ https://paul.rosham.com/blog/proxmox-lxc-bind-mount-cephfs-unprivileged/ Tue, 28 Jul 2026 00:00:00 GMT Create a CephFS subvolume with a quota, scope a cephx key to its path, mount it on every node with a systemd unit, and bind-mount it into an unprivileged container that holds no Ceph key. df inside the container shows the quota, not the cluster. https://paul.rosham.com/blog/proxmox-cannot-migrate-local-bind-mount-point/ https://paul.rosham.com/blog/proxmox-cannot-migrate-local-bind-mount-point/ Fri, 24 Jul 2026 00:00:00 GMT The error means Proxmox cannot promise the directory behind mp0 exists on the target node. Mount the same path on every node with a systemd unit, check it, and only then add shared=1. https://paul.rosham.com/blog/cephfs-lxc-bind-mount-snapshots-pbs/ https://paul.rosham.com/blog/cephfs-lxc-bind-mount-snapshots-pbs/ Tue, 21 Jul 2026 00:00:00 GMT Guests are disposable; the data is not in them. Put user files on a CephFS subvolume, bind-mount it into an unprivileged container, snapshot on a schedule, and back the snapshot up, because vzdump skips bind mounts. https://paul.rosham.com/blog/proxmox-ceph-replace-osd-rejoin-node/ https://paul.rosham.com/blog/proxmox-ceph-replace-osd-rejoin-node/ Fri, 03 Jul 2026 00:00:00 GMT Two runbooks that share a shape. For a dead disk: identify, out, stop, destroy, swap, create, watch recovery. For a dead node: drain it, remove its Ceph roles, pvecm delnode, reinstall by PXE, pvecm add, recreate the roles, and keep HA fencing in mind throughout. https://paul.rosham.com/blog/proxmox-cluster-api-vip-keepalived/ https://paul.rosham.com/blog/proxmox-cluster-api-vip-keepalived/ Fri, 26 Jun 2026 00:00:00 GMT One address for the Proxmox API that moves to a node whose pveproxy actually answers, using keepalived VRRP with a track script. A node with a wedged API still answers ARP, so the check has to ask port 8006, not the kernel. https://paul.rosham.com/blog/proxmox-three-node-ceph-cluster-setup/ https://paul.rosham.com/blog/proxmox-three-node-ceph-cluster-setup/ Sat, 20 Jun 2026 00:00:00 GMT From three second-hand R630-class servers with IT-mode HBAs to a quorate cluster with one replicated RBD pool, HA on every guest and no local storage for guests. The commands, in order, and what to check after each. https://paul.rosham.com/blog/proxmox-ceph-three-node-cluster-used-hardware/ https://paul.rosham.com/blog/proxmox-ceph-three-node-cluster-used-hardware/ Thu, 18 Jun 2026 00:00:00 GMT Second-hand servers are cheap. What costs money later is a storage decision that blocks migration. Here is the cluster layout that keeps HA, migration and backup working. https://paul.rosham.com/blog/proxmox-automated-install-answer-file-pxe/ https://paul.rosham.com/blog/proxmox-automated-install-answer-file-pxe/ Tue, 16 Jun 2026 00:00:00 GMT Build a self-installing Proxmox VE ISO with proxmox-auto-install-assistant, serve the TOML answer from the jumphost, boot it through iDRAC virtual media or PXE, and avoid the two traps that cost me an afternoon each. https://paul.rosham.com/blog/pbs-prune-verify-gc-schedules/ https://paul.rosham.com/blog/pbs-prune-verify-gc-schedules/ Wed, 10 Jun 2026 00:00:00 GMT Prune decides what to keep, garbage collection reclaims what nothing references, verify checks that what is kept is still readable. Each is harmless alone and all three get in each other's way when they overlap. A weekly calendar that keeps them apart, and apart from the backup window and the restore drill. https://paul.rosham.com/blog/opnsense-unbound-split-horizon-dns/ https://paul.rosham.com/blog/opnsense-unbound-split-horizon-dns/ Fri, 05 Jun 2026 00:00:00 GMT Internal zones forwarded to the per-site authoritative server, public names overridden to internal addresses for internal clients, and DNS over TLS to the upstream. Plus the test that proves the public answer is right, which is not the one you run from your laptop. https://paul.rosham.com/blog/proxmox-rbd-snapshots-cron-path-trap/ https://paul.rosham.com/blog/proxmox-rbd-snapshots-cron-path-trap/ Fri, 29 May 2026 00:00:00 GMT A snapshot script that ran from cron three times a day, reported success for months, and never created a snapshot, because cron's PATH does not include /usr/sbin and the output went to /dev/null. The fixed script, and the checks that make it fail loudly. https://paul.rosham.com/blog/pbs-sync-job-task-ok-datastore-empty/ https://paul.rosham.com/blog/pbs-sync-job-task-ok-datastore-empty/ Wed, 27 May 2026 00:00:00 GMT A pull sync that finishes green and copies nothing has one of three causes: the namespaces do not line up, an earlier run is still holding the locks, or the token cannot see anything upstream. One check for each, and a way to see what the job sees before running it. https://paul.rosham.com/blog/pbs-pull-replication-between-sites/ https://paul.rosham.com/blog/pbs-pull-replication-between-sites/ Tue, 26 May 2026 00:00:00 GMT The remote PBS pulls, with a token that can only read one datastore, a pinned fingerprint, and a per-source namespace so guest IDs from two clusters never collide. Every command, in order, then the storage entry that lets the surviving cluster restore. https://paul.rosham.com/blog/proxmox-backup-server-site-replication/ https://paul.rosham.com/blog/proxmox-backup-server-site-replication/ Sun, 24 May 2026 00:00:00 GMT A standalone PBS per site is the easy part. The part most setups skip is getting a copy to the other site, verified, with a token that cannot do damage. https://paul.rosham.com/blog/opnsense-api-python-firewall-rules/ https://paul.rosham.com/blog/opnsense-api-python-firewall-rules/ Fri, 15 May 2026 00:00:00 GMT The OPNsense API is good enough to run an estate's firewall policy from a script, as long as the script reads back every write, marks what it owns, and remembers that the HA sync will not run on its own. A working pattern with requests. https://paul.rosham.com/blog/catalyst-3850-ios-xe-upgrade-install-mode/ https://paul.rosham.com/blog/catalyst-3850-ios-xe-upgrade-install-mode/ Fri, 08 May 2026 00:00:00 GMT Confirm install mode, clean the flash, copy and verify the image, install it to every member in one command, reload with a console attached, and keep the old image on flash for the rollback you hope not to need. https://paul.rosham.com/blog/opnsense-vlan-trunk-kea-dhcp/ https://paul.rosham.com/blog/opnsense-vlan-trunk-kea-dhcp/ Sat, 02 May 2026 00:00:00 GMT Adding a tenant VLAN to an OPNsense pair fed by a Catalyst trunk, from the switch port to the CARP gateway, the firewall rule order that keeps tenants apart, and a Kea DHCP pool for provisioning. Done twice, because there are two nodes. https://paul.rosham.com/blog/catalyst-3850-port-channel-up-no-traffic/ https://paul.rosham.com/blog/catalyst-3850-port-channel-up-no-traffic/ Fri, 01 May 2026 00:00:00 GMT LACP bundles, the MAC table has entries, and nothing gets through. The port is tagging the wrong way for its host class, and the switch has no reason to tell you. https://paul.rosham.com/blog/catalyst-3850-lacp-port-channel-proxmox-bond/ https://paul.rosham.com/blog/catalyst-3850-lacp-port-channel-proxmox-bond/ Wed, 29 Apr 2026 00:00:00 GMT One port on each stack member, a port-channel in LACP active mode, a trunk whose native VLAN does not exist, and an 802.3ad bond under a VLAN-aware bridge on the Proxmox node. Both halves, and how to check they agree. https://paul.rosham.com/blog/catalyst-3850-trunk-native-vlan-wan-as-vlan/ https://paul.rosham.com/blog/catalyst-3850-trunk-native-vlan-wan-as-vlan/ Tue, 28 Apr 2026 00:00:00 GMT Set every trunk's native VLAN to one you never create, so untagged frames vanish instead of landing in VLAN 1. Then carry the provider's WAN handoff through the stack as a tagged VLAN to both firewalls. https://paul.rosham.com/blog/catalyst-3850-stack-priority-add-member/ https://paul.rosham.com/blog/catalyst-3850-stack-priority-add-member/ Mon, 27 Apr 2026 00:00:00 GMT Cable the stack ring, set priorities in EXEC mode (not config mode), match the software before the new member joins, and power on one switch at a time. The traps are the priority command and auto-upgrade. https://paul.rosham.com/blog/switch-stack-conventions-proxmox-opnsense/ https://paul.rosham.com/blog/switch-stack-conventions-proxmox-opnsense/ Sun, 26 Apr 2026 00:00:00 GMT A used two-member switch stack gives you cross-stack LACP for the price of a prosumer switch. The value is in the conventions, not the hardware. https://paul.rosham.com/blog/proxmox-cloud-init-ubuntu-template-static-ip/ https://paul.rosham.com/blog/proxmox-cloud-init-ubuntu-template-static-ip/ Fri, 17 Apr 2026 00:00:00 GMT Turn an Ubuntu cloud image into a Proxmox template, clone it, and give each clone a static address, gateway, nameserver, SSH key, VLAN tag and a MAC derived from the address. Every value comes from the host plan, so nothing is typed twice. https://paul.rosham.com/blog/opnsense-port-forward-behind-carp/ https://paul.rosham.com/blog/opnsense-port-forward-behind-carp/ Fri, 03 Apr 2026 00:00:00 GMT A port forward on an HA pair has to land on the shared WAN address, not a node's own, and it has to exist on both nodes. The forward, its filter rule, why NAT reflection is the wrong fix for the inside test, and how to test from outside properly. https://paul.rosham.com/blog/vlan-addressing-scheme-small-data-center/ https://paul.rosham.com/blog/vlan-addressing-scheme-small-data-center/ Sun, 29 Mar 2026 00:00:00 GMT Segmentation fails when nobody can remember it. Encode site, VLAN and role into the address so a firewall rule, a DHCP reservation and a MAC can all be derived from a tenant number. https://paul.rosham.com/blog/infrastructure-on-a-very-low-budget/ https://paul.rosham.com/blog/infrastructure-on-a-very-low-budget/ Sat, 28 Mar 2026 00:00:00 GMT What it is like to build a multi-site estate out of open source and second-hand hardware with no support contract to call. The challenges that keep coming back, the rules that came out of them, and an index of the posts that cover each decision.