Design decisions
Design decisions: 12 posts, the choices behind a small infrastructure estate, each with what it cost and what it taught.
An el-cheapo blog on Cloudflare: static Next.js, TinaCMS in local mode, no monthly billThis site costs a domain registration and nothing else. Posts are Markdown in a Git repository, edited in a local CMS, built to static files and served from Cloudflare Workers. Here is the design and the three things that were decided on purpose.
Announce your own /24 from an OPNsense pair: BGP without a router budgetProvider-assigned addresses lock you to a contract and turn every service IP into a migration project. A portable prefix, an ASN and FRR on OPNsense are enough. The decisions that made the first announcement boring.
Self-hosted NetBird as the operator plane: firewalls as routing peers, policies as transportOne small VM runs the control plane, with its database shipped to a warm standby every thirty seconds. The firewalls are the routing peers. NetBird policies are treated as transport; real authorisation lives at the service.
What a webshell taught a small hosting shop: rebuild, never cleanA pre-authentication flaw, exploited days after the fix shipped. A read-only document root that pinned the malware in place. A rebuild reinfected within hours. The timeline, anonymised, and the controls that came out of it.
OPNsense HA that actually fails over: VHID 1 everywhere, and the sync that is not automaticTwo firewalls are only a pair if the backup holds the master's config. The VHID numbering that scales past VLAN 255, the config sync that only runs when asked, and the drift audit that caught rules living on one node.
Two roots of trust: Authentik for people, Vault for machinesPeople and machines have different lifecycles, so they get different roots of trust. The protocol ladder, the login policy, the credential shapes, and the one rule that neither root may depend on something it protects.
CephFS subvolumes under LXC: snapshots your backups can seeGuests are disposable; the data is not in them. Put user files on a CephFS subvolume, bind-mount it into an unprivileged container, snapshot on a schedule, and back the snapshot up, because vzdump skips bind mounts.
Ex-lease Dell Servers, one Ceph pool, no local-zfsSecond-hand servers are cheap. What costs money later is a storage decision that blocks migration. Here is the cluster layout that keeps HA, migration and backup working.
Proxmox Backup Server that leaves the buildingA standalone PBS per site is the easy part. The part most setups skip is getting a copy to the other site, verified, with a token that cannot do damage.
Second-hand Catalyst, first-class conventionsA used two-member switch stack gives you cross-stack LACP for the price of a prosumer switch. The value is in the conventions, not the hardware.
A VLAN scheme you can derive in your headSegmentation fails when nobody can remember it. Encode site, VLAN and role into the address so a firewall rule, a DHCP reservation and a MAC can all be derived from a tenant number.
Infrastructure engineering on a (very low) budget: start hereWhat it is like to build a multi-site estate out of open source and second-hand hardware with no support contract to call. The challenges that keep coming back, the rules that came out of them, and an index of the posts that cover each decision.