From 5f8394e0c02e47a0360cdc0db9e1589877ad2a12 Mon Sep 17 00:00:00 2001 From: Ruben Hensen Date: Sat, 24 Oct 2026 15:17:15 +0200 Subject: [PATCH] docs: plan a second SSD for Longhorn Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction. --- docs/longhorn-second-ssd.md | 124 ++++++++++++++++++++++++++++++++++++ docs/upgrade-k8s-1.33.md | 2 +- 2 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 docs/longhorn-second-ssd.md diff --git a/docs/longhorn-second-ssd.md b/docs/longhorn-second-ssd.md new file mode 100644 index 0000000..1d185a4 --- /dev/null +++ b/docs/longhorn-second-ssd.md @@ -0,0 +1,124 @@ +# A second SSD for Longhorn + +## Why + +Root, the dqlite database, every Longhorn replica, container images and +emptyDirs all sit on one Apacer AS350 1TB, a DRAM-less 2.5" SATA SSD. When +something writes to it for minutes (Longhorn backups at 00:00, the Velero +backup at 01:00, kopia maintenance, image pulls), its write latency climbs +and dqlite, which syncs every commit, stalls for seconds. Controllers then +lose their leases and restart ("Still stalling after the VACUUM" in +docs/upgrade-k8s-1.33.md). Longer leases and throttled maintenance soften +that; moving Longhorn's data to its own NVMe SSD takes away most of the +writes dqlite competes with. + +The alternative, moving dqlite to the new SSD, is less work in data but +not in risk: its directory lives under the microk8s snap's revision +directory, which snapd copies on every refresh, so a bind mount there is +fragile. Longhorn supports several disks per node and moves replicas +between them live. + +## Hardware + +- opti1 is a Dell OptiPlex 5060 (BIOS 1.29.0); the i5-8600T points to the + Micro, but check the case. Every 5060 has one M.2 2280 slot that takes + PCIe NVMe (Gen3 x4); Dell sells 2280 Gen3 x4 drives for the Micro. The + only storage controller is the SATA AHCI one and no NVMe device is + present, so the slot should be empty. +- Buy an M.2 2280 NVMe SSD **with a DRAM cache**, 2 TB. Gen4 drives work and + run at Gen3 speed, which is plenty and cooler. Examples with DRAM: + Samsung 990 Pro, WD Black SN850X, Kingston KC3000, Crucial T500. Avoid + DRAM-less and QLC drives (WD SN770, Samsung 990 EVO, Crucial P3); they + have the same weakness as the current SSD. In the Micro, skip a heatsink + that doesn't fit under the cover. +- Size: Longhorn holds 339 GiB on disk now, of which 297 GiB is the immich + library (126 GB in use inside the volume; the rest is snapshots, see + below). Personal immich accounts will add uploads; volumes can be sized + up to 551 GiB in total today. 2 TB leaves room; 1 TB would fit for now. + +## Before: shrink what has to move + +- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in + snapshots from before October 2026, nearly all on immich-claim + (pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from + 2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no + longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS + don't depend on them. Deleting a snapshot merges it into the next one, + which writes a lot, so do this in a quiet hour, one volume at a time. +- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem") + so blocks of deleted files are released. +- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a + Longhorn system backup. +- [ ] Set Longhorn's `concurrent-replica-rebuild-per-node-limit` from 5 to 1 + for the move, so only one replica copies at a time. It is not in the + Helm values, so set it in the UI (Settings, General) or with + `kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type merge -p '{"value":"1"}'`. + +## Install (downtime: everything, about 15 minutes) + +- [ ] `sudo shutdown -h now` on opti1. microk8s stops, the volumes detach. +- [ ] Unplug, open the case, put the SSD in the M.2 slot and screw it down. +- [ ] Boot into the BIOS (F2): the NVMe drive shows up under storage, SATA + operation stays AHCI, and the boot sequence still starts "ubuntu" from + the SATA SSD. +- [ ] Boot. `lsblk` shows `nvme0n1`; wait until everything is green again + (gatus on s1 and s2). + +## Prepare the disk (sudo on opti1) + +```bash +lsblk -o NAME,SIZE,MODEL /dev/nvme0n1 # the new, empty drive +sudo wipefs -a /dev/nvme0n1 +sudo parted -s /dev/nvme0n1 mklabel gpt mkpart longhorn ext4 1MiB 100% +sudo mkfs.ext4 -L longhorn-nvme /dev/nvme0n1p1 +sudo mkdir -p /var/lib/longhorn-nvme +echo "UUID=$(sudo blkid -s UUID -o value /dev/nvme0n1p1) /var/lib/longhorn-nvme ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab +sudo systemctl daemon-reload && sudo mount /var/lib/longhorn-nvme +df -h /var/lib/longhorn-nvme +``` + +Longhorn wants an extent-based filesystem (ext4 or xfs) mounted at a +directory, no symlink. With `nofail` a missing drive doesn't block the +boot; Longhorn then sees an empty directory on the root filesystem with +the wrong filesystem ID and won't use it. Ubuntu's weekly `fstrim.timer` +covers the new filesystem too. + +## Move Longhorn's data (no downtime) + +- [ ] Add the disk to the Longhorn node: + ```bash + kubectl -n longhorn-system patch nodes.longhorn.io opti1 --type merge -p '{"spec":{"disks":{"nvme":{"path":"/var/lib/longhorn-nvme","diskType":"filesystem","allowScheduling":true,"evictionRequested":false,"storageReserved":53687091200,"tags":[]}}}}' + ``` + (50 GiB reserved.) Wait until the disk is Ready and Schedulable. +- [ ] On the old disk (`default-disk-5db8685563bf9bf9`, `/var/lib/longhorn/`) + set `allowScheduling: false`, then `evictionRequested: true`. Longhorn + builds a new replica of each volume on the NVMe drive and drops the old + one once the rebuild succeeds; it attaches detached volumes for this + itself. All 20 volumes have one replica, so each briefly has two. +- [ ] Watch the old disk's replica count go to 0 (UI, Node, or + `kubectl -n longhorn-system get nodes.longhorn.io opti1 -o yaml`). The + immich library takes longest. Copying reads from the SATA SSD, so expect + a few slow API moments meanwhile; best in the evening. +- [ ] Remove the old disk from `spec.disks` once it has no replicas, and set + the rebuild limit back to 5. + +To stop halfway: set `evictionRequested` back to false; replicas already +moved stay on the NVMe drive, the rest on the SATA SSD, and both disks +work. + +## Afterwards + +- [ ] Compare with before: lease updates over 5s + (`apiserver_request_duration_seconds` for PUT on leases), controller + restarts, and sda's latency during the nightly backups + (`~/iosample.sh`). +- [ ] Optional, if stalls remain: container images + (`/var/snap/microk8s/common/var/lib/containerd`) and the kubelet's + emptyDirs, such as kopia's scratch space, still write to the SATA SSD. + +## Sources + +- Dell parts for the OptiPlex 5060 Micro, with M.2 2280 PCIe NVMe Gen3 x4 + drives: https://dell.com/en-au/shop/pfydresults/6163895 +- Longhorn, eviction: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/disks-or-nodes-eviction/ +- Longhorn, multiple disks: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/multidisk/ diff --git a/docs/upgrade-k8s-1.33.md b/docs/upgrade-k8s-1.33.md index e88e45d..65dcb69 100644 --- a/docs/upgrade-k8s-1.33.md +++ b/docs/upgrade-k8s-1.33.md @@ -263,7 +263,7 @@ whatever time of day the hourly schedule had drifted to. - [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put numbers on the disk latency during tonight's backups -- [ ] Structural fix: a second SSD. The OptiPlex 5060 (Micro, going by the +- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with DRAM for Longhorn's data (replicas move over live by evicting the old