docs: plan a second SSD for Longhorn

Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.

docs/longhorn-second-ssd.md

@@ -0,0 +1,124 @@
+# A second SSD for Longhorn
+
+## Why
+
+Root, the dqlite database, every Longhorn replica, container images and
+emptyDirs all sit on one Apacer AS350 1TB, a DRAM-less 2.5" SATA SSD. When
+something writes to it for minutes (Longhorn backups at 00:00, the Velero
+backup at 01:00, kopia maintenance, image pulls), its write latency climbs
+and dqlite, which syncs every commit, stalls for seconds. Controllers then
+lose their leases and restart ("Still stalling after the VACUUM" in
+docs/upgrade-k8s-1.33.md). Longer leases and throttled maintenance soften
+that; moving Longhorn's data to its own NVMe SSD takes away most of the
+writes dqlite competes with.
+
+The alternative, moving dqlite to the new SSD, is less work in data but
+not in risk: its directory lives under the microk8s snap's revision
+directory, which snapd copies on every refresh, so a bind mount there is
+fragile. Longhorn supports several disks per node and moves replicas
+between them live.
+
+## Hardware
+
+- opti1 is a Dell OptiPlex 5060 (BIOS 1.29.0); the i5-8600T points to the
+  Micro, but check the case. Every 5060 has one M.2 2280 slot that takes
+  PCIe NVMe (Gen3 x4); Dell sells 2280 Gen3 x4 drives for the Micro. The
+  only storage controller is the SATA AHCI one and no NVMe device is
+  present, so the slot should be empty.
+- Buy an M.2 2280 NVMe SSD **with a DRAM cache**, 2 TB. Gen4 drives work and
+  run at Gen3 speed, which is plenty and cooler. Examples with DRAM:
+  Samsung 990 Pro, WD Black SN850X, Kingston KC3000, Crucial T500. Avoid
+  DRAM-less and QLC drives (WD SN770, Samsung 990 EVO, Crucial P3); they
+  have the same weakness as the current SSD. In the Micro, skip a heatsink
+  that doesn't fit under the cover.
+- Size: Longhorn holds 339 GiB on disk now, of which 297 GiB is the immich
+  library (126 GB in use inside the volume; the rest is snapshots, see
+  below). Personal immich accounts will add uploads; volumes can be sized
+  up to 551 GiB in total today. 2 TB leaves room; 1 TB would fit for now.
+
+## Before: shrink what has to move
+
+- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
+  snapshots from before October 2026, nearly all on immich-claim
+  (pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
+  2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
+  longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
+  don't depend on them. Deleting a snapshot merges it into the next one,
+  which writes a lot, so do this in a quiet hour, one volume at a time.
+- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem")
+  so blocks of deleted files are released.
+- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a
+  Longhorn system backup.
+- [ ] Set Longhorn's `concurrent-replica-rebuild-per-node-limit` from 5 to 1
+  for the move, so only one replica copies at a time. It is not in the
+  Helm values, so set it in the UI (Settings, General) or with
+  `kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type merge -p '{"value":"1"}'`.
+
+## Install (downtime: everything, about 15 minutes)
+
+- [ ] `sudo shutdown -h now` on opti1. microk8s stops, the volumes detach.
+- [ ] Unplug, open the case, put the SSD in the M.2 slot and screw it down.
+- [ ] Boot into the BIOS (F2): the NVMe drive shows up under storage, SATA
+  operation stays AHCI, and the boot sequence still starts "ubuntu" from
+  the SATA SSD.
+- [ ] Boot. `lsblk` shows `nvme0n1`; wait until everything is green again
+  (gatus on s1 and s2).
+
+## Prepare the disk (sudo on opti1)
+
+```bash
+lsblk -o NAME,SIZE,MODEL /dev/nvme0n1          # the new, empty drive
+sudo wipefs -a /dev/nvme0n1
+sudo parted -s /dev/nvme0n1 mklabel gpt mkpart longhorn ext4 1MiB 100%
+sudo mkfs.ext4 -L longhorn-nvme /dev/nvme0n1p1
+sudo mkdir -p /var/lib/longhorn-nvme
+echo "UUID=$(sudo blkid -s UUID -o value /dev/nvme0n1p1) /var/lib/longhorn-nvme ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
+sudo systemctl daemon-reload && sudo mount /var/lib/longhorn-nvme
+df -h /var/lib/longhorn-nvme
+```
+
+Longhorn wants an extent-based filesystem (ext4 or xfs) mounted at a
+directory, no symlink. With `nofail` a missing drive doesn't block the
+boot; Longhorn then sees an empty directory on the root filesystem with
+the wrong filesystem ID and won't use it. Ubuntu's weekly `fstrim.timer`
+covers the new filesystem too.
+
+## Move Longhorn's data (no downtime)
+
+- [ ] Add the disk to the Longhorn node:
+  ```bash
+  kubectl -n longhorn-system patch nodes.longhorn.io opti1 --type merge -p '{"spec":{"disks":{"nvme":{"path":"/var/lib/longhorn-nvme","diskType":"filesystem","allowScheduling":true,"evictionRequested":false,"storageReserved":53687091200,"tags":[]}}}}'
+  ```
+  (50 GiB reserved.) Wait until the disk is Ready and Schedulable.
+- [ ] On the old disk (`default-disk-5db8685563bf9bf9`, `/var/lib/longhorn/`)
+  set `allowScheduling: false`, then `evictionRequested: true`. Longhorn
+  builds a new replica of each volume on the NVMe drive and drops the old
+  one once the rebuild succeeds; it attaches detached volumes for this
+  itself. All 20 volumes have one replica, so each briefly has two.
+- [ ] Watch the old disk's replica count go to 0 (UI, Node, or
+  `kubectl -n longhorn-system get nodes.longhorn.io opti1 -o yaml`). The
+  immich library takes longest. Copying reads from the SATA SSD, so expect
+  a few slow API moments meanwhile; best in the evening.
+- [ ] Remove the old disk from `spec.disks` once it has no replicas, and set
+  the rebuild limit back to 5.
+
+To stop halfway: set `evictionRequested` back to false; replicas already
+moved stay on the NVMe drive, the rest on the SATA SSD, and both disks
+work.
+
+## Afterwards
+
+- [ ] Compare with before: lease updates over 5s
+  (`apiserver_request_duration_seconds` for PUT on leases), controller
+  restarts, and sda's latency during the nightly backups
+  (`~/iosample.sh`).
+- [ ] Optional, if stalls remain: container images
+  (`/var/snap/microk8s/common/var/lib/containerd`) and the kubelet's
+  emptyDirs, such as kopia's scratch space, still write to the SATA SSD.
+
+## Sources
+
+- Dell parts for the OptiPlex 5060 Micro, with M.2 2280 PCIe NVMe Gen3 x4
+  drives: https://dell.com/en-au/shop/pfydresults/6163895
+- Longhorn, eviction: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/disks-or-nodes-eviction/
+- Longhorn, multiple disks: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/multidisk/

docs/upgrade-k8s-1.33.md

@@ -263,7 +263,7 @@ whatever time of day the hourly schedule had drifted to.
 - [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
   for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
   numbers on the disk latency during tonight's backups
-- [ ] Structural fix: a second SSD. The OptiPlex 5060 (Micro, going by the
+- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
   i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
   present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with
   DRAM for Longhorn's data (replicas move over live by evicting the old