From 1f56e67e8b8e4ad4abd33a380f4dd76f2bf8d6d7 Mon Sep 17 00:00:00 2001 From: Ruben Hensen Date: Sun, 25 Oct 2026 03:42:13 +0100 Subject: [PATCH] docs: what the disk did in the last day, and foto's DNS --- docs/longhorn-second-ssd.md | 9 +++++++++ docs/roadmap.md | 15 +++++++++------ docs/upgrade-k8s-1.33.md | 19 +++++++++++++++++-- 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/docs/longhorn-second-ssd.md b/docs/longhorn-second-ssd.md index a479cb0..4ed9b4e 100644 --- a/docs/longhorn-second-ssd.md +++ b/docs/longhorn-second-ssd.md @@ -45,6 +45,15 @@ between them live. longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS don't depend on them. Deleting a snapshot merges it into the next one, which writes a lot, so do this in a quiet hour, one volume at a time. + Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's + old snapshots were removed at 14:43 UTC and Longhorn merged them one + after another, about 45 minutes each, until 23:15. For those 8.5 hours + sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on + average, with "full" IO pressure at 45-75% (`~/iosample.log`); the + instance manager wrote 2.3 TB and read 2.8 TB in 30 hours. What is left + on immich-claim is one 233 GiB snapshot (`system-backup-0b7f8c6047294b36`, + from the Longhorn system backup of 2026-10-22) that everything merged + into; 148 snapshots hold 273 GiB across all volumes. - [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem") so blocks of deleted files are released. - [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a diff --git a/docs/roadmap.md b/docs/roadmap.md index 41f227b..3b6835b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -19,8 +19,9 @@ through ntfy. node001 runs mail and tunnels the public hosts home. From docs/upgrade-k8s-1.33.md, docs/immich-scouts-online.md and docs/longhorn-second-ssd.md: -- [ ] Read `~/iosample.log` on opti1 once it stops (2026-10-25 14:35 UTC): - sda's latency during the nightly Longhorn and Velero backups +- [x] Read `~/iosample.log` on opti1: the worst stretch was not the + backups but 8.5 hours of Longhorn merging immich-claim's deleted + snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md - [ ] From 2026-10-27, if immich runs without problems: delete `immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB), the Longhorn snapshots and backups `immich-pg14-pre-vchord` and @@ -38,12 +39,14 @@ docs/longhorn-second-ssd.md: ### 1. hensen.io expires -foto.scoutingmbg.nl is a CNAME to mbg.hensen.io, so when hensen.io lapses -the group loses its photos. +foto.scoutingmbg.nl was a CNAME to mbg.hensen.io. Since 2026-10-24 it is a +CNAME to rubenhensen.nl (node001, which tunnels it home), but only on the +group's ns2: ns1 is stuck on an old serial and still answers +mbg.hensen.io (docs/immich-scouts-online.md). - [x] Expiry date: 2027-01-09 (whois, 2026-10-25; registrar Key-Systems) -- [ ] Before then: renew hensen.io for a year, or have the group's DNS - point foto at the cluster without hensen.io +- [ ] Have the group's DNS host fix ns1; keep hensen.io until it answers + rubenhensen.nl too ### 2. Off-site backups diff --git a/docs/upgrade-k8s-1.33.md b/docs/upgrade-k8s-1.33.md index e489334..235a3fa 100644 --- a/docs/upgrade-k8s-1.33.md +++ b/docs/upgrade-k8s-1.33.md @@ -267,9 +267,24 @@ whatever time of day the hourly schedule had drifted to. CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway (`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and global manager have no setting for it -- [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s +- [x] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put - numbers on the disk latency during tonight's backups + numbers on the disk latency during tonight's backups. Read on 2026-10-25 + at 02:35 UTC, per 10 minutes: + - Quiet: ~50 writes/s, 0.5-2 ms per write on average, "full" IO + pressure under 15% + - 14:30-23:15: 250-440 writes/s and 200-400 reads/s, 6-20 ms per write, + pressure 45-75%. Longhorn merging immich-claim's deleted snapshots + (docs/longhorn-second-ssd.md) + - Longhorn backups 00:00-00:15: up to 1,220 reads/s, pressure 69%; the + Velero backup at 01:00 was incremental and short (pressure 29%) + - The authentik and Vault upgrades 01:20-01:40 (image pulls, + migrations): 19-33 ms per write, single 2s windows near 2 s per write, + pressure 78%. The CSI sidecars and Longhorn's global manager lost + their leases at 01:31 + Ubuntu's sysstat already keeps per-device history every 10 minutes + without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at + ~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging - [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with