From 0ad6ca816c04b01e7edd6ebbd376ec8666bf111f Mon Sep 17 00:00:00 2001 From: Ruben Hensen Date: Sun, 25 Oct 2026 12:35:47 +0100 Subject: [PATCH] docs: who cleaned up the snapshots, and the stalls per minute Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held. --- docs/longhorn-second-ssd.md | 5 +++-- docs/upgrade-k8s-1.33.md | 7 +++++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/longhorn-second-ssd.md b/docs/longhorn-second-ssd.md index b5bad0b..c838241 100644 --- a/docs/longhorn-second-ssd.md +++ b/docs/longhorn-second-ssd.md @@ -38,14 +38,15 @@ between them live. ## Before: shrink what has to move -- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in +- [x] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in snapshots from before October 2026, nearly all on immich-claim (pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from 2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS don't depend on them. Deleting a snapshot merges it into the next one, which writes a lot, so do this in a quiet hour, one volume at a time. - Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's + Happened on 2026-10-24, from a Claude Code session at Ruben's request + (the 45 snapshots from before October 2026, small volumes first): immich-claim's old snapshots were removed at 14:43 UTC and Longhorn merged them one after another, about 45 minutes each, until 23:15. For those 8.5 hours sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on diff --git a/docs/upgrade-k8s-1.33.md b/docs/upgrade-k8s-1.33.md index 235a3fa..9093bb0 100644 --- a/docs/upgrade-k8s-1.33.md +++ b/docs/upgrade-k8s-1.33.md @@ -282,6 +282,13 @@ whatever time of day the hourly schedule had drifted to. migrations): 19-33 ms per write, single 2s windows near 2 s per write, pressure 78%. The CSI sidecars and Longhorn's global manager lost their leases at 01:31 + Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25 + 11:30, leaving out the CNPG range cancels): all 67 minutes with errors + had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and + "full" pressure at 45-78%, against a median minute peak of 24 ms. The + worst minute after the merges was the rollout of eight merged PRs at + 2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller, + CNPG and Envoy Gateway held their 60s leases through all of it Ubuntu's sysstat already keeps per-device history every 10 minutes without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at ~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging