docs: who cleaned up the snapshots, and the stalls per minute

Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
This commit is contained in:
Ruben Hensen committed 2026-10-25 12:35:47 +01:00
1 parent 3e252799a4
commit 0ad6ca816c
2 files changed
+10 -2

No files matched your search

+3 -2
View File
@@ -38,14 +38,15 @@ between them live.
## Before: shrink what has to move
- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
- [x] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
snapshots from before October 2026, nearly all on immich-claim
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
don't depend on them. Deleting a snapshot merges it into the next one,
which writes a lot, so do this in a quiet hour, one volume at a time.
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
Happened on 2026-10-24, from a Claude Code session at Ruben's request
(the 45 snapshots from before October 2026, small volumes first): immich-claim's
old snapshots were removed at 14:43 UTC and Longhorn merged them one
after another, about 45 minutes each, until 23:15. For those 8.5 hours
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
+7
View File
@@ -282,6 +282,13 @@ whatever time of day the hourly schedule had drifted to.
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
pressure 78%. The CSI sidecars and Longhorn's global manager lost
their leases at 01:31
Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25
11:30, leaving out the CNPG range cancels): all 67 minutes with errors
had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and
"full" pressure at 45-78%, against a median minute peak of 24 ms. The
worst minute after the merges was the rollout of eight merged PRs at
2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller,
CNPG and Envoy Gateway held their 60s leases through all of it
Ubuntu's sysstat already keeps per-device history every 10 minutes
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging