mirror of
https://github.com/rubenhensen/k8scd.git
synced 2026-11-15 14:29:09 +01:00
docs: who cleaned up the snapshots, and the stalls per minute
Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
This commit is contained in:
1 parent
3e252799a4
commit
0ad6ca816c
2 files changed
+10
-2
No files matched your search
@@ -38,14 +38,15 @@ between them live.
|
||||
|
||||
## Before: shrink what has to move
|
||||
|
||||
- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
|
||||
- [x] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
|
||||
snapshots from before October 2026, nearly all on immich-claim
|
||||
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
|
||||
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
|
||||
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
|
||||
don't depend on them. Deleting a snapshot merges it into the next one,
|
||||
which writes a lot, so do this in a quiet hour, one volume at a time.
|
||||
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
|
||||
Happened on 2026-10-24, from a Claude Code session at Ruben's request
|
||||
(the 45 snapshots from before October 2026, small volumes first): immich-claim's
|
||||
old snapshots were removed at 14:43 UTC and Longhorn merged them one
|
||||
after another, about 45 minutes each, until 23:15. For those 8.5 hours
|
||||
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
|
||||
|
||||
@@ -282,6 +282,13 @@ whatever time of day the hourly schedule had drifted to.
|
||||
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
|
||||
pressure 78%. The CSI sidecars and Longhorn's global manager lost
|
||||
their leases at 01:31
|
||||
Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25
|
||||
11:30, leaving out the CNPG range cancels): all 67 minutes with errors
|
||||
had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and
|
||||
"full" pressure at 45-78%, against a median minute peak of 24 ms. The
|
||||
worst minute after the merges was the rollout of eight merged PRs at
|
||||
2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller,
|
||||
CNPG and Envoy Gateway held their 60s leases through all of it
|
||||
Ubuntu's sysstat already keeps per-device history every 10 minutes
|
||||
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
|
||||
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
|
||||
|
||||
Reference in new issue
Block a user