mirror of
https://github.com/rubenhensen/k8scd.git
synced 2026-11-16 06:29:09 +01:00
docs: who cleaned up the snapshots, and the stalls per minute
Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
This commit is contained in:
1 parent
3e252799a4
commit
0ad6ca816c
2 files changed
+10
-2
No files matched your search
@@ -38,14 +38,15 @@ between them live.
|
|||||||
|
|
||||||
## Before: shrink what has to move
|
## Before: shrink what has to move
|
||||||
|
|
||||||
- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
|
- [x] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
|
||||||
snapshots from before October 2026, nearly all on immich-claim
|
snapshots from before October 2026, nearly all on immich-claim
|
||||||
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
|
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
|
||||||
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
|
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
|
||||||
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
|
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
|
||||||
don't depend on them. Deleting a snapshot merges it into the next one,
|
don't depend on them. Deleting a snapshot merges it into the next one,
|
||||||
which writes a lot, so do this in a quiet hour, one volume at a time.
|
which writes a lot, so do this in a quiet hour, one volume at a time.
|
||||||
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
|
Happened on 2026-10-24, from a Claude Code session at Ruben's request
|
||||||
|
(the 45 snapshots from before October 2026, small volumes first): immich-claim's
|
||||||
old snapshots were removed at 14:43 UTC and Longhorn merged them one
|
old snapshots were removed at 14:43 UTC and Longhorn merged them one
|
||||||
after another, about 45 minutes each, until 23:15. For those 8.5 hours
|
after another, about 45 minutes each, until 23:15. For those 8.5 hours
|
||||||
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
|
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
|
||||||
|
|||||||
@@ -282,6 +282,13 @@ whatever time of day the hourly schedule had drifted to.
|
|||||||
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
|
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
|
||||||
pressure 78%. The CSI sidecars and Longhorn's global manager lost
|
pressure 78%. The CSI sidecars and Longhorn's global manager lost
|
||||||
their leases at 01:31
|
their leases at 01:31
|
||||||
|
Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25
|
||||||
|
11:30, leaving out the CNPG range cancels): all 67 minutes with errors
|
||||||
|
had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and
|
||||||
|
"full" pressure at 45-78%, against a median minute peak of 24 ms. The
|
||||||
|
worst minute after the merges was the rollout of eight merged PRs at
|
||||||
|
2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller,
|
||||||
|
CNPG and Envoy Gateway held their 60s leases through all of it
|
||||||
Ubuntu's sysstat already keeps per-device history every 10 minutes
|
Ubuntu's sysstat already keeps per-device history every 10 minutes
|
||||||
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
|
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
|
||||||
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
|
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
|
||||||
|
|||||||
Reference in new issue
Block a user