docs: what the disk did in the last day, and foto's DNS

This commit is contained in:
Ruben Hensen committed 2026-10-25 03:42:13 +01:00
1 parent cadac4cebf
commit 1f56e67e8b
3 files changed
+35 -8

No files matched your search

+9
View File
@@ -45,6 +45,15 @@ between them live.
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
don't depend on them. Deleting a snapshot merges it into the next one,
which writes a lot, so do this in a quiet hour, one volume at a time.
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
old snapshots were removed at 14:43 UTC and Longhorn merged them one
after another, about 45 minutes each, until 23:15. For those 8.5 hours
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
average, with "full" IO pressure at 45-75% (`~/iosample.log`); the
instance manager wrote 2.3 TB and read 2.8 TB in 30 hours. What is left
on immich-claim is one 233 GiB snapshot (`system-backup-0b7f8c6047294b36`,
from the Longhorn system backup of 2026-10-22) that everything merged
into; 148 snapshots hold 273 GiB across all volumes.
- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem")
so blocks of deleted files are released.
- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a
+9 -6
View File
@@ -19,8 +19,9 @@ through ntfy. node001 runs mail and tunnels the public hosts home.
From docs/upgrade-k8s-1.33.md, docs/immich-scouts-online.md and
docs/longhorn-second-ssd.md:
- [ ] Read `~/iosample.log` on opti1 once it stops (2026-10-25 14:35 UTC):
sda's latency during the nightly Longhorn and Velero backups
- [x] Read `~/iosample.log` on opti1: the worst stretch was not the
backups but 8.5 hours of Longhorn merging immich-claim's deleted
snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md
- [ ] From 2026-10-27, if immich runs without problems: delete
`immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB),
the Longhorn snapshots and backups `immich-pg14-pre-vchord` and
@@ -38,12 +39,14 @@ docs/longhorn-second-ssd.md:
### 1. hensen.io expires
foto.scoutingmbg.nl is a CNAME to mbg.hensen.io, so when hensen.io lapses
the group loses its photos.
foto.scoutingmbg.nl was a CNAME to mbg.hensen.io. Since 2026-10-24 it is a
CNAME to rubenhensen.nl (node001, which tunnels it home), but only on the
group's ns2: ns1 is stuck on an old serial and still answers
mbg.hensen.io (docs/immich-scouts-online.md).
- [x] Expiry date: 2027-01-09 (whois, 2026-10-25; registrar Key-Systems)
- [ ] Before then: renew hensen.io for a year, or have the group's DNS
point foto at the cluster without hensen.io
- [ ] Have the group's DNS host fix ns1; keep hensen.io until it answers
rubenhensen.nl too
### 2. Off-site backups
+17 -2
View File
@@ -267,9 +267,24 @@ whatever time of day the hourly schedule had drifted to.
CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway
(`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and
global manager have no setting for it
- [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
- [x] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
numbers on the disk latency during tonight's backups
numbers on the disk latency during tonight's backups. Read on 2026-10-25
at 02:35 UTC, per 10 minutes:
- Quiet: ~50 writes/s, 0.5-2 ms per write on average, "full" IO
pressure under 15%
- 14:30-23:15: 250-440 writes/s and 200-400 reads/s, 6-20 ms per write,
pressure 45-75%. Longhorn merging immich-claim's deleted snapshots
(docs/longhorn-second-ssd.md)
- Longhorn backups 00:00-00:15: up to 1,220 reads/s, pressure 69%; the
Velero backup at 01:00 was incremental and short (pressure 29%)
- The authentik and Vault upgrades 01:20-01:40 (image pulls,
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
pressure 78%. The CSI sidecars and Longhorn's global manager lost
their leases at 01:31
Ubuntu's sysstat already keeps per-device history every 10 minutes
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with