mirror of
https://github.com/rubenhensen/k8scd.git
synced 2026-11-16 06:29:09 +01:00
docs: what the disk did in the last day, and foto's DNS
This commit is contained in:
1 parent
cadac4cebf
commit
1f56e67e8b
3 files changed
+35
-8
No files matched your search
@@ -45,6 +45,15 @@ between them live.
|
|||||||
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
|
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
|
||||||
don't depend on them. Deleting a snapshot merges it into the next one,
|
don't depend on them. Deleting a snapshot merges it into the next one,
|
||||||
which writes a lot, so do this in a quiet hour, one volume at a time.
|
which writes a lot, so do this in a quiet hour, one volume at a time.
|
||||||
|
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
|
||||||
|
old snapshots were removed at 14:43 UTC and Longhorn merged them one
|
||||||
|
after another, about 45 minutes each, until 23:15. For those 8.5 hours
|
||||||
|
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
|
||||||
|
average, with "full" IO pressure at 45-75% (`~/iosample.log`); the
|
||||||
|
instance manager wrote 2.3 TB and read 2.8 TB in 30 hours. What is left
|
||||||
|
on immich-claim is one 233 GiB snapshot (`system-backup-0b7f8c6047294b36`,
|
||||||
|
from the Longhorn system backup of 2026-10-22) that everything merged
|
||||||
|
into; 148 snapshots hold 273 GiB across all volumes.
|
||||||
- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem")
|
- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem")
|
||||||
so blocks of deleted files are released.
|
so blocks of deleted files are released.
|
||||||
- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a
|
- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a
|
||||||
|
|||||||
+9
-6
@@ -19,8 +19,9 @@ through ntfy. node001 runs mail and tunnels the public hosts home.
|
|||||||
From docs/upgrade-k8s-1.33.md, docs/immich-scouts-online.md and
|
From docs/upgrade-k8s-1.33.md, docs/immich-scouts-online.md and
|
||||||
docs/longhorn-second-ssd.md:
|
docs/longhorn-second-ssd.md:
|
||||||
|
|
||||||
- [ ] Read `~/iosample.log` on opti1 once it stops (2026-10-25 14:35 UTC):
|
- [x] Read `~/iosample.log` on opti1: the worst stretch was not the
|
||||||
sda's latency during the nightly Longhorn and Velero backups
|
backups but 8.5 hours of Longhorn merging immich-claim's deleted
|
||||||
|
snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md
|
||||||
- [ ] From 2026-10-27, if immich runs without problems: delete
|
- [ ] From 2026-10-27, if immich runs without problems: delete
|
||||||
`immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB),
|
`immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB),
|
||||||
the Longhorn snapshots and backups `immich-pg14-pre-vchord` and
|
the Longhorn snapshots and backups `immich-pg14-pre-vchord` and
|
||||||
@@ -38,12 +39,14 @@ docs/longhorn-second-ssd.md:
|
|||||||
|
|
||||||
### 1. hensen.io expires
|
### 1. hensen.io expires
|
||||||
|
|
||||||
foto.scoutingmbg.nl is a CNAME to mbg.hensen.io, so when hensen.io lapses
|
foto.scoutingmbg.nl was a CNAME to mbg.hensen.io. Since 2026-10-24 it is a
|
||||||
the group loses its photos.
|
CNAME to rubenhensen.nl (node001, which tunnels it home), but only on the
|
||||||
|
group's ns2: ns1 is stuck on an old serial and still answers
|
||||||
|
mbg.hensen.io (docs/immich-scouts-online.md).
|
||||||
|
|
||||||
- [x] Expiry date: 2027-01-09 (whois, 2026-10-25; registrar Key-Systems)
|
- [x] Expiry date: 2027-01-09 (whois, 2026-10-25; registrar Key-Systems)
|
||||||
- [ ] Before then: renew hensen.io for a year, or have the group's DNS
|
- [ ] Have the group's DNS host fix ns1; keep hensen.io until it answers
|
||||||
point foto at the cluster without hensen.io
|
rubenhensen.nl too
|
||||||
|
|
||||||
### 2. Off-site backups
|
### 2. Off-site backups
|
||||||
|
|
||||||
|
|||||||
@@ -267,9 +267,24 @@ whatever time of day the hourly schedule had drifted to.
|
|||||||
CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway
|
CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway
|
||||||
(`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and
|
(`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and
|
||||||
global manager have no setting for it
|
global manager have no setting for it
|
||||||
- [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
|
- [x] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
|
||||||
for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
|
for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
|
||||||
numbers on the disk latency during tonight's backups
|
numbers on the disk latency during tonight's backups. Read on 2026-10-25
|
||||||
|
at 02:35 UTC, per 10 minutes:
|
||||||
|
- Quiet: ~50 writes/s, 0.5-2 ms per write on average, "full" IO
|
||||||
|
pressure under 15%
|
||||||
|
- 14:30-23:15: 250-440 writes/s and 200-400 reads/s, 6-20 ms per write,
|
||||||
|
pressure 45-75%. Longhorn merging immich-claim's deleted snapshots
|
||||||
|
(docs/longhorn-second-ssd.md)
|
||||||
|
- Longhorn backups 00:00-00:15: up to 1,220 reads/s, pressure 69%; the
|
||||||
|
Velero backup at 01:00 was incremental and short (pressure 29%)
|
||||||
|
- The authentik and Vault upgrades 01:20-01:40 (image pulls,
|
||||||
|
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
|
||||||
|
pressure 78%. The CSI sidecars and Longhorn's global manager lost
|
||||||
|
their leases at 01:31
|
||||||
|
Ubuntu's sysstat already keeps per-device history every 10 minutes
|
||||||
|
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
|
||||||
|
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
|
||||||
- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
|
- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
|
||||||
i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
|
i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
|
||||||
present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with
|
present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with
|
||||||
|
|||||||
Reference in new issue
Block a user