Compare commits

..
23 Commits
Author SHA1 Message Date
Ruben Hensen 46117ee658 chore(deps): update helm chart external-secrets to v2.14.0 2026-10-25 12:49:06 +01:00
Ruben Hensen 2a5bbc7f44 docs: the scoutingmbg.nl DNS questions are answered
Only the four records moved to node001, hensen.io is no longer needed, and SPF's a mechanism doesn't matter since all mail goes through Google.
2026-10-25 12:39:57 +01:00
Ruben Hensen d7b0372f1a docs: record the rest of the merged PRs 2026-10-25 12:37:25 +01:00
Ruben Hensen 0ad6ca816c docs: who cleaned up the snapshots, and the stalls per minute
Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
2026-10-25 12:35:47 +01:00
Ruben Hensen 3e252799a4 Merge pull request #26 from rubenhensen/renovate/gitea-gitea-28.x
chore(deps): update gitea/gitea docker tag to v28
2026-10-25 12:25:03 +01:00
Ruben Hensen bfd5619e9e Merge pull request #27 from rubenhensen/renovate/github.com-canonical-go-dqlite-v3-3.x
fix(deps): update module github.com/canonical/go-dqlite/v3 to v3.0.4
2026-10-25 12:24:30 +01:00
Ruben Hensen 0b38d8dec8 Merge pull request #33 from rubenhensen/requests
Set resource requests from measured usage
2026-10-25 12:23:54 +01:00
Ruben Hensen 083353fb9d Merge pull request #23 from rubenhensen/renovate/binwiederhier-ntfy-2.x
chore(deps): update binwiederhier/ntfy docker tag to v2.34.0
2026-10-25 12:22:45 +01:00
Ruben Hensen 23f9c5c60a docs: record the merged Renovate PRs 2026-10-25 12:21:38 +01:00
Ruben Hensen a8bf3d5b64 Merge pull request #31 from rubenhensen/renovate/python-3.x
chore(deps): update python docker tag to v3.15.1
2026-10-25 12:17:31 +01:00
Ruben Hensen 618954ae2c Merge pull request #25 from rubenhensen/renovate/wordpress-0.x
chore(deps): update helm chart wordpress to v0.16.7
2026-10-25 12:17:17 +01:00
Ruben Hensen 44dc85d306 Merge pull request #24 from rubenhensen/renovate/https-github.com-kubernetes-csi-external-snapshotter.git-8.x
chore(deps): update dependency https://github.com/kubernetes-csi/external-snapshotter.git to v8.7.0
2026-10-25 12:16:25 +01:00
Ruben Hensen b60735f6b0 Merge pull request #28 from rubenhensen/renovate/cert-manager-1.21.x
chore(deps): update helm chart cert-manager to v1.21.3
2026-10-25 12:14:55 +01:00
Ruben Hensen 3a57e5be48 docs: WordPress went out of memory under a crawler 2026-10-25 09:45:33 +01:00
Ruben Hensen 1611f5d1ef docs: PR 33's window and Renovate's PR limit 2026-10-25 09:44:20 +01:00
Ruben Hensen 68b692a112 docs: requests are in PR 33 2026-10-25 07:10:37 +01:00
renovate[bot] 6b2f5855d7 chore(deps): update python docker tag to v3.15.1 2026-10-25 03:10:52 +00:00
renovate[bot] c25b8883da chore(deps): update gitea/gitea docker tag to v28 2026-10-25 03:06:44 +00:00
renovate[bot] 18f9e2efe6 chore(deps): update helm chart wordpress to v0.16.7 2026-10-25 03:06:35 +00:00
renovate[bot] 5f5f5b01f0 chore(deps): update dependency https://github.com/kubernetes-csi/external-snapshotter.git to v8.7.0 2026-10-25 03:06:27 +00:00
renovate[bot] 6206ab5a74 chore(deps): update binwiederhier/ntfy docker tag to v2.34.0 2026-10-25 03:06:23 +00:00
renovate[bot] 075b3686c3 chore(deps): update helm chart cert-manager to v1.21.3 2026-10-25 03:05:55 +00:00
renovate[bot] 45d1ea0d93 fix(deps): update module github.com/canonical/go-dqlite/v3 to v3.0.4 2026-10-23 20:04:51 +00:00
13 changed files with 75 additions and 52 deletions

No files matched your search

+1 -1
View File
@@ -12,7 +12,7 @@ spec:
project: default
source:
repoURL: https://charts.jetstack.io
targetRevision: v1.21.2
targetRevision: v1.21.3
chart: cert-manager
helm:
parameters:
+1 -1
View File
@@ -17,7 +17,7 @@ spec:
sources:
- chart: external-secrets
repoURL: https://charts.external-secrets.io
targetRevision: 2.13.0
targetRevision: 2.14.0
helm:
values: |
resources:
+1 -1
View File
@@ -12,7 +12,7 @@ spec:
project: default
source:
repoURL: https://groundhog2k.github.io/helm-charts/
targetRevision: 0.15.0
targetRevision: 0.16.7
chart: wordpress
helm:
values: |
+1 -1
View File
@@ -7,7 +7,7 @@ spec:
project: default
source:
repoURL: https://github.com/kubernetes-csi/external-snapshotter.git
targetRevision: v8.2.1
targetRevision: v8.7.0
path: deploy/kubernetes/snapshot-controller
kustomize:
patches:
+7 -9
View File
@@ -123,15 +123,13 @@ D. auth.scoutingmbg.nl (2026-10-25)
## Open points
- DNS (2026-10-24): the group's DNS host (ns1/ns2.webhosted.nl) changed
foto.scoutingmbg.nl from a CNAME to `mbg.hensen.io` (the home IP) to a
CNAME to `rubenhensen.nl`, which is node001 on Hetzner. Only ns2 has it
(serial 2026102471); ns1 is stuck on serial 2026060640 from June and
still answers `mbg.hensen.io`. node001 now tunnels foto.scoutingmbg.nl to
the home cluster like its other hosts, so both answers work. Keep
hensen.io until ns1 has caught up; ask the DNS host to fix ns1, and
decide whether foto should stay behind node001 or point straight home
(CNAME `scoutingmbg.nl`)
- DNS (2026-10-24/25): the group's DNS host (ns1/ns2.webhosted.nl) pointed
scoutingmbg.nl, www, foto and auth at node001 (rubenhensen.nl); those are
all the records it moved, and node001 tunnels all four home. ns1 lagged
behind ns2 (a zone serial from June) until 2026-10-24 16:50 UTC. Nothing
uses mbg.hensen.io anymore, so hensen.io can expire. The SPF record's
`a` now means node001, which is fine: the WordPress site sends no mail
and the group's mail goes through Google (`mx`)
- Public shared links become public for real once sol-auth is gone.
- Photos leiding back up from their phones stay private until they add
them to a shared album.
+3 -2
View File
@@ -38,14 +38,15 @@ between them live.
## Before: shrink what has to move
- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
- [x] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
snapshots from before October 2026, nearly all on immich-claim
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
don't depend on them. Deleting a snapshot merges it into the next one,
which writes a lot, so do this in a quiet hour, one volume at a time.
Happened on 2026-10-24 (who started it isn't in the logs): immich-claim's
Happened on 2026-10-24, from a Claude Code session at Ruben's request
(the 45 snapshots from before October 2026, small volumes first): immich-claim's
old snapshots were removed at 14:43 UTC and Longhorn merged them one
after another, about 45 minutes each, until 23:15. For those 8.5 hours
sda ran 250-440 writes/s and 200-400 reads/s at 6-20 ms per write on
+45 -33
View File
@@ -13,7 +13,8 @@ volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the
NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to MinIO at
01:00 (daily kept 30 days, weekly 90 days). Gatus on s1 (node001, Hetzner)
and s2 (home) watch each other and alert through ntfy; s2 also gets an
hourly report on the backups and Vault's seal from inside the cluster.
hourly report from inside the cluster on the backups, Vault's seal, the
Argo CD apps and the CNPG clusters.
Every app namespace has NetworkPolicies, and CI renders and validates all
apps on every PR. node001 runs mail and tunnels the public hosts home.
@@ -33,8 +34,9 @@ docs/longhorn-second-ssd.md:
- [ ] The upgrade's dqlite and backend copies in `/root/` on opti1
(`backend-*.tar`, `dqlite-pre-*`)
- [ ] Log in to immich as a jeugdlid (quota 0)
- [ ] Back to 1.36/stable once it is at v1.36.5 or later. The snap is held,
so microk8s gets no updates at all until someone refreshes it
- [ ] Back to 1.36/stable once it is at v1.36.5 or later (still v1.36.2 on
2026-10-25). The snap is held, so microk8s gets no updates at all until
someone refreshes it
- [ ] Second SSD, docs/longhorn-second-ssd.md
- [ ] Optional: `MACHINE_LEARNING_MODEL_REVISION=v2` for immich ML
@@ -265,8 +267,29 @@ Counted 2026-10-25 over the 73 running containers:
external-secrets, envoy-gateway-system, velero, mariadb-system, dns,
default) still have none; `default` can reach the LAN and everything
without a policy
- [ ] Requests and memory limits, the big ones first (immich, authentik,
Argo CD's repo-server)
- [x] Requests: PR #33, merged 2026-10-25 11:23 UTC, for everything that
had none except the Vault server, from `kubectl top` every 5 minutes
from 03:13 to 08:40 UTC (highest memory plus 10%, CPU at p95, 10m
floor). Rolled out in about eight minutes; both CNPG instances
restarted and came back healthy, the OIDC endpoints, the logins and
immich answered, no OOM kills. The node now has 3770m CPU (62%, of which
1440m is Longhorn's two instance managers) and 9158Mi memory (28%)
requested. Still without requests: Longhorn (22 containers, its chart),
the Vault servers, the LDAP outpost, immich's PG14 cluster and three in
kube-system
- [ ] Memory limits, once there are daytime numbers: the night ones miss
immich's machine learning at work (it idles at 234Mi)
- [ ] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on
2026-10-25, its third restart: a crawler asked for 88 pages and assets
within a few seconds (normally ~6 a minute), Apache started a PHP worker
for each, and the container went past its 1Gi limit (500s first, then
the kill; back within seconds). Unrelated to the NetworkPolicies; memory
sat at ~500Mi before. Fixes, in order: cap Apache's prefork workers
(`MaxRequestWorkers` ~16, so a burst queues instead of filling memory),
a page cache plugin. Rate limiting per visitor in Envoy doesn't work
yet: node001 passes TLS through, so every request arrives from
46.224.26.65; that needs the PROXY protocol from node001's nginx to Envoy
(ClientTrafficPolicy `enableProxyProtocol`)
- [x] Probes for gitea (`/api/healthz`), freshrss (TCP, since
mod_auth_openidc answers 401 to non-browsers) and serpbear (`/login`)
(2026-10-25). Liveness only after 5 failures 30s apart, to ride out the
@@ -314,34 +337,23 @@ PR before Argo CD applies it.
cover every app folder, a regex manager reads `imageName` in
`*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off
(those need pg_upgrade or a dump). Expect a batch of new PRs
(2026-10-25; `renovate-config-validator` passes)
- [ ] The open PRs, all validated on 2026-10-25 against main as it is now
(rebases requested, so the checks show on the PRs once Renovate has
pushed). Merging is yours; merging deploys:
- #28 cert-manager v1.21.3: security release, "All users should
upgrade" (drops the compromised eMudhra root, fixes an ARI renewal
loop and a private key copy path). Merge first
- #24 external-snapshotter v8.2.1 → v8.7.0: 8.6 moved VolumeGroupSnapshot
to v1 and 8.7 turns its feature gate on; the cluster already serves the
v1 CRDs. Merge; close #15
- #25 WordPress chart 0.15.0 → 0.16.7: the image is pinned
(7.1.3-php8.3-apache), so the render only changes labels and
checksums (a pod restart). Merge; #19 (0.15.1) is superseded
- #26 Gitea 1.27.3 → 28.2.0 (Gitea renumbered after 1.27): security
fixes; the breaking changes are to egress host lists
(`ALLOWED_DOMAINS` and friends), none of which are set here. Database
migrations can't be undone, so merge after a fresh backup
- #23 ntfy v2.28.0 → v2.34.0: six releases in 17 days; 2.31.0 had a
database load bug that ntfy.sh rolled back (fixed in 2.31.1). No
breaking migrations. Fine, or wait a few days for 2.34 to settle
- #17 external-secrets 2.13.0 → 2.14.0: no breaking changes in the
notes; every secret goes through it, so merge when there's time to
watch a force-sync
- #27 go-dqlite v3.0.4 for `tools/dqlite-vacuum`: builds; harmless
- #29 yaml.v3 for the same tool: builds, but the code imports yaml.v2,
so it only adds an unused requirement. Close
- #31 python 3.15.1 for `gatus/backup-check.yaml`: the check's tests
(made-up data for every failure) pass on that image. Merge
(2026-10-25; `renovate-config-validator` passes). They only come once
fewer than 10 Renovate PRs are open (`config:recommended`'s
`prConcurrentLimit`); until then they wait under "Rate-Limited" on the
Dependency Dashboard (#2)
- [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager
v1.21.3 (security release; all three pods on v1.21.3), #24
external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as
v8.2.1 shipped v8.2.0; both replicas up, no errors), #25 WordPress chart
0.16.7 (image unchanged, site 200), #31 python 3.15.1 (the cluster check
ran clean on it). Closed: #19, #29, and #15 by Renovate. Later the same
day: #23 ntfy v2.34.0, #27 go-dqlite v3.0.4, #33 (see 7) and #26 Gitea
28.2.0 (migrations up to 357 ran clean, `/api/healthz` 200; no backup of
its own beforehand, the nightly ones are the way back)
- [ ] Still open: #17 external-secrets 2.13.0 → 2.14.0, no breaking
changes in the notes; every secret goes through it, so merge when
there's time to watch a force-sync. It only changes `targetRevision`, so
it still merges cleanly after #33 added values to the same file
### 9. One node
+7
View File
@@ -282,6 +282,13 @@ whatever time of day the hourly schedule had drifted to.
migrations): 19-33 ms per write, single 2s windows near 2 s per write,
pressure 78%. The CSI sidecars and Longhorn's global manager lost
their leases at 01:31
Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25
11:30, leaving out the CNPG range cancels): all 67 minutes with errors
had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and
"full" pressure at 45-78%, against a median minute peak of 24 ms. The
worst minute after the merges was the rollout of eight merged PRs at
2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller,
CNPG and Envoy Gateway held their 60s leases through all of it
Ubuntu's sysstat already keeps per-device history every 10 minutes
without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
+1 -1
View File
@@ -256,7 +256,7 @@ spec:
type: RuntimeDefault
containers:
- name: check
image: python:3.13.16-alpine3.24
image: python:3.15.1-alpine3.24
command: ["python", "/script/check.py"]
env:
- name: PYTHONDONTWRITEBYTECODE
+1 -1
View File
@@ -20,7 +20,7 @@ spec:
fsGroup: 1000
containers:
- name: gitea
image: gitea/gitea:1.27.3
image: gitea/gitea:28.2.0
env:
- name: USER_UID
value: "1000"
+1 -1
View File
@@ -16,7 +16,7 @@ spec:
spec:
containers:
- name: ntfy
image: binwiederhier/ntfy:v2.28.0
image: binwiederhier/ntfy:v2.34.0
args: ["serve"]
env:
- name: NTFY_BASE_URL
+1 -1
View File
@@ -3,7 +3,7 @@ module dqlite-vacuum
go 1.27.3
require (
github.com/canonical/go-dqlite/v3 v3.0.1
github.com/canonical/go-dqlite/v3 v3.0.4
gopkg.in/yaml.v2 v2.4.0
)
+5
View File
@@ -2,7 +2,10 @@ github.com/Rican7/retry v0.3.1 h1:scY4IbO8swckzoA/11HgBwaZRJEyY9vaNJshcdhp1Mc=
github.com/Rican7/retry v0.3.1/go.mod h1:CxSDrhAyXmTMeEuRAnArMu1FHu48vtfjLREWqVl7Vw0=
github.com/canonical/go-dqlite/v3 v3.0.1 h1:eUTcEHetVi1+zr+ZFz7PU0G+0eK66GZ7DKhi+wHs1f0=
github.com/canonical/go-dqlite/v3 v3.0.1/go.mod h1:Kb/9JVog9XeIlnlnSyO1JgMoE4PoKXZCB8fJNcvvHAc=
github.com/canonical/go-dqlite/v3 v3.0.4 h1:JhITsrvVL0pii9arxDksXfsRqBI+je1VpcqVuzDllUM=
github.com/canonical/go-dqlite/v3 v3.0.4/go.mod h1:6O+E6MiYesijRUOECyywp16iT+N4QOrvjDWWiMJ/xF0=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -26,7 +29,9 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
github.com/spf13/cobra v1.10.1/go.mod h1:7SmJGaTHFVBY0jW4NXGluQoLvhqFQM+6XSKD+P4XaB0=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=