Ruben Hensen
c359384ade
renovate: look at every app folder, and at CNPG images
2026-10-25 04:24:56 +01:00
Ruben Hensen
a4fc386ee8
docs: record the new probes
2026-10-25 04:22:28 +01:00
Ruben Hensen
0dc0bc4322
docs: authentik's base URL is already set
2026-10-25 04:22:06 +01:00
Ruben Hensen
d843f31d86
serpbear: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
da23357946
freshrss: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
b9d88f41b9
gitea: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
3c4a8cc8db
docs: bring the authentik README up to date
2026-10-25 04:16:48 +01:00
Ruben Hensen
d17e78130d
docs: record metrics-server and plan the restore test
2026-10-25 04:15:43 +01:00
Ruben Hensen
22c5a325d6
Add metrics-server
2026-10-25 04:11:59 +01:00
Ruben Hensen
ef07307f93
renovate: keep CI's Helm at Argo CD's version
2026-10-25 04:10:30 +01:00
Ruben Hensen
aa940134ea
docs: record the CI and the Renovate triage
2026-10-25 04:10:14 +01:00
Ruben Hensen
f86ed09cd8
renovate: drop the github-tags manager, which duplicated the argocd one
2026-10-25 04:09:52 +01:00
Ruben Hensen
6d2bc379ba
ci: retry when a registry rate-limits
2026-10-25 04:06:53 +01:00
Ruben Hensen
67c21c30d8
Render and validate every app on PRs and pushes
2026-10-25 03:57:24 +01:00
Ruben Hensen
0c52d8a467
apps: list external-snapshotter's automated sync once
2026-10-25 03:57:24 +01:00
Ruben Hensen
f2e4c5aff6
docs: record the backup check
2026-10-25 03:49:43 +01:00
Ruben Hensen
d836e4608c
gatus: check backups and Vault's seal from inside the cluster
2026-10-25 03:48:05 +01:00
Ruben Hensen
1f56e67e8b
docs: what the disk did in the last day, and foto's DNS
2026-10-25 03:42:13 +01:00
Ruben Hensen
cadac4cebf
docs: record the cleanup and what is left of it
2026-10-25 03:29:55 +01:00
Ruben Hensen
bc71061db7
Remove commented-out files, the empty secrets app and a stale script copy
2026-10-25 03:27:32 +01:00
Ruben Hensen
7d3a64e5c8
docs: record the NetworkPolicies for the other apps and Vault
2026-10-25 03:24:07 +01:00
Ruben Hensen
e137e79b75
vault: only ESO, Envoy and the node reach Vault
2026-10-25 03:21:53 +01:00
Ruben Hensen
67dc4f0783
Limit the other public apps to their own pods, DNS and the internet
2026-10-25 03:21:53 +01:00
Ruben Hensen
684f536f2f
docs: record the NetworkPolicies for immich, mbgwp and authentik
2026-10-25 03:17:11 +01:00
Ruben Hensen
1439466af4
authentik: limit the pods to each other, DNS and the internet
2026-10-25 03:15:20 +01:00
Ruben Hensen
57514c3c14
immich: limit the pods to each other, DNS and the internet
2026-10-25 03:12:02 +01:00
Ruben Hensen
57851bf43a
mbgwp: limit WordPress to its database, DNS and the internet
2026-10-25 03:09:06 +01:00
Ruben Hensen
ada909ee02
docs: plan off-site backups through a friend's bucket
2026-10-25 03:05:39 +01:00
Ruben Hensen
345de16552
docs: record the Vault 2.2.1 upgrade
2026-10-25 02:52:52 +01:00
Ruben Hensen
7396cc3ed6
vault: fix the listing-visibility flag in the OIDC job
2026-10-25 02:52:02 +01:00
Ruben Hensen
26ca83457c
vault: upgrade to 2.2.1, with a script to roll the pods
2026-10-25 02:40:45 +01:00
Ruben Hensen
4709194483
docs: tick off the authentik login test
2026-10-25 02:40:45 +01:00
Ruben Hensen
ad2d6fbae5
docs: record the authentik 2026.8 upgrade
2026-10-25 02:34:51 +01:00
Ruben Hensen
077711472e
authentik: upgrade to 2026.8.5
2026-10-25 02:31:00 +01:00
Ruben Hensen
43f741141d
authentik: upgrade to 2026.5.6
2026-10-25 02:23:15 +01:00
Ruben Hensen
cbee4955d3
authentik: drop the redis values, which the chart no longer has
2026-10-25 02:23:15 +01:00
Ruben Hensen
a70dff6fdd
docs: add a roadmap in place of the wishlist
2026-10-25 02:16:31 +01:00
Ruben Hensen
ab0a09a356
docs: the immich login moved to auth.scoutingmbg.nl
2026-10-25 02:08:59 +01:00
Ruben Hensen
b2649a682c
immich: log in through auth.scoutingmbg.nl
...
The Scouts Online connection now redirects to auth.scoutingmbg.nl, the
group's own domain for the authentik login, so immich's OAuth issuer
moves there too.
2026-10-25 02:03:50 +01:00
Ruben Hensen
ce3846bcd3
node001: tunnel scoutingmbg.nl and www to the home cluster
...
The group's DNS moved scoutingmbg.nl (A to node001) and www (CNAME to
rubenhensen.nl) here as well, so the WordPress site got node001's mail
certificate. Pass them through to the home cluster like foto and auth.
2026-10-25 02:59:00 +02:00
Ruben Hensen
96c13c63f7
authentik: serve auth.scoutingmbg.nl
...
A Gateway listener with its own certificate, the authentik route on both
hostnames, and an authentik brand for auth.scoutingmbg.nl that logs in
through Scouts Online and then sends people to the photos. The
authentik.rubenhensen.nl brand stays as it is.
2026-10-24 18:52:24 +02:00
Ruben Hensen
70151b0ef4
authentik: certificate for auth.scoutingmbg.nl
...
The login of the Scouting MBG photos moves to the group's own domain; the Gateway listener follows once the certificate is there.
2026-10-24 18:51:27 +02:00
Ruben Hensen
f5e8b1e633
node001: tunnel auth.scoutingmbg.nl to the home cluster
...
The group's DNS points auth.scoutingmbg.nl at rubenhensen.nl too; it will carry the authentik login for immich.
2026-10-24 17:58:34 +02:00
Ruben Hensen
ae2e5ad9fa
docs: record the foto.scoutingmbg.nl DNS change
2026-10-24 17:48:35 +02:00
Ruben Hensen
8cf87483d6
node001: tunnel foto.scoutingmbg.nl to the home cluster
...
The group's DNS now points foto.scoutingmbg.nl at rubenhensen.nl, which is
this server, instead of mbg.hensen.io (the home IP). Resolvers that had
picked up the change got node001's own certificate and no immich. Add the
host to the SNI passthrough and the port 80 proxy like the other tunneled
hosts, so it works through either record.
2026-10-24 17:47:14 +02:00
Ruben Hensen
c5d2ebaefc
docs: record the Envoy retry policy
2026-10-24 17:06:29 +02:00
Ruben Hensen
5f8394e0c0
docs: plan a second SSD for Longhorn
...
Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.
2026-10-24 15:17:15 +02:00
Ruben Hensen
396eb2665c
docs: record why controllers still lost their leases
...
Heavy writes on the node's only SSD stall dqlite; the fixes so far, the running measurement and the second-SSD option.
2026-10-24 14:46:09 +02:00
Ruben Hensen
858904384c
Give leader-elected controllers longer leases
...
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
2026-10-24 14:43:38 +02:00
Ruben Hensen
9014898fc5
velero: run kopia maintenance daily and throttled
...
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
2026-10-24 14:43:38 +02:00
Ruben Hensen