Commit Graph
100 Commits
Author SHA1 Message Date
Ruben Hensen 0de86dd65c Merge pull request #36 from rubenhensen/renovate-re2
Renovate: skip the non-app folders without a lookahead
2026-10-25 17:41:32 +01:00
Ruben Hensen d7ffdc05f6 docs: Renovate rejected the lookahead (#32) 2026-10-25 17:33:39 +01:00
Ruben Hensen c0f8034e14 renovate: skip the non-app folders through ignorePaths, not a lookahead 2026-10-25 17:33:39 +01:00
Ruben Hensen a15a4f7c75 Merge pull request #34 from rubenhensen/velero-garage-secret
Garage on the NAS, and Velero's key for it
2026-10-25 17:23:33 +01:00
Ruben Hensen b072099cc1 nas: Garage keeps its metadata in SQLite 2026-10-25 17:08:13 +01:00
Ruben Hensen e10ca337f1 velero: a key for Garage, from Vault 2026-10-25 16:28:43 +01:00
Ruben Hensen c72b08867f nas: Garage, which takes over from MinIO 2026-10-25 16:28:43 +01:00
Ruben Hensen d85362ce3b docs: the restore test passed against the real backups 2026-10-25 14:57:18 +01:00
Ruben Hensen 018f528fb3 docs: a WordPress cache without plugins 2026-10-25 14:56:10 +01:00
Ruben Hensen 714712b862 docs: WordPress is capped at 12 workers 2026-10-25 13:11:20 +01:00
Ruben Hensen 7dc13466cb mbgwp: cap Apache at 12 workers so a crawler can't run it out of memory 2026-10-25 13:07:30 +01:00
Ruben Hensen 9b77596bcb backup-testing: take endpoint, region and bucket from Velero's location 2026-10-25 12:59:25 +01:00
Ruben Hensen fbf2cb2d92 docs: the unseal key is kept safe 2026-10-25 12:57:18 +01:00
Ruben Hensen 122873ca5c docs: hensen.io can expire 2026-10-25 12:54:32 +01:00
Ruben Hensen 71449dd185 docs: record the external-secrets upgrade 2026-10-25 12:54:01 +01:00
Ruben Hensen 297d2dffa5 Merge pull request #17 from rubenhensen/renovate/external-secrets-2.x
chore(deps): update helm chart external-secrets to v2.14.0
2026-10-25 12:50:28 +01:00
Ruben Hensen 46117ee658 chore(deps): update helm chart external-secrets to v2.14.0 2026-10-25 12:49:06 +01:00
Ruben Hensen 2a5bbc7f44 docs: the scoutingmbg.nl DNS questions are answered
Only the four records moved to node001, hensen.io is no longer needed, and SPF's a mechanism doesn't matter since all mail goes through Google.
2026-10-25 12:39:57 +01:00
Ruben Hensen d7b0372f1a docs: record the rest of the merged PRs 2026-10-25 12:37:25 +01:00
Ruben Hensen 0ad6ca816c docs: who cleaned up the snapshots, and the stalls per minute
Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
2026-10-25 12:35:47 +01:00
Ruben Hensen 3e252799a4 Merge pull request #26 from rubenhensen/renovate/gitea-gitea-28.x
chore(deps): update gitea/gitea docker tag to v28
2026-10-25 12:25:03 +01:00
Ruben Hensen bfd5619e9e Merge pull request #27 from rubenhensen/renovate/github.com-canonical-go-dqlite-v3-3.x
fix(deps): update module github.com/canonical/go-dqlite/v3 to v3.0.4
2026-10-25 12:24:30 +01:00
Ruben Hensen 0b38d8dec8 Merge pull request #33 from rubenhensen/requests
Set resource requests from measured usage
2026-10-25 12:23:54 +01:00
Ruben Hensen 083353fb9d Merge pull request #23 from rubenhensen/renovate/binwiederhier-ntfy-2.x
chore(deps): update binwiederhier/ntfy docker tag to v2.34.0
2026-10-25 12:22:45 +01:00
Ruben Hensen 23f9c5c60a docs: record the merged Renovate PRs 2026-10-25 12:21:38 +01:00
Ruben Hensen a8bf3d5b64 Merge pull request #31 from rubenhensen/renovate/python-3.x
chore(deps): update python docker tag to v3.15.1
2026-10-25 12:17:31 +01:00
Ruben Hensen 618954ae2c Merge pull request #25 from rubenhensen/renovate/wordpress-0.x
chore(deps): update helm chart wordpress to v0.16.7
2026-10-25 12:17:17 +01:00
Ruben Hensen 44dc85d306 Merge pull request #24 from rubenhensen/renovate/https-github.com-kubernetes-csi-external-snapshotter.git-8.x
chore(deps): update dependency https://github.com/kubernetes-csi/external-snapshotter.git to v8.7.0
2026-10-25 12:16:25 +01:00
Ruben Hensen b60735f6b0 Merge pull request #28 from rubenhensen/renovate/cert-manager-1.21.x
chore(deps): update helm chart cert-manager to v1.21.3
2026-10-25 12:14:55 +01:00
Ruben Hensen 3a57e5be48 docs: WordPress went out of memory under a crawler 2026-10-25 09:45:33 +01:00
Ruben Hensen 1611f5d1ef docs: PR 33's window and Renovate's PR limit 2026-10-25 09:44:20 +01:00
Ruben Hensen 5bb1422594 Raise two requests after the morning's numbers 2026-10-25 09:43:11 +01:00
Ruben Hensen 68b692a112 docs: requests are in PR 33 2026-10-25 07:10:37 +01:00
Ruben Hensen 72f49bed51 Set resource requests from measured usage 2026-10-25 07:08:16 +01:00
Ruben Hensen f4b42757cd backup-testing: remove k3s's anonymous volumes too 2026-10-25 07:04:17 +01:00
Ruben Hensen f529bd99ee docs: triage PR 31 2026-10-25 07:02:07 +01:00
Ruben Hensen 74dec430b8 docs: the restore test is ready to run 2026-10-25 07:01:04 +01:00
Ruben Hensen 6fe0acc061 backup-testing: a restore test from a read-only copy of the bucket 2026-10-25 07:00:49 +01:00
Ruben Hensen bb34b601d9 docs: the system backups' snapshots hold 258 GiB 2026-10-25 04:35:58 +01:00
Ruben Hensen e98668d5fa docs: the cluster check has six parts 2026-10-25 04:35:02 +01:00
Ruben Hensen 389f911823 gatus: also report unhealthy Argo CD apps and CNPG clusters 2026-10-25 04:34:32 +01:00
Ruben Hensen 8f1c6a5b6e docs: record the repo rules 2026-10-25 04:33:01 +01:00
Ruben Hensen 770631fdc0 ci: check the repo's own manifests for secrets in git and unpinned images 2026-10-25 04:32:13 +01:00
Ruben Hensen 855317b84d freshrss: pin the alpine/git init container 2026-10-25 04:32:13 +01:00
Ruben Hensen d69f2638d1 docs: record the vulnerability scan and the old Longhorn instance manager 2026-10-25 04:29:57 +01:00
Ruben Hensen 90e552c9f5 docs: update the roadmap's state 2026-10-25 04:25:43 +01:00
Ruben Hensen ad14995ba0 docs: record the wider Renovate coverage 2026-10-25 04:25:10 +01:00
Ruben Hensen c359384ade renovate: look at every app folder, and at CNPG images 2026-10-25 04:24:56 +01:00
Ruben Hensen a4fc386ee8 docs: record the new probes 2026-10-25 04:22:28 +01:00
Ruben Hensen 0dc0bc4322 docs: authentik's base URL is already set 2026-10-25 04:22:06 +01:00
Ruben Hensen d843f31d86 serpbear: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen da23357946 freshrss: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen b9d88f41b9 gitea: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen 3c4a8cc8db docs: bring the authentik README up to date 2026-10-25 04:16:48 +01:00
Ruben Hensen d17e78130d docs: record metrics-server and plan the restore test 2026-10-25 04:15:43 +01:00
Ruben Hensen 22c5a325d6 Add metrics-server 2026-10-25 04:11:59 +01:00
Ruben Hensen ef07307f93 renovate: keep CI's Helm at Argo CD's version 2026-10-25 04:10:30 +01:00
Ruben Hensen aa940134ea docs: record the CI and the Renovate triage 2026-10-25 04:10:14 +01:00
Ruben Hensen f86ed09cd8 renovate: drop the github-tags manager, which duplicated the argocd one 2026-10-25 04:09:52 +01:00
Ruben Hensen 6d2bc379ba ci: retry when a registry rate-limits 2026-10-25 04:06:53 +01:00
Ruben Hensen 67c21c30d8 Render and validate every app on PRs and pushes 2026-10-25 03:57:24 +01:00
Ruben Hensen 0c52d8a467 apps: list external-snapshotter's automated sync once 2026-10-25 03:57:24 +01:00
Ruben Hensen f2e4c5aff6 docs: record the backup check 2026-10-25 03:49:43 +01:00
Ruben Hensen d836e4608c gatus: check backups and Vault's seal from inside the cluster 2026-10-25 03:48:05 +01:00
Ruben Hensen 1f56e67e8b docs: what the disk did in the last day, and foto's DNS 2026-10-25 03:42:13 +01:00
Ruben Hensen cadac4cebf docs: record the cleanup and what is left of it 2026-10-25 03:29:55 +01:00
Ruben Hensen bc71061db7 Remove commented-out files, the empty secrets app and a stale script copy 2026-10-25 03:27:32 +01:00
Ruben Hensen 7d3a64e5c8 docs: record the NetworkPolicies for the other apps and Vault 2026-10-25 03:24:07 +01:00
Ruben Hensen e137e79b75 vault: only ESO, Envoy and the node reach Vault 2026-10-25 03:21:53 +01:00
Ruben Hensen 67dc4f0783 Limit the other public apps to their own pods, DNS and the internet 2026-10-25 03:21:53 +01:00
Ruben Hensen 684f536f2f docs: record the NetworkPolicies for immich, mbgwp and authentik 2026-10-25 03:17:11 +01:00
Ruben Hensen 1439466af4 authentik: limit the pods to each other, DNS and the internet 2026-10-25 03:15:20 +01:00
Ruben Hensen 57514c3c14 immich: limit the pods to each other, DNS and the internet 2026-10-25 03:12:02 +01:00
Ruben Hensen 57851bf43a mbgwp: limit WordPress to its database, DNS and the internet 2026-10-25 03:09:06 +01:00
Ruben Hensen ada909ee02 docs: plan off-site backups through a friend's bucket 2026-10-25 03:05:39 +01:00
Ruben Hensen 345de16552 docs: record the Vault 2.2.1 upgrade 2026-10-25 02:52:52 +01:00
Ruben Hensen 7396cc3ed6 vault: fix the listing-visibility flag in the OIDC job 2026-10-25 02:52:02 +01:00
Ruben Hensen 26ca83457c vault: upgrade to 2.2.1, with a script to roll the pods 2026-10-25 02:40:45 +01:00
Ruben Hensen 4709194483 docs: tick off the authentik login test 2026-10-25 02:40:45 +01:00
Ruben Hensen ad2d6fbae5 docs: record the authentik 2026.8 upgrade 2026-10-25 02:34:51 +01:00
Ruben Hensen 077711472e authentik: upgrade to 2026.8.5 2026-10-25 02:31:00 +01:00
Ruben Hensen 43f741141d authentik: upgrade to 2026.5.6 2026-10-25 02:23:15 +01:00
Ruben Hensen cbee4955d3 authentik: drop the redis values, which the chart no longer has 2026-10-25 02:23:15 +01:00
Ruben Hensen a70dff6fdd docs: add a roadmap in place of the wishlist 2026-10-25 02:16:31 +01:00
Ruben Hensen ab0a09a356 docs: the immich login moved to auth.scoutingmbg.nl 2026-10-25 02:08:59 +01:00
Ruben Hensen b2649a682c immich: log in through auth.scoutingmbg.nl
The Scouts Online connection now redirects to auth.scoutingmbg.nl, the
group's own domain for the authentik login, so immich's OAuth issuer
moves there too.
2026-10-25 02:03:50 +01:00
Ruben Hensen ce3846bcd3 node001: tunnel scoutingmbg.nl and www to the home cluster
The group's DNS moved scoutingmbg.nl (A to node001) and www (CNAME to
rubenhensen.nl) here as well, so the WordPress site got node001's mail
certificate. Pass them through to the home cluster like foto and auth.
2026-10-25 02:59:00 +02:00
Ruben Hensen 96c13c63f7 authentik: serve auth.scoutingmbg.nl
A Gateway listener with its own certificate, the authentik route on both
hostnames, and an authentik brand for auth.scoutingmbg.nl that logs in
through Scouts Online and then sends people to the photos. The
authentik.rubenhensen.nl brand stays as it is.
2026-10-24 18:52:24 +02:00
Ruben Hensen 70151b0ef4 authentik: certificate for auth.scoutingmbg.nl
The login of the Scouting MBG photos moves to the group's own domain; the Gateway listener follows once the certificate is there.
2026-10-24 18:51:27 +02:00
Ruben Hensen f5e8b1e633 node001: tunnel auth.scoutingmbg.nl to the home cluster
The group's DNS points auth.scoutingmbg.nl at rubenhensen.nl too; it will carry the authentik login for immich.
2026-10-24 17:58:34 +02:00
Ruben Hensen ae2e5ad9fa docs: record the foto.scoutingmbg.nl DNS change 2026-10-24 17:48:35 +02:00
Ruben Hensen 8cf87483d6 node001: tunnel foto.scoutingmbg.nl to the home cluster
The group's DNS now points foto.scoutingmbg.nl at rubenhensen.nl, which is
this server, instead of mbg.hensen.io (the home IP). Resolvers that had
picked up the change got node001's own certificate and no immich. Add the
host to the SNI passthrough and the port 80 proxy like the other tunneled
hosts, so it works through either record.
2026-10-24 17:47:14 +02:00
Ruben Hensen c5d2ebaefc docs: record the Envoy retry policy 2026-10-24 17:06:29 +02:00
Ruben Hensen 5f8394e0c0 docs: plan a second SSD for Longhorn
Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.
2026-10-24 15:17:15 +02:00
Ruben Hensen 396eb2665c docs: record why controllers still lost their leases
Heavy writes on the node's only SSD stall dqlite; the fixes so far, the running measurement and the second-SSD option.
2026-10-24 14:46:09 +02:00
Ruben Hensen 858904384c Give leader-elected controllers longer leases
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
2026-10-24 14:43:38 +02:00
Ruben Hensen 9014898fc5 velero: run kopia maintenance daily and throttled
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
2026-10-24 14:43:38 +02:00
Ruben Hensen 111a9ddcea docs: foto@ is deleted 2026-10-24 14:20:07 +02:00
Ruben Hensen a8296a514e docs: foto@'s likes are deleted 2026-10-24 14:19:00 +02:00
Ruben Hensen 712fa1fe65 docs: sol-auth is gone 2026-10-24 14:15:14 +02:00