Ruben Hensen
7dc13466cb
mbgwp: cap Apache at 12 workers so a crawler can't run it out of memory
2026-10-25 13:07:30 +01:00
Ruben Hensen
9b77596bcb
backup-testing: take endpoint, region and bucket from Velero's location
2026-10-25 12:59:25 +01:00
Ruben Hensen
fbf2cb2d92
docs: the unseal key is kept safe
2026-10-25 12:57:18 +01:00
Ruben Hensen
122873ca5c
docs: hensen.io can expire
2026-10-25 12:54:32 +01:00
Ruben Hensen
71449dd185
docs: record the external-secrets upgrade
2026-10-25 12:54:01 +01:00
Ruben Hensen
297d2dffa5
Merge pull request #17 from rubenhensen/renovate/external-secrets-2.x
...
chore(deps): update helm chart external-secrets to v2.14.0
2026-10-25 12:50:28 +01:00
Ruben Hensen
46117ee658
chore(deps): update helm chart external-secrets to v2.14.0
2026-10-25 12:49:06 +01:00
Ruben Hensen
2a5bbc7f44
docs: the scoutingmbg.nl DNS questions are answered
...
Only the four records moved to node001, hensen.io is no longer needed, and SPF's a mechanism doesn't matter since all mail goes through Google.
2026-10-25 12:39:57 +01:00
Ruben Hensen
d7b0372f1a
docs: record the rest of the merged PRs
2026-10-25 12:37:25 +01:00
Ruben Hensen
0ad6ca816c
docs: who cleaned up the snapshots, and the stalls per minute
...
Every minute with dqlite errors in the 23 hours sampled had the SSD fully busy; the controllers with 60s leases held.
2026-10-25 12:35:47 +01:00
Ruben Hensen
3e252799a4
Merge pull request #26 from rubenhensen/renovate/gitea-gitea-28.x
...
chore(deps): update gitea/gitea docker tag to v28
2026-10-25 12:25:03 +01:00
Ruben Hensen
bfd5619e9e
Merge pull request #27 from rubenhensen/renovate/github.com-canonical-go-dqlite-v3-3.x
...
fix(deps): update module github.com/canonical/go-dqlite/v3 to v3.0.4
2026-10-25 12:24:30 +01:00
Ruben Hensen
0b38d8dec8
Merge pull request #33 from rubenhensen/requests
...
Set resource requests from measured usage
2026-10-25 12:23:54 +01:00
Ruben Hensen
083353fb9d
Merge pull request #23 from rubenhensen/renovate/binwiederhier-ntfy-2.x
...
chore(deps): update binwiederhier/ntfy docker tag to v2.34.0
2026-10-25 12:22:45 +01:00
Ruben Hensen
23f9c5c60a
docs: record the merged Renovate PRs
2026-10-25 12:21:38 +01:00
Ruben Hensen
a8bf3d5b64
Merge pull request #31 from rubenhensen/renovate/python-3.x
...
chore(deps): update python docker tag to v3.15.1
2026-10-25 12:17:31 +01:00
Ruben Hensen
618954ae2c
Merge pull request #25 from rubenhensen/renovate/wordpress-0.x
...
chore(deps): update helm chart wordpress to v0.16.7
2026-10-25 12:17:17 +01:00
Ruben Hensen
44dc85d306
Merge pull request #24 from rubenhensen/renovate/https-github.com-kubernetes-csi-external-snapshotter.git-8.x
...
chore(deps): update dependency https://github.com/kubernetes-csi/external-snapshotter.git to v8.7.0
2026-10-25 12:16:25 +01:00
Ruben Hensen
b60735f6b0
Merge pull request #28 from rubenhensen/renovate/cert-manager-1.21.x
...
chore(deps): update helm chart cert-manager to v1.21.3
2026-10-25 12:14:55 +01:00
Ruben Hensen
3a57e5be48
docs: WordPress went out of memory under a crawler
2026-10-25 09:45:33 +01:00
Ruben Hensen
1611f5d1ef
docs: PR 33's window and Renovate's PR limit
2026-10-25 09:44:20 +01:00
Ruben Hensen
5bb1422594
Raise two requests after the morning's numbers
2026-10-25 09:43:11 +01:00
Ruben Hensen
68b692a112
docs: requests are in PR 33
2026-10-25 07:10:37 +01:00
Ruben Hensen
72f49bed51
Set resource requests from measured usage
2026-10-25 07:08:16 +01:00
Ruben Hensen
f4b42757cd
backup-testing: remove k3s's anonymous volumes too
2026-10-25 07:04:17 +01:00
Ruben Hensen
f529bd99ee
docs: triage PR 31
2026-10-25 07:02:07 +01:00
Ruben Hensen
74dec430b8
docs: the restore test is ready to run
2026-10-25 07:01:04 +01:00
Ruben Hensen
6fe0acc061
backup-testing: a restore test from a read-only copy of the bucket
2026-10-25 07:00:49 +01:00
Ruben Hensen
bb34b601d9
docs: the system backups' snapshots hold 258 GiB
2026-10-25 04:35:58 +01:00
Ruben Hensen
e98668d5fa
docs: the cluster check has six parts
2026-10-25 04:35:02 +01:00
Ruben Hensen
389f911823
gatus: also report unhealthy Argo CD apps and CNPG clusters
2026-10-25 04:34:32 +01:00
Ruben Hensen
8f1c6a5b6e
docs: record the repo rules
2026-10-25 04:33:01 +01:00
Ruben Hensen
770631fdc0
ci: check the repo's own manifests for secrets in git and unpinned images
2026-10-25 04:32:13 +01:00
Ruben Hensen
855317b84d
freshrss: pin the alpine/git init container
2026-10-25 04:32:13 +01:00
Ruben Hensen
d69f2638d1
docs: record the vulnerability scan and the old Longhorn instance manager
2026-10-25 04:29:57 +01:00
Ruben Hensen
90e552c9f5
docs: update the roadmap's state
2026-10-25 04:25:43 +01:00
Ruben Hensen
ad14995ba0
docs: record the wider Renovate coverage
2026-10-25 04:25:10 +01:00
Ruben Hensen
c359384ade
renovate: look at every app folder, and at CNPG images
2026-10-25 04:24:56 +01:00
Ruben Hensen
a4fc386ee8
docs: record the new probes
2026-10-25 04:22:28 +01:00
Ruben Hensen
0dc0bc4322
docs: authentik's base URL is already set
2026-10-25 04:22:06 +01:00
Ruben Hensen
d843f31d86
serpbear: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
da23357946
freshrss: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
b9d88f41b9
gitea: add health probes
2026-10-25 04:19:29 +01:00
Ruben Hensen
3c4a8cc8db
docs: bring the authentik README up to date
2026-10-25 04:16:48 +01:00
Ruben Hensen
d17e78130d
docs: record metrics-server and plan the restore test
2026-10-25 04:15:43 +01:00
Ruben Hensen
22c5a325d6
Add metrics-server
2026-10-25 04:11:59 +01:00
Ruben Hensen
ef07307f93
renovate: keep CI's Helm at Argo CD's version
2026-10-25 04:10:30 +01:00
Ruben Hensen
aa940134ea
docs: record the CI and the Renovate triage
2026-10-25 04:10:14 +01:00
Ruben Hensen
f86ed09cd8
renovate: drop the github-tags manager, which duplicated the argocd one
2026-10-25 04:09:52 +01:00
Ruben Hensen
6d2bc379ba
ci: retry when a registry rate-limits
2026-10-25 04:06:53 +01:00
Ruben Hensen
67c21c30d8
Render and validate every app on PRs and pushes
2026-10-25 03:57:24 +01:00
Ruben Hensen
0c52d8a467
apps: list external-snapshotter's automated sync once
2026-10-25 03:57:24 +01:00
Ruben Hensen
f2e4c5aff6
docs: record the backup check
2026-10-25 03:49:43 +01:00
Ruben Hensen
d836e4608c
gatus: check backups and Vault's seal from inside the cluster
2026-10-25 03:48:05 +01:00
Ruben Hensen
1f56e67e8b
docs: what the disk did in the last day, and foto's DNS
2026-10-25 03:42:13 +01:00
Ruben Hensen
cadac4cebf
docs: record the cleanup and what is left of it
2026-10-25 03:29:55 +01:00
Ruben Hensen
bc71061db7
Remove commented-out files, the empty secrets app and a stale script copy
2026-10-25 03:27:32 +01:00
Ruben Hensen
7d3a64e5c8
docs: record the NetworkPolicies for the other apps and Vault
2026-10-25 03:24:07 +01:00
Ruben Hensen
e137e79b75
vault: only ESO, Envoy and the node reach Vault
2026-10-25 03:21:53 +01:00
Ruben Hensen
67dc4f0783
Limit the other public apps to their own pods, DNS and the internet
2026-10-25 03:21:53 +01:00
Ruben Hensen
684f536f2f
docs: record the NetworkPolicies for immich, mbgwp and authentik
2026-10-25 03:17:11 +01:00
Ruben Hensen
1439466af4
authentik: limit the pods to each other, DNS and the internet
2026-10-25 03:15:20 +01:00
Ruben Hensen
57514c3c14
immich: limit the pods to each other, DNS and the internet
2026-10-25 03:12:02 +01:00
Ruben Hensen
57851bf43a
mbgwp: limit WordPress to its database, DNS and the internet
2026-10-25 03:09:06 +01:00
Ruben Hensen
ada909ee02
docs: plan off-site backups through a friend's bucket
2026-10-25 03:05:39 +01:00
Ruben Hensen
345de16552
docs: record the Vault 2.2.1 upgrade
2026-10-25 02:52:52 +01:00
Ruben Hensen
7396cc3ed6
vault: fix the listing-visibility flag in the OIDC job
2026-10-25 02:52:02 +01:00
Ruben Hensen
26ca83457c
vault: upgrade to 2.2.1, with a script to roll the pods
2026-10-25 02:40:45 +01:00
Ruben Hensen
4709194483
docs: tick off the authentik login test
2026-10-25 02:40:45 +01:00
Ruben Hensen
ad2d6fbae5
docs: record the authentik 2026.8 upgrade
2026-10-25 02:34:51 +01:00
Ruben Hensen
077711472e
authentik: upgrade to 2026.8.5
2026-10-25 02:31:00 +01:00
Ruben Hensen
43f741141d
authentik: upgrade to 2026.5.6
2026-10-25 02:23:15 +01:00
Ruben Hensen
cbee4955d3
authentik: drop the redis values, which the chart no longer has
2026-10-25 02:23:15 +01:00
Ruben Hensen
a70dff6fdd
docs: add a roadmap in place of the wishlist
2026-10-25 02:16:31 +01:00
Ruben Hensen
ab0a09a356
docs: the immich login moved to auth.scoutingmbg.nl
2026-10-25 02:08:59 +01:00
Ruben Hensen
b2649a682c
immich: log in through auth.scoutingmbg.nl
...
The Scouts Online connection now redirects to auth.scoutingmbg.nl, the
group's own domain for the authentik login, so immich's OAuth issuer
moves there too.
2026-10-25 02:03:50 +01:00
Ruben Hensen
ce3846bcd3
node001: tunnel scoutingmbg.nl and www to the home cluster
...
The group's DNS moved scoutingmbg.nl (A to node001) and www (CNAME to
rubenhensen.nl) here as well, so the WordPress site got node001's mail
certificate. Pass them through to the home cluster like foto and auth.
2026-10-25 02:59:00 +02:00
Ruben Hensen
96c13c63f7
authentik: serve auth.scoutingmbg.nl
...
A Gateway listener with its own certificate, the authentik route on both
hostnames, and an authentik brand for auth.scoutingmbg.nl that logs in
through Scouts Online and then sends people to the photos. The
authentik.rubenhensen.nl brand stays as it is.
2026-10-24 18:52:24 +02:00
Ruben Hensen
70151b0ef4
authentik: certificate for auth.scoutingmbg.nl
...
The login of the Scouting MBG photos moves to the group's own domain; the Gateway listener follows once the certificate is there.
2026-10-24 18:51:27 +02:00
Ruben Hensen
f5e8b1e633
node001: tunnel auth.scoutingmbg.nl to the home cluster
...
The group's DNS points auth.scoutingmbg.nl at rubenhensen.nl too; it will carry the authentik login for immich.
2026-10-24 17:58:34 +02:00
Ruben Hensen
ae2e5ad9fa
docs: record the foto.scoutingmbg.nl DNS change
2026-10-24 17:48:35 +02:00
Ruben Hensen
8cf87483d6
node001: tunnel foto.scoutingmbg.nl to the home cluster
...
The group's DNS now points foto.scoutingmbg.nl at rubenhensen.nl, which is
this server, instead of mbg.hensen.io (the home IP). Resolvers that had
picked up the change got node001's own certificate and no immich. Add the
host to the SNI passthrough and the port 80 proxy like the other tunneled
hosts, so it works through either record.
2026-10-24 17:47:14 +02:00
Ruben Hensen
c5d2ebaefc
docs: record the Envoy retry policy
2026-10-24 17:06:29 +02:00
Ruben Hensen
5f8394e0c0
docs: plan a second SSD for Longhorn
...
Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.
2026-10-24 15:17:15 +02:00
Ruben Hensen
396eb2665c
docs: record why controllers still lost their leases
...
Heavy writes on the node's only SSD stall dqlite; the fixes so far, the running measurement and the second-SSD option.
2026-10-24 14:46:09 +02:00
Ruben Hensen
858904384c
Give leader-elected controllers longer leases
...
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
2026-10-24 14:43:38 +02:00
Ruben Hensen
9014898fc5
velero: run kopia maintenance daily and throttled
...
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
2026-10-24 14:43:38 +02:00
Ruben Hensen
111a9ddcea
docs: foto@ is deleted
2026-10-24 14:20:07 +02:00
Ruben Hensen
a8296a514e
docs: foto@'s likes are deleted
2026-10-24 14:19:00 +02:00
Ruben Hensen
712fa1fe65
docs: sol-auth is gone
2026-10-24 14:15:14 +02:00
Ruben Hensen
c3505bc8e3
immich: remove sol-auth
...
foto.scoutingmbg.nl goes straight to immich since the switch to Scouts
Online, so the Apache with mod_auth_openid (OpenID 2) in front of it has
no traffic left.
2026-10-24 14:14:43 +02:00
Ruben Hensen
6c6f473f63
docs: fotoupload@ sessions ended
2026-10-24 14:13:40 +02:00
Ruben Hensen
a5478d60e6
docs: share-sync joins albums that kaderleden create
2026-10-24 14:11:07 +02:00
Ruben Hensen
88f68cd698
immich: share albums that kaderleden create automatically
...
Nobody uses the group account anymore, so new albums belong to the
leiding who make them, and share-sync only shared albums the group
account was in. It now first adds the group account as editor to every
album an uploader owns (straight in the database: the API only lets an
album's owner or editors share it), after which the album goes to
everyone like the others. Albums whose name starts with "Privé" or
"Prive" are left alone.
2026-10-24 14:10:30 +02:00
Ruben Hensen
f69b790a58
docs: immich now logs in with Scouts Online only
2026-10-24 14:04:43 +02:00
Ruben Hensen
71af3a9342
immich: fix hiding album invitations in share-sync
...
immich stores a notification's data as a JSON string inside the jsonb column, so data->>'albumId' was always null and no invitation was hidden. Unwrap it first.
2026-10-24 14:03:02 +02:00
Ruben Hensen
348cce6d5d
gatus: check immich's ping endpoint for foto.scoutingmbg.nl
...
Without sol-auth in front, /_app is no longer the open path to check;
/api/server/ping answers without a login.
2026-10-24 14:01:23 +02:00
Ruben Hensen
abed11dbaa
immich: log in with Scouts Online only
...
foto.scoutingmbg.nl now goes straight to immich instead of through
sol-auth. immich starts the Scouts Online login right away, the password
login is off and the login page no longer shows the password of the
shared viewing account. share-sync is on, so everyone gets every album
and the group's library in their timeline.
sol-auth stays deployed for now, so switching the route back is the way
back.
2026-10-24 14:01:23 +02:00
Ruben Hensen
86a0c59c6b
immich: add share-sync, suspended
...
immich has no user groups, so a CronJob shares the group account's
albums with every user, as editor when they may upload (quota other than
0, which authentik sets from Scouts Online) and as viewer otherwise. It
also shares the group account's library as a partner and shows it in
every timeline, since immich only searches your own and your timeline
partners' photos; hides the album invitations for albums older than a
week; and ends OAuth sessions older than 30 days, so people who left the
group lose access at their next login.
Suspended until immich logs in through Scouts Online only.
2026-10-24 13:48:07 +02:00
Ruben Hensen
ebf28c1bf1
docs: step B of the Scouts Online login is live
2026-10-24 13:43:24 +02:00