Commit Graph
100 Commits
Author SHA1 Message Date
Ruben Hensen 44dc85d306 Merge pull request #24 from rubenhensen/renovate/https-github.com-kubernetes-csi-external-snapshotter.git-8.x
chore(deps): update dependency https://github.com/kubernetes-csi/external-snapshotter.git to v8.7.0
2026-10-25 12:16:25 +01:00
Ruben Hensen b60735f6b0 Merge pull request #28 from rubenhensen/renovate/cert-manager-1.21.x
chore(deps): update helm chart cert-manager to v1.21.3
2026-10-25 12:14:55 +01:00
Ruben Hensen 3a57e5be48 docs: WordPress went out of memory under a crawler 2026-10-25 09:45:33 +01:00
Ruben Hensen 1611f5d1ef docs: PR 33's window and Renovate's PR limit 2026-10-25 09:44:20 +01:00
Ruben Hensen 68b692a112 docs: requests are in PR 33 2026-10-25 07:10:37 +01:00
Ruben Hensen f4b42757cd backup-testing: remove k3s's anonymous volumes too 2026-10-25 07:04:17 +01:00
Ruben Hensen f529bd99ee docs: triage PR 31 2026-10-25 07:02:07 +01:00
Ruben Hensen 74dec430b8 docs: the restore test is ready to run 2026-10-25 07:01:04 +01:00
Ruben Hensen 6fe0acc061 backup-testing: a restore test from a read-only copy of the bucket 2026-10-25 07:00:49 +01:00
Ruben Hensen bb34b601d9 docs: the system backups' snapshots hold 258 GiB 2026-10-25 04:35:58 +01:00
Ruben Hensen e98668d5fa docs: the cluster check has six parts 2026-10-25 04:35:02 +01:00
Ruben Hensen 389f911823 gatus: also report unhealthy Argo CD apps and CNPG clusters 2026-10-25 04:34:32 +01:00
Ruben Hensen 8f1c6a5b6e docs: record the repo rules 2026-10-25 04:33:01 +01:00
Ruben Hensen 770631fdc0 ci: check the repo's own manifests for secrets in git and unpinned images 2026-10-25 04:32:13 +01:00
Ruben Hensen 855317b84d freshrss: pin the alpine/git init container 2026-10-25 04:32:13 +01:00
Ruben Hensen d69f2638d1 docs: record the vulnerability scan and the old Longhorn instance manager 2026-10-25 04:29:57 +01:00
Ruben Hensen 90e552c9f5 docs: update the roadmap's state 2026-10-25 04:25:43 +01:00
Ruben Hensen ad14995ba0 docs: record the wider Renovate coverage 2026-10-25 04:25:10 +01:00
Ruben Hensen c359384ade renovate: look at every app folder, and at CNPG images 2026-10-25 04:24:56 +01:00
Ruben Hensen a4fc386ee8 docs: record the new probes 2026-10-25 04:22:28 +01:00
Ruben Hensen 0dc0bc4322 docs: authentik's base URL is already set 2026-10-25 04:22:06 +01:00
Ruben Hensen d843f31d86 serpbear: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen da23357946 freshrss: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen b9d88f41b9 gitea: add health probes 2026-10-25 04:19:29 +01:00
Ruben Hensen 3c4a8cc8db docs: bring the authentik README up to date 2026-10-25 04:16:48 +01:00
Ruben Hensen d17e78130d docs: record metrics-server and plan the restore test 2026-10-25 04:15:43 +01:00
Ruben Hensen 22c5a325d6 Add metrics-server 2026-10-25 04:11:59 +01:00
Ruben Hensen ef07307f93 renovate: keep CI's Helm at Argo CD's version 2026-10-25 04:10:30 +01:00
Ruben Hensen aa940134ea docs: record the CI and the Renovate triage 2026-10-25 04:10:14 +01:00
Ruben Hensen f86ed09cd8 renovate: drop the github-tags manager, which duplicated the argocd one 2026-10-25 04:09:52 +01:00
Ruben Hensen 6d2bc379ba ci: retry when a registry rate-limits 2026-10-25 04:06:53 +01:00
Ruben Hensen 67c21c30d8 Render and validate every app on PRs and pushes 2026-10-25 03:57:24 +01:00
Ruben Hensen 0c52d8a467 apps: list external-snapshotter's automated sync once 2026-10-25 03:57:24 +01:00
Ruben Hensen f2e4c5aff6 docs: record the backup check 2026-10-25 03:49:43 +01:00
Ruben Hensen d836e4608c gatus: check backups and Vault's seal from inside the cluster 2026-10-25 03:48:05 +01:00
Ruben Hensen 1f56e67e8b docs: what the disk did in the last day, and foto's DNS 2026-10-25 03:42:13 +01:00
Ruben Hensen cadac4cebf docs: record the cleanup and what is left of it 2026-10-25 03:29:55 +01:00
Ruben Hensen bc71061db7 Remove commented-out files, the empty secrets app and a stale script copy 2026-10-25 03:27:32 +01:00
Ruben Hensen 7d3a64e5c8 docs: record the NetworkPolicies for the other apps and Vault 2026-10-25 03:24:07 +01:00
Ruben Hensen e137e79b75 vault: only ESO, Envoy and the node reach Vault 2026-10-25 03:21:53 +01:00
Ruben Hensen 67dc4f0783 Limit the other public apps to their own pods, DNS and the internet 2026-10-25 03:21:53 +01:00
Ruben Hensen 684f536f2f docs: record the NetworkPolicies for immich, mbgwp and authentik 2026-10-25 03:17:11 +01:00
Ruben Hensen 1439466af4 authentik: limit the pods to each other, DNS and the internet 2026-10-25 03:15:20 +01:00
Ruben Hensen 57514c3c14 immich: limit the pods to each other, DNS and the internet 2026-10-25 03:12:02 +01:00
Ruben Hensen 57851bf43a mbgwp: limit WordPress to its database, DNS and the internet 2026-10-25 03:09:06 +01:00
Ruben Hensen ada909ee02 docs: plan off-site backups through a friend's bucket 2026-10-25 03:05:39 +01:00
Ruben Hensen 345de16552 docs: record the Vault 2.2.1 upgrade 2026-10-25 02:52:52 +01:00
Ruben Hensen 7396cc3ed6 vault: fix the listing-visibility flag in the OIDC job 2026-10-25 02:52:02 +01:00
Ruben Hensen 26ca83457c vault: upgrade to 2.2.1, with a script to roll the pods 2026-10-25 02:40:45 +01:00
Ruben Hensen 4709194483 docs: tick off the authentik login test 2026-10-25 02:40:45 +01:00
Ruben Hensen ad2d6fbae5 docs: record the authentik 2026.8 upgrade 2026-10-25 02:34:51 +01:00
Ruben Hensen 077711472e authentik: upgrade to 2026.8.5 2026-10-25 02:31:00 +01:00
Ruben Hensen 43f741141d authentik: upgrade to 2026.5.6 2026-10-25 02:23:15 +01:00
Ruben Hensen cbee4955d3 authentik: drop the redis values, which the chart no longer has 2026-10-25 02:23:15 +01:00
Ruben Hensen a70dff6fdd docs: add a roadmap in place of the wishlist 2026-10-25 02:16:31 +01:00
Ruben Hensen ab0a09a356 docs: the immich login moved to auth.scoutingmbg.nl 2026-10-25 02:08:59 +01:00
Ruben Hensen b2649a682c immich: log in through auth.scoutingmbg.nl
The Scouts Online connection now redirects to auth.scoutingmbg.nl, the
group's own domain for the authentik login, so immich's OAuth issuer
moves there too.
2026-10-25 02:03:50 +01:00
Ruben Hensen ce3846bcd3 node001: tunnel scoutingmbg.nl and www to the home cluster
The group's DNS moved scoutingmbg.nl (A to node001) and www (CNAME to
rubenhensen.nl) here as well, so the WordPress site got node001's mail
certificate. Pass them through to the home cluster like foto and auth.
2026-10-25 02:59:00 +02:00
Ruben Hensen 96c13c63f7 authentik: serve auth.scoutingmbg.nl
A Gateway listener with its own certificate, the authentik route on both
hostnames, and an authentik brand for auth.scoutingmbg.nl that logs in
through Scouts Online and then sends people to the photos. The
authentik.rubenhensen.nl brand stays as it is.
2026-10-24 18:52:24 +02:00
Ruben Hensen 70151b0ef4 authentik: certificate for auth.scoutingmbg.nl
The login of the Scouting MBG photos moves to the group's own domain; the Gateway listener follows once the certificate is there.
2026-10-24 18:51:27 +02:00
Ruben Hensen f5e8b1e633 node001: tunnel auth.scoutingmbg.nl to the home cluster
The group's DNS points auth.scoutingmbg.nl at rubenhensen.nl too; it will carry the authentik login for immich.
2026-10-24 17:58:34 +02:00
Ruben Hensen ae2e5ad9fa docs: record the foto.scoutingmbg.nl DNS change 2026-10-24 17:48:35 +02:00
Ruben Hensen 8cf87483d6 node001: tunnel foto.scoutingmbg.nl to the home cluster
The group's DNS now points foto.scoutingmbg.nl at rubenhensen.nl, which is
this server, instead of mbg.hensen.io (the home IP). Resolvers that had
picked up the change got node001's own certificate and no immich. Add the
host to the SNI passthrough and the port 80 proxy like the other tunneled
hosts, so it works through either record.
2026-10-24 17:47:14 +02:00
Ruben Hensen c5d2ebaefc docs: record the Envoy retry policy 2026-10-24 17:06:29 +02:00
Ruben Hensen 5f8394e0c0 docs: plan a second SSD for Longhorn
Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.
2026-10-24 15:17:15 +02:00
Ruben Hensen 396eb2665c docs: record why controllers still lost their leases
Heavy writes on the node's only SSD stall dqlite; the fixes so far, the running measurement and the second-SSD option.
2026-10-24 14:46:09 +02:00
Ruben Hensen 858904384c Give leader-elected controllers longer leases
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
2026-10-24 14:43:38 +02:00
Ruben Hensen 9014898fc5 velero: run kopia maintenance daily and throttled
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
2026-10-24 14:43:38 +02:00
Ruben Hensen 111a9ddcea docs: foto@ is deleted 2026-10-24 14:20:07 +02:00
Ruben Hensen a8296a514e docs: foto@'s likes are deleted 2026-10-24 14:19:00 +02:00
Ruben Hensen 712fa1fe65 docs: sol-auth is gone 2026-10-24 14:15:14 +02:00
Ruben Hensen c3505bc8e3 immich: remove sol-auth
foto.scoutingmbg.nl goes straight to immich since the switch to Scouts
Online, so the Apache with mod_auth_openid (OpenID 2) in front of it has
no traffic left.
2026-10-24 14:14:43 +02:00
Ruben Hensen 6c6f473f63 docs: fotoupload@ sessions ended 2026-10-24 14:13:40 +02:00
Ruben Hensen a5478d60e6 docs: share-sync joins albums that kaderleden create 2026-10-24 14:11:07 +02:00
Ruben Hensen 88f68cd698 immich: share albums that kaderleden create automatically
Nobody uses the group account anymore, so new albums belong to the
leiding who make them, and share-sync only shared albums the group
account was in. It now first adds the group account as editor to every
album an uploader owns (straight in the database: the API only lets an
album's owner or editors share it), after which the album goes to
everyone like the others. Albums whose name starts with "Privé" or
"Prive" are left alone.
2026-10-24 14:10:30 +02:00
Ruben Hensen f69b790a58 docs: immich now logs in with Scouts Online only 2026-10-24 14:04:43 +02:00
Ruben Hensen 71af3a9342 immich: fix hiding album invitations in share-sync
immich stores a notification's data as a JSON string inside the jsonb column, so data->>'albumId' was always null and no invitation was hidden. Unwrap it first.
2026-10-24 14:03:02 +02:00
Ruben Hensen 348cce6d5d gatus: check immich's ping endpoint for foto.scoutingmbg.nl
Without sol-auth in front, /_app is no longer the open path to check;
/api/server/ping answers without a login.
2026-10-24 14:01:23 +02:00
Ruben Hensen abed11dbaa immich: log in with Scouts Online only
foto.scoutingmbg.nl now goes straight to immich instead of through
sol-auth. immich starts the Scouts Online login right away, the password
login is off and the login page no longer shows the password of the
shared viewing account. share-sync is on, so everyone gets every album
and the group's library in their timeline.

sol-auth stays deployed for now, so switching the route back is the way
back.
2026-10-24 14:01:23 +02:00
Ruben Hensen 86a0c59c6b immich: add share-sync, suspended
immich has no user groups, so a CronJob shares the group account's
albums with every user, as editor when they may upload (quota other than
0, which authentik sets from Scouts Online) and as viewer otherwise. It
also shares the group account's library as a partner and shows it in
every timeline, since immich only searches your own and your timeline
partners' photos; hides the album invitations for albums older than a
week; and ends OAuth sessions older than 30 days, so people who left the
group lose access at their next login.

Suspended until immich logs in through Scouts Online only.
2026-10-24 13:48:07 +02:00
Ruben Hensen ebf28c1bf1 docs: step B of the Scouts Online login is live 2026-10-24 13:43:24 +02:00
Ruben Hensen 135bca4a83 authentik: log in to immich with Scouts Online
Adds Scouts Online 3 (login.scouting.nl) as an OIDC source. Its userinfo
has the member's roles within the group; a property mapping puts every
member in "MBG leden" and those with an active Kaderlid role also in
"MBG kader", on every login. Members become external users. Family
members can share an email address and immich needs a unique one, so
those get a plus address with their member number.

The immich provider now authenticates through a flow whose only stage
is an identification stage with this one source and no fields, which
sends people straight on to Scouts Online. The source's own flows only
run through the source.
2026-10-24 13:37:12 +02:00
Ruben Hensen 1c6b42b20b docs: step A of the Scouts Online login is live 2026-10-24 13:25:02 +02:00
Ruben Hensen f3f36c3718 docs: plan the immich login with Scouts Online
Building blocks, design and the three steps towards personal accounts with upload rights from Scouts Online roles.
2026-10-24 13:19:52 +02:00
Ruben Hensen 399387bfb7 immich: configure through a config file with OAuth
immich gets its settings from a Secret that ESO renders with the OAuth
client secret from Vault. It keeps the settings that were in the database
(external domain, login page message, storage template) and adds OAuth
through authentik next to the password login, with auto register. With a
config file the settings are read-only in the admin UI.
2026-10-24 13:19:52 +02:00
Ruben Hensen 48b0873ca3 authentik: add an OIDC provider for immich
The Scouting MBG photos (immich) will log in through this authentik,
which later passes on Scouts Online. The provider issues immich_role and
immich_quota: admins and the members of "MBG kader" may upload (-1),
everyone else gets a quota of 0, which makes immich refuse uploads.
"MBG leden" and "MBG kader" are filled by the Scouts Online source in a
next step; until then only authentik Admins can use the app.

The scouting blueprints and their secrets are mounted as directories
rather than with subPath, and the blueprint reads the client secret with
!File instead of !Env, so later changes reach authentik without a restart.
2026-10-24 13:19:52 +02:00
Ruben Hensen 278f6fc55f authentik: limit personal apps to authentik Admins
Actual Budget, Argo CD, FreshRSS, Stalwart and Vault had no policy
binding, so any authentik user could sign in to them. Scouts Online
members are about to get accounts in this authentik for immich; bind
each app to the authentik Admins group, as Longhorn already was.
2026-10-24 13:04:33 +02:00
Ruben Hensen f4e74a9503 docs: record the immich 3 upgrade
v3.3.5 with chart 0.13.4 is running; note the safety net to clean up later and the optional revised ML models.
2026-10-24 12:33:43 +02:00
Ruben Hensen c270bcc12b immich: upgrade to v3.3.5 and chart 0.13.4
immich 3 drops pgvecto.rs, which the database no longer uses since the
move to PostgreSQL 17 with VectorChord 1.1.1 (supported range >=0.3 <2)
and pgvector 0.8.6 (>=0.7 <0.9). None of the removed environment
variables are set here. The mobile app cannot pass sol-auth, so there are
no app clients to upgrade first.

Charts after 0.12.0 are only published to the OCI registry. Since 0.13.0
the chart validates its values, and emptyDir volumes no longer accept
size, accessMode or storageClass; they did nothing anyway. Valkey follows
the chart's new default, 9.1.2. Each component now gets its own
ServiceAccount without a mounted token.

Safety net: Longhorn snapshot immich-pg17-pre-v3 and a pg_dump in
~/immich-pg17-pre-v3.dump on opti1, since immich cannot be downgraded.
2026-10-24 12:29:35 +02:00
Ruben Hensen 14e61ee5ee gateway: retry requests on dropped backend connections
When a backend closes a kept-alive connection just as Envoy sends a
request over it, Envoy returns a 503 (response flag UC) and does not
retry by default. ingress-nginx retried these, so the race never showed.

The Apache in front of immich (sol-auth) recycles its processes after 30
connections, which made this happen during page loads: three chunks got
a 503 and immich failed with "error loading dynamically imported module".
authentik and ntfy saw it too, less often.

Retry up to twice on connection failures and resets, for every route on
the Gateway.
2026-10-24 12:19:37 +02:00
Ruben Hensen e7696107c0 docs: record the immich database migration
immich now runs v2.7.5 on PostgreSQL 17 with VectorChord. The PostgreSQL 14 cluster stays for a few days as a fallback.
2026-10-24 02:19:57 +02:00
Ruben Hensen a8a186b4d3 immich: upgrade to v2.7.5
v2.4.1 only accepts VectorChord >=0.3 <0.6 and refused to start on the new PostgreSQL 17 cluster with VectorChord 1.1.1; from v2.5.0 the range is >=0.3 <2. v2.7.5 is the last 2.x release; the 3.x upgrade comes separately.
2026-10-24 02:12:53 +02:00
Ruben Hensen 8596e8a428 immich: use the PostgreSQL 17 cluster and start the server again
The import into immich-postgres-17 matched the old database table for table (assets, embeddings, faces, people, albums, migrations). On start immich builds its VectorChord indexes.
2026-10-24 02:06:41 +02:00
Ruben Hensen 251f22c46f immich: add a PostgreSQL 17 cluster with VectorChord
Imported once from the PostgreSQL 14 cluster, whose embeddings were converted to real[] and whose pgvecto.rs extension was dropped first (immich's manual migration). After the import the columns become vector(512) again under VectorChord 1.1.1 with pgvector 0.8.6. A logical import because CNPG can't upgrade in place from the bullseye-based pgvecto.rs image to the bookworm-based VectorChord one.
2026-10-24 02:02:02 +02:00
Ruben Hensen 60ff978b1b immich: stop the server for the database migration
The database moves from PostgreSQL 14 with pgvecto.rs to a new PostgreSQL 17 cluster with VectorChord. Immich 3 no longer supports pgvecto.rs, and PostgreSQL 14 reaches end of life on 2026-11-12.
2026-10-24 00:18:28 +02:00
Ruben Hensen 6517c21937 mbgwp: remove the unused wordpress-external NodePort
It selected pods of the old mbgwp-helm release, so since the move to the official chart it had no endpoints and only held node port 30989 open.
2026-10-24 00:08:19 +02:00
Ruben Hensen 8357d31f9c docs: record the Longhorn NetworkPolicies and poll interval 2026-10-24 00:05:44 +02:00
Ruben Hensen 5208379310 longhorn: enable the internal NetworkPolicies
With them only Longhorn's own components can reach the longhorn-manager API, which any pod could otherwise call without authentication to manage volumes. 1.12 was kept off because its instance-manager policy left the host-side iSCSI initiator out; 1.13 opens port 3260 to any source while v1DataEngineInitiatorSourceCIDRs is empty.
2026-10-24 00:02:32 +02:00
Ruben Hensen f2936e69ec docs: record the switch to annotation tracking 2026-10-24 00:01:16 +02:00
Ruben Hensen 7c8cd28f71 argocd: track resources by annotation
Drops the label tracking pin kept through the 3.x upgrades, so Argo CD uses its default argocd.argoproj.io/tracking-id annotation. Labels can be copied by other tools (cert-manager's ingress-shim copied app.kubernetes.io/instance onto Certificates); the annotation names the exact resource. Every app gets synced once right after, so resources removed later are still recognised for pruning.
2026-10-23 23:54:06 +02:00