The group's DNS moved scoutingmbg.nl (A to node001) and www (CNAME to
rubenhensen.nl) here as well, so the WordPress site got node001's mail
certificate. Pass them through to the home cluster like foto and auth.
A Gateway listener with its own certificate, the authentik route on both
hostnames, and an authentik brand for auth.scoutingmbg.nl that logs in
through Scouts Online and then sends people to the photos. The
authentik.rubenhensen.nl brand stays as it is.
The group's DNS now points foto.scoutingmbg.nl at rubenhensen.nl, which is
this server, instead of mbg.hensen.io (the home IP). Resolvers that had
picked up the change got node001's own certificate and no immich. Add the
host to the SNI passthrough and the port 80 proxy like the other tunneled
hosts, so it works through either record.
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
foto.scoutingmbg.nl goes straight to immich since the switch to Scouts
Online, so the Apache with mod_auth_openid (OpenID 2) in front of it has
no traffic left.
Nobody uses the group account anymore, so new albums belong to the
leiding who make them, and share-sync only shared albums the group
account was in. It now first adds the group account as editor to every
album an uploader owns (straight in the database: the API only lets an
album's owner or editors share it), after which the album goes to
everyone like the others. Albums whose name starts with "Privé" or
"Prive" are left alone.
immich stores a notification's data as a JSON string inside the jsonb column, so data->>'albumId' was always null and no invitation was hidden. Unwrap it first.
foto.scoutingmbg.nl now goes straight to immich instead of through
sol-auth. immich starts the Scouts Online login right away, the password
login is off and the login page no longer shows the password of the
shared viewing account. share-sync is on, so everyone gets every album
and the group's library in their timeline.
sol-auth stays deployed for now, so switching the route back is the way
back.
immich has no user groups, so a CronJob shares the group account's
albums with every user, as editor when they may upload (quota other than
0, which authentik sets from Scouts Online) and as viewer otherwise. It
also shares the group account's library as a partner and shows it in
every timeline, since immich only searches your own and your timeline
partners' photos; hides the album invitations for albums older than a
week; and ends OAuth sessions older than 30 days, so people who left the
group lose access at their next login.
Suspended until immich logs in through Scouts Online only.
Adds Scouts Online 3 (login.scouting.nl) as an OIDC source. Its userinfo
has the member's roles within the group; a property mapping puts every
member in "MBG leden" and those with an active Kaderlid role also in
"MBG kader", on every login. Members become external users. Family
members can share an email address and immich needs a unique one, so
those get a plus address with their member number.
The immich provider now authenticates through a flow whose only stage
is an identification stage with this one source and no fields, which
sends people straight on to Scouts Online. The source's own flows only
run through the source.
immich gets its settings from a Secret that ESO renders with the OAuth
client secret from Vault. It keeps the settings that were in the database
(external domain, login page message, storage template) and adds OAuth
through authentik next to the password login, with auto register. With a
config file the settings are read-only in the admin UI.
The Scouting MBG photos (immich) will log in through this authentik,
which later passes on Scouts Online. The provider issues immich_role and
immich_quota: admins and the members of "MBG kader" may upload (-1),
everyone else gets a quota of 0, which makes immich refuse uploads.
"MBG leden" and "MBG kader" are filled by the Scouts Online source in a
next step; until then only authentik Admins can use the app.
The scouting blueprints and their secrets are mounted as directories
rather than with subPath, and the blueprint reads the client secret with
!File instead of !Env, so later changes reach authentik without a restart.
Actual Budget, Argo CD, FreshRSS, Stalwart and Vault had no policy
binding, so any authentik user could sign in to them. Scouts Online
members are about to get accounts in this authentik for immich; bind
each app to the authentik Admins group, as Longhorn already was.
immich 3 drops pgvecto.rs, which the database no longer uses since the
move to PostgreSQL 17 with VectorChord 1.1.1 (supported range >=0.3 <2)
and pgvector 0.8.6 (>=0.7 <0.9). None of the removed environment
variables are set here. The mobile app cannot pass sol-auth, so there are
no app clients to upgrade first.
Charts after 0.12.0 are only published to the OCI registry. Since 0.13.0
the chart validates its values, and emptyDir volumes no longer accept
size, accessMode or storageClass; they did nothing anyway. Valkey follows
the chart's new default, 9.1.2. Each component now gets its own
ServiceAccount without a mounted token.
Safety net: Longhorn snapshot immich-pg17-pre-v3 and a pg_dump in
~/immich-pg17-pre-v3.dump on opti1, since immich cannot be downgraded.
When a backend closes a kept-alive connection just as Envoy sends a
request over it, Envoy returns a 503 (response flag UC) and does not
retry by default. ingress-nginx retried these, so the race never showed.
The Apache in front of immich (sol-auth) recycles its processes after 30
connections, which made this happen during page loads: three chunks got
a 503 and immich failed with "error loading dynamically imported module".
authentik and ntfy saw it too, less often.
Retry up to twice on connection failures and resets, for every route on
the Gateway.
v2.4.1 only accepts VectorChord >=0.3 <0.6 and refused to start on the new PostgreSQL 17 cluster with VectorChord 1.1.1; from v2.5.0 the range is >=0.3 <2. v2.7.5 is the last 2.x release; the 3.x upgrade comes separately.
The import into immich-postgres-17 matched the old database table for table (assets, embeddings, faces, people, albums, migrations). On start immich builds its VectorChord indexes.
Imported once from the PostgreSQL 14 cluster, whose embeddings were converted to real[] and whose pgvecto.rs extension was dropped first (immich's manual migration). After the import the columns become vector(512) again under VectorChord 1.1.1 with pgvector 0.8.6. A logical import because CNPG can't upgrade in place from the bullseye-based pgvecto.rs image to the bookworm-based VectorChord one.
The database moves from PostgreSQL 14 with pgvecto.rs to a new PostgreSQL 17 cluster with VectorChord. Immich 3 no longer supports pgvecto.rs, and PostgreSQL 14 reaches end of life on 2026-11-12.
With them only Longhorn's own components can reach the longhorn-manager API, which any pod could otherwise call without authentication to manage volumes. 1.12 was kept off because its instance-manager policy left the host-side iSCSI initiator out; 1.13 opens port 3260 to any source while v1DataEngineInitiatorSourceCIDRs is empty.
Drops the label tracking pin kept through the 3.x upgrades, so Argo CD uses its default argocd.argoproj.io/tracking-id annotation. Labels can be copied by other tools (cert-manager's ingress-shim copied app.kubernetes.io/instance onto Certificates); the annotation names the exact resource. Every app gets synced once right after, so resources removed later are still recognised for pruning.
configMapGenerator gives every config change a new gatus-config-<hash> ConfigMap, and without prune the old one stayed behind and left the app OutOfSync.
Each poll updates the backupvolume objects, which made up about 8% of the k8s-dqlite writes at the 300s interval. The recurring backup runs once a night, so 30 minutes is plenty.
Nothing deploys disable/: the apps in it (ente-photos, minio, mist, owntracks, traccar, ttrss, whisper and others) are gone from the cluster apart from some old TLS secrets. git history keeps them. The argocd-cm patch copy was an old leftover that kustomize never used.
Ruben Hensen