Pin authentik-postgres to the image CNPG filled in at creation
(postgresql:17.4). Later operators default to PostgreSQL 18, and since
1.26 a higher major in the image triggers an offline in-place major
upgrade. immich-postgres already pins its pgvecto.rs image.
Turn on ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES so the five operator hops
(1.25 -> 1.30, one minor at a time as the docs recommend) swap the
instance manager inside the running pods instead of restarting both
single-instance databases each time.
Velero 1.15 is only tested up to Kubernetes 1.31; 1.16 is tested on 1.33.
The AWS plugin moves along to v1.12.2. Argo CD applies the chart's crds/
directory itself, so the CRD schema update doesn't need the disabled
upgrade hook. No backup was running.
All certificates stayed Ready through the five hops, and a test Certificate for s2.rubenhensen.nl was issued by Let's Encrypt over HTTP-01. The 1.19 chart left a duplicate tokenrequest RoleBinding behind, removed by hand.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
Last hop of the Longhorn upgrade for Kubernetes 1.33. Stays on 1.11.2:
1.11.3 ships csi-provisioner v6.3.0 and needs Kubernetes 1.34. The
1.11.2 chart still declares kubeVersion >=1.25 and ships csi-provisioner
v5.3.0, and renders our values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-10-2-202610221449 is Ready. PV
nodeAffinity recorded to compare after the upgrade (1.11.0 had a
regression there, fixed in 1.11.2).
Third hop towards 1.11.2. 1.10 removes the longhorn.io/v1beta1 API, which
requires every Longhorn CRD to store only v1beta2; all 22 already do (the
check from the 1.10 important notes). Settings move to a per-data-engine
JSON format, but the chart renders our defaultSettings and
defaultBackupStore values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-9-2-202610221421 is Ready.
Second hop towards 1.11.2. 1.9 stops serving longhorn.io/v1beta1 (nothing
in this repo uses it) and renames orphan-auto-deletion (false here) to
orphan-resource-auto-deletion during the upgrade. Chart values are
unchanged from 1.8.2.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, every Longhorn CRD already stores only v1beta2, and system backup
pre-upgrade-v1-8-2-202610221408 is Ready.
First hop from 1.7.3 to 1.11.2 (docs/upgrade-k8s-1.33.md). 1.8 drops the
backup-target settings in favour of a "default" BackupTarget resource, so
the NFS target and poll interval move into defaultBackupStore. Until now
they only lived in the cluster.
Checked beforehand: no Faulted volumes, no backing images, all volumes on
engine v1.7.3, system backup pre-upgrade-v1-7-3-20261022 and Velero
backup pre-longhorn-1-8-202610221326 both complete.
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
Envoy Gateway v1.9 needs Kubernetes 1.33, and microk8s 1.32 is out of
upstream support. docs/upgrade-k8s-1.33.md lists what has to move first
and in which order. Renovate is held below Longhorn 1.11.3, since that
and newer releases need Kubernetes 1.34.
After the microk8s ingress addon went from ingress-nginx 1.8.0 to 1.13.8,
ha.rubenhensen.nl returned 503: the controller now resolves externalName
as a DNS name and "192.168.1.202" fails the lookup. A selector-less
Service with an EndpointSlice pointing at the IP is the supported way to
route to an address outside the cluster.
Neither is used any more: the blog moves to a self-written site, and
whishper needs more compute than this setup has. Ghost also ran Node 18,
which won't get the eMudhra root store fix.
Drops both apps, the blog DNS record, Gatus check and node001 tunnel
entry, their ca-bundle namespaces, and the leftover ExternalSecret for the
old hensen.io blog namespace. Final Velero backups keep the data:
ghost-blog-final-202610220955 until 22 October 2027 and
whisper-final-202610220955 until 20 January 2027.
Neither host has a DNS record, so both were unreachable and cert-manager
has been failing to renew their certificates since 6 April. DSM stays
reachable on the LAN at 192.168.1.99. The nas app only held that ingress
and its ExternalName service, so it goes entirely.
Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
Velero 1.15 builds kopia repository maintenance jobs from a Deployment
called "velero" in its namespace. Ours was velero-helm, so every
BackupRepository reported 'error to build maintenance job: Deployment.apps
"velero" not found' and the repos on the NAS were never cleaned up, some
not since March 2025. fullnameOverride renames the Deployment, Service,
ServiceAccount and RBAC; the old velero-helm objects get deleted by hand
since this app doesn't prune.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). The server gets --use-openssl-ca for Node, machine-learning points requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). whishper also gets --use-openssl-ca for Node, and both point requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Nine more namespaces need patched-ca-bundle, the bundle from Alpine
ca-certificates-bundle 20261020-r0 without the roots above the compromised
eMudhra intermediates. Rather than another nine ~190KB copies, a small
chart renders the ConfigMap from one ca-certificates.crt into every
namespace in values.yaml. argocd, authentik and actual-budget keep their
own copies for now.
lingo-api ran PHP 8.1.34 (php:8.1-apache) and was reachable at
https://lingo.rubenhensen.nl/api/. PHP 8.1 is EOL and gets no fix for
CVE-2026-109618 and CVE-2026-109619, so it was exposed to the PHP2Shell
RCE. It is no longer used, so drop it entirely: the app, its Gatus
check, the DNS record and the node001 tunnel entry.
wordpress:7.1.3-php8.3-apache was rebuilt at 15:57 UTC on PHP 8.3.36,
which fixes CVE-2026-109618 and CVE-2026-109619. WordPress core lives on
the volume, so the tag only moves PHP (8.3.30 -> 8.3.36) and Apache. Its
ca-certificates (20250419+deb13u1) no longer carries the roots above the
compromised eMudhra intermediates either.
freshrss/freshrss:1.30.3, released today, runs PHP 8.4.27 from Debian
trixie security (8.4.27-1~deb13u1), which fixes CVE-2026-109618 and
CVE-2026-109619. Pin it instead of :latest so the running version is the
checked one. Its ca-certificates (20250419+deb13u1) also no longer
carries the roots above the compromised eMudhra intermediates.
Reverts 8c2bd89. The NAS had fallen back to a DHCP address (.10), which
also moved the UniFi controller it hosts away from where the gateway
and access points inform, and left the Longhorn backup target at
nfs://192.168.1.99 unreachable. It now has 192.168.1.99 set statically
in DSM again, so everything that pointed at .99 works without changes.
At 512Mi the server has been OOMKilled 314 times. With the bucket
reachable again it died 23 seconds into a manual backup of ~14.7k items,
which then got marked Failed on restart. opti1 has 32GB.
The hook's init container pulls docker.io/bitnami/kubectl:1.32, which
Bitnami has removed, so the sync for the new NAS address hung on
ImagePullBackOff. The chart stays at 8.4.0 and the CRDs are already in
place.
The NAS moved from 192.168.1.99 to .10, so the backup storage location
has been Unavailable ("no route to host"). Every backup since at least
26 July failed, and 154 expired ones are stuck in Deleting because Velero
can't reach the bucket to remove them. MinIO answers on .10:9000 and DSM
on .10:5000.
Node uses its built-in roots, which still include emSign ECC Root CA - G3,
and the Node fix is still pending. Switch to OpenSSL's store with
--use-openssl-ca and point it at patched-ca-bundle, mounted over
/etc/ssl/certs. That covers bank sync and the authentik OIDC calls.
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
The Argo CD v2.14.2 image (Ubuntu 24.04, ca-certificates 20240203) and the
dex image still trust the eMudhra intermediates whose keys leaked
(https://emudhra.we-have-a-problem.com), and the repo server fetches git
and Helm repos over that trust store. A forged certificate there means
arbitrary manifests in the cluster.
patched-ca-bundle holds the bundle from Alpine ca-certificates-bundle
20261020-r0 and replaces /etc/ssl/certs in every Argo CD container except
redis. The whole directory goes, since OpenSSL and Go also read the
per-root files next to ca-certificates.crt. It uses server-side apply to
keep the bundle out of the last-applied annotation.
Ruben Hensen