ESO 0.16.2 is the one release that serves both v1beta1 and v1. 0.17
stops serving v1beta1, so manifests still on v1beta1 would fail to sync
from the next hop on. For these resources the schemas are name-identical
(conversion strategy None); only the apiVersion changes.
Needed from ESO 0.19, whose CRDs are too big for client-side apply. Turned on now, on 0.10.7, so Argo CD migrates the managed fields before the upgrade starts.
Dropping the anti-affinity wasn't enough: during the rollout the old ReplicaSet kept recreating its pod with the old anti-affinity, which still blocked the new one. Recreate stops the old pod first; nothing uses the injector, so the short gap doesn't matter.
The chart requires injector pods on different nodes. With one node and one replica the rolling update to vault-k8s 1.7.6 deadlocked: the new pod stayed Pending until the old one left, which waits for the new one. The server already sets affinity to "" for the same reason.
Chart 0.28.1 is only tested up to Kubernetes 1.30. 0.34.1 brings the
injector to vault-k8s 1.7.6. The server image is pinned to the running
1.17.2: the StatefulSet uses OnDelete, so nothing restarts now, but a
later pod restart would otherwise pull the chart's default and Shamir
unsealing needs the keys at hand for a Vault upgrade. Rendered with the
pin, the StatefulSet only changes in formatting and an extra check in the
config script that doesn't apply here.
Pin authentik-postgres to the image CNPG filled in at creation
(postgresql:17.4). Later operators default to PostgreSQL 18, and since
1.26 a higher major in the image triggers an offline in-place major
upgrade. immich-postgres already pins its pgvecto.rs image.
Turn on ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES so the five operator hops
(1.25 -> 1.30, one minor at a time as the docs recommend) swap the
instance manager inside the running pods instead of restarting both
single-instance databases each time.
Velero 1.15 is only tested up to Kubernetes 1.31; 1.16 is tested on 1.33.
The AWS plugin moves along to v1.12.2. Argo CD applies the chart's crds/
directory itself, so the CRD schema update doesn't need the disabled
upgrade hook. No backup was running.
All certificates stayed Ready through the five hops, and a test Certificate for s2.rubenhensen.nl was issued by Let's Encrypt over HTTP-01. The 1.19 chart left a duplicate tokenrequest RoleBinding behind, removed by hand.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
Last hop of the Longhorn upgrade for Kubernetes 1.33. Stays on 1.11.2:
1.11.3 ships csi-provisioner v6.3.0 and needs Kubernetes 1.34. The
1.11.2 chart still declares kubeVersion >=1.25 and ships csi-provisioner
v5.3.0, and renders our values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-10-2-202610221449 is Ready. PV
nodeAffinity recorded to compare after the upgrade (1.11.0 had a
regression there, fixed in 1.11.2).
Third hop towards 1.11.2. 1.10 removes the longhorn.io/v1beta1 API, which
requires every Longhorn CRD to store only v1beta2; all 22 already do (the
check from the 1.10 important notes). Settings move to a per-data-engine
JSON format, but the chart renders our defaultSettings and
defaultBackupStore values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-9-2-202610221421 is Ready.
Second hop towards 1.11.2. 1.9 stops serving longhorn.io/v1beta1 (nothing
in this repo uses it) and renames orphan-auto-deletion (false here) to
orphan-resource-auto-deletion during the upgrade. Chart values are
unchanged from 1.8.2.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, every Longhorn CRD already stores only v1beta2, and system backup
pre-upgrade-v1-8-2-202610221408 is Ready.
First hop from 1.7.3 to 1.11.2 (docs/upgrade-k8s-1.33.md). 1.8 drops the
backup-target settings in favour of a "default" BackupTarget resource, so
the NFS target and poll interval move into defaultBackupStore. Until now
they only lived in the cluster.
Checked beforehand: no Faulted volumes, no backing images, all volumes on
engine v1.7.3, system backup pre-upgrade-v1-7-3-20261022 and Velero
backup pre-longhorn-1-8-202610221326 both complete.
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
Envoy Gateway v1.9 needs Kubernetes 1.33, and microk8s 1.32 is out of
upstream support. docs/upgrade-k8s-1.33.md lists what has to move first
and in which order. Renovate is held below Longhorn 1.11.3, since that
and newer releases need Kubernetes 1.34.
After the microk8s ingress addon went from ingress-nginx 1.8.0 to 1.13.8,
ha.rubenhensen.nl returned 503: the controller now resolves externalName
as a DNS name and "192.168.1.202" fails the lookup. A selector-less
Service with an EndpointSlice pointing at the IP is the supported way to
route to an address outside the cluster.
Neither is used any more: the blog moves to a self-written site, and
whishper needs more compute than this setup has. Ghost also ran Node 18,
which won't get the eMudhra root store fix.
Drops both apps, the blog DNS record, Gatus check and node001 tunnel
entry, their ca-bundle namespaces, and the leftover ExternalSecret for the
old hensen.io blog namespace. Final Velero backups keep the data:
ghost-blog-final-202610220955 until 22 October 2027 and
whisper-final-202610220955 until 20 January 2027.
Neither host has a DNS record, so both were unreachable and cert-manager
has been failing to renew their certificates since 6 April. DSM stays
reachable on the LAN at 192.168.1.99. The nas app only held that ingress
and its ExternalName service, so it goes entirely.
Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
Velero 1.15 builds kopia repository maintenance jobs from a Deployment
called "velero" in its namespace. Ours was velero-helm, so every
BackupRepository reported 'error to build maintenance job: Deployment.apps
"velero" not found' and the repos on the NAS were never cleaned up, some
not since March 2025. fullnameOverride renames the Deployment, Service,
ServiceAccount and RBAC; the old velero-helm objects get deleted by hand
since this app doesn't prune.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
Ruben Hensen