dqlite supports VACUUM from v1.18.2, which microk8s 1.34 ships, but the dqlite shell wraps every statement in a transaction and SQLite refuses to VACUUM inside one. dqlite-vacuum sends it through the go-dqlite driver instead and otherwise just prints page statistics. dqlite-vacuum.sh makes the backups first.
Rehearsed on opti1 on a throwaway node in /dev/shm: 657 MB to 41 MB in 0.6s, with concurrent writes unaffected and the data intact after restarts.
v1.34.9 runs with the dqlite snapshot tuning intact. LIST latency is unchanged from 1.33, so the ConsistentListFromCache stalls of canonical/microk8s#5568 don't show here.
k8s-dqlite v1.8.1 in microk8s 1.34 already bundles dqlite 1.18.2, so the database can be vacuumed from 1.34 on. Also notes the containerd 2 jump at 1.35, the feature gates the 1.34 hook adds for kine, and the front-proxy certificate that expired in 2025.
Longhorn now sits behind an authentik login on the Gateway instead of nginx basic auth, so the check expects a 302 without following it. Same change for s1 on node001.
ingress-nginx is gone and every app is served by the Envoy Gateway. The certificates were created by cert-manager's ingress-shim and are owned by the Ingresses, so each app now declares its Certificate itself, with the same name, secret and usages, so nothing gets reissued. gateway-shim can't take over because the certificates live in the app namespaces.
immich drops mbg.hensen.io, which has no route, so that one certificate is reissued through the Gateway's HTTP-01 solver.
Also removes the Longhorn test-port redirect URI and updates comments that still described nginx.
Longhorn has no users of its own; nginx guarded it with basic auth using apr1 hashes, which Envoy Gateway does not support. An Envoy Gateway SecurityPolicy now sends visitors through an authentik OIDC application that only admins may use. The client secret lives in Vault at kv/longhorn-oidc and reaches both authentik and Envoy through External Secrets.
Envoy moves from the test ports to 80 and 443, and Let's Encrypt HTTP-01 challenges go through an HTTPRoute on the Gateway. Push only after sudo microk8s disable ingress, since both want the same host ports.
With server.insecure, TLS ends at nginx or Envoy and argocd-server serves plain HTTP, which is how Argo CD documents the Gateway API setup. The nginx Ingress now talks HTTP to the http port; the ssl-passthrough annotation did nothing without the controller flag. argocd-folder diffs server-side for the new HTTPRoute.
Each app gets an HTTPS listener on the Gateway with its existing certificate, an HTTPRoute, and a ReferenceGrant for that certificate. The routes drop Envoy's 15s request timeout, as nginx has no overall limit either. Argo CD and Longhorn follow separately; the Ingresses stay until the switch-over.
The API server fills in Gateway API defaults such as the parentRef group and kind and a PathPrefix / match, so Argo CD's client-side diff kept gateway-folder OutOfSync.
First app on the Gateway API: an HTTPS listener for s2.rubenhensen.nl using the existing certificate in the gatus namespace, an HTTPRoute, and a ReferenceGrant for the certificate. Plain HTTP on the Gateway now redirects to HTTPS. The Ingress stays until the switch-over.
Envoy runs as a DaemonSet on host ports 10080 and 10443 for now, so it can be tested next to ingress-nginx, which keeps 80 and 443. The Gateway has a plain HTTP listener and no routes yet.
First step of the move from the retired ingress-nginx to the Gateway API. This installs the controller with the Gateway API and Envoy Gateway CRDs. No Gateway exists yet, so no traffic changes.
1.21 supports Kubernetes 1.33 to 1.36. The chart no longer creates the tokenrequest Role and RoleBinding; nothing uses them, as the only issuer is ACME with HTTP-01 and no serviceAccountRef.
The nightly backup of 2026-10-23 ended PartiallyFailed because a repository maintenance job pod finished while Velero was reading its emptyDir volumes. emptyDir holds nothing worth restoring, so a volume policy now skips it for both schedules.
The node runs v1.33.13 (snap revision 9067). kubelite logs the #5057 message but has not panicked or restarted, and the dqlite snapshot tuning carried over.
Argo CD 3.x adds pre-delete-finalizer.argocd.argoproj.io and its /cleanup variant to Applications with PreDelete hooks, which the Longhorn chart has. With only resources-finalizer in git, the apps app-of-apps stayed OutOfSync on helm-longhorn.
Over the first 40 minutes after the restart: snapshots 24.7 minutes apart instead of ~3, none of 9,445 lease updates took over 4s, and neither the CNPG operator nor the snapshot-controller restarted.
1.34 brings the reworked k8s-dqlite write path, 1.36 a dqlite that can VACUUM the mostly empty datastore. Lists the per-hop checks, the one-way schema migration at 1.34, the components that still need to move for 1.35/1.36, and the staged dqlite tuning script on opti1.
The dqlite SQLite file is ~709 MB with 95% free pages, so every 1,024 commits dqlite writes a ~150 MB snapshot that stalls commits for seconds and makes controllers lose leader election. Plan: larger snapshot threshold in tuning.yaml now, microk8s 1.34+ for the real fix.
Required from 3.3, whose ApplicationSet CRD no longer fits the last-applied annotation of client-side apply. Turned on while still on 3.2 so the managed-field migration happens before the CRD grows.
Argo CD 3.0 switches the default to annotation tracking. Per the 2.14->3.0
upgrade notes, a resource whose first sync after the switch is a deletion
gets orphaned instead of pruned. Pinning label tracking (the 2.14 default,
so no change today) keeps the upgrade to 3.x free of that; moving to
annotations can follow once everything runs 3.4.
ESO 0.16.2 is the one release that serves both v1beta1 and v1. 0.17
stops serving v1beta1, so manifests still on v1beta1 would fail to sync
from the next hop on. For these resources the schemas are name-identical
(conversion strategy None); only the apiVersion changes.
Needed from ESO 0.19, whose CRDs are too big for client-side apply. Turned on now, on 0.10.7, so Argo CD migrates the managed fields before the upgrade starts.
Dropping the anti-affinity wasn't enough: during the rollout the old ReplicaSet kept recreating its pod with the old anti-affinity, which still blocked the new one. Recreate stops the old pod first; nothing uses the injector, so the short gap doesn't matter.
The chart requires injector pods on different nodes. With one node and one replica the rolling update to vault-k8s 1.7.6 deadlocked: the new pod stayed Pending until the old one left, which waits for the new one. The server already sets affinity to "" for the same reason.
Chart 0.28.1 is only tested up to Kubernetes 1.30. 0.34.1 brings the
injector to vault-k8s 1.7.6. The server image is pinned to the running
1.17.2: the StatefulSet uses OnDelete, so nothing restarts now, but a
later pod restart would otherwise pull the chart's default and Shamir
unsealing needs the keys at hand for a Vault upgrade. Rendered with the
pin, the StatefulSet only changes in formatting and an extra check in the
config script that doesn't apply here.
Pin authentik-postgres to the image CNPG filled in at creation
(postgresql:17.4). Later operators default to PostgreSQL 18, and since
1.26 a higher major in the image triggers an offline in-place major
upgrade. immich-postgres already pins its pgvecto.rs image.
Turn on ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES so the five operator hops
(1.25 -> 1.30, one minor at a time as the docs recommend) swap the
instance manager inside the running pods instead of restarting both
single-instance databases each time.
Velero 1.15 is only tested up to Kubernetes 1.31; 1.16 is tested on 1.33.
The AWS plugin moves along to v1.12.2. Argo CD applies the chart's crds/
directory itself, so the CRD schema update doesn't need the disabled
upgrade hook. No backup was running.
All certificates stayed Ready through the five hops, and a test Certificate for s2.rubenhensen.nl was issued by Let's Encrypt over HTTP-01. The 1.19 chart left a duplicate tokenrequest RoleBinding behind, removed by hand.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
Last hop of the Longhorn upgrade for Kubernetes 1.33. Stays on 1.11.2:
1.11.3 ships csi-provisioner v6.3.0 and needs Kubernetes 1.34. The
1.11.2 chart still declares kubeVersion >=1.25 and ships csi-provisioner
v5.3.0, and renders our values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-10-2-202610221449 is Ready. PV
nodeAffinity recorded to compare after the upgrade (1.11.0 had a
regression there, fixed in 1.11.2).
Third hop towards 1.11.2. 1.10 removes the longhorn.io/v1beta1 API, which
requires every Longhorn CRD to store only v1beta2; all 22 already do (the
check from the 1.10 important notes). Settings move to a per-data-engine
JSON format, but the chart renders our defaultSettings and
defaultBackupStore values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-9-2-202610221421 is Ready.
Second hop towards 1.11.2. 1.9 stops serving longhorn.io/v1beta1 (nothing
in this repo uses it) and renames orphan-auto-deletion (false here) to
orphan-resource-auto-deletion during the upgrade. Chart values are
unchanged from 1.8.2.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, every Longhorn CRD already stores only v1beta2, and system backup
pre-upgrade-v1-8-2-202610221408 is Ready.
First hop from 1.7.3 to 1.11.2 (docs/upgrade-k8s-1.33.md). 1.8 drops the
backup-target settings in favour of a "default" BackupTarget resource, so
the NFS target and poll interval move into defaultBackupStore. Until now
they only lived in the cluster.
Checked beforehand: no Faulted volumes, no backing images, all volumes on
engine v1.7.3, system backup pre-upgrade-v1-7-3-20261022 and Velero
backup pre-longhorn-1-8-202610221326 both complete.
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
Envoy Gateway v1.9 needs Kubernetes 1.33, and microk8s 1.32 is out of
upstream support. docs/upgrade-k8s-1.33.md lists what has to move first
and in which order. Renovate is held below Longhorn 1.11.3, since that
and newer releases need Kubernetes 1.34.