k8scd/docs/upgrade-k8s-1.33.md

Upgrade to Kubernetes 1.33

The cluster runs microk8s 1.32, which is out of upstream support, and Envoy Gateway v1.9 (the planned ingress-nginx replacement) needs 1.33 or newer. Several components are too old for 1.33 and have to move first, most of them one minor version at a time. Everything up to step 7 happens on 1.32.

Compatibility research from 2026-10-22. Recheck release notes before each step; versions move.

1. Preparation

• Every Longhorn volume on engine image v1.7.3, old engine images deleted
• Longhorn system backup (pre-upgrade-v1-7-3-20261022)
• Renovate held at Longhorn 1.11.x (1.11.3 and 1.13 need k8s 1.34)

2. Longhorn 1.7.3 → 1.11.2

One minor per step, engine images upgraded after each one. Check volumes, the NFS backup target, the recurring backup job and the Velero bak snapshots after every step. Fresh Velero backup before starting.

• 1.8.2: backup-target settings move to a default BackupTarget (notes)
• 1.9.2
• Migrate every Longhorn CRD to v1beta2 only in storedVersions. Already the case after 1.8.2/1.9.2; recheck right before 1.10. Mandatory before 1.10; skipping it leaves only a forced downgrade (notes)
• 1.10.2: v1beta1 API removed, settings move to a v1/v2 JSON format. Engine images are now docker.io/longhornio/...
• 1.11.2. Not 1.11.3 or 1.12.x: those need k8s 1.34

3. Smaller components

• CloudNativePG 1.25.1 → 1.26 → 1.27 → 1.28 → 1.29 → 1.30.1 (1.27-1.29 are EOL already). In-place instance manager updates are on; the databases restarted once, on the first hop
• Velero 1.15.2 → 1.16.2 (chart 10.1.3), plugin-for-aws v1.12.2
• Vault chart 0.28.1 → 0.34.1 (vault-k8s 1.7.6), server pinned to 1.17.2. Upgrading the Vault server itself needs the unseal keys
• cert-manager 1.15.5 → 1.16.5 → 1.17.4 → 1.18.6 → 1.19.6 → 1.20.4 (1.18 changes the rotationPolicy default to Always)

4. External Secrets 0.10.7 → 2.13.0

• Every ExternalSecret and the ClusterSecretStore on external-secrets.io/v1
• One minor at a time, 28 hops. The ESO app syncs with server-side apply. At 0.16.2 the old conversion webhook blocked the new CRDs; setting spec.conversion to None by hand on the three CRDs fixed it
• Once every object is stored as v1, drop v1beta1 from the CRDs' status.storedVersions (needed before ESO removes v1beta1 entirely). Done 2026-10-23: the 20 ExternalSecrets and the ClusterSecretStore rewritten (checked in kine that the latest revision of each is v1), then storedVersions: ["v1"] on externalsecrets, clusterexternalsecrets, secretstores and clustersecretstores. Two old v1beta1 managedFields entries on vault-backend removed. Pitfall: kubectl get -o json | kubectl replace -f - rewrites an existing last-applied-configuration with the whole object (resourceVersion, uid, finalizers), after which Argo CD's apply failed on vault-folder ("resourceVersion must be specified") and would have stripped ESO's finalizer. Removing the annotation let Argo CD write a clean one

5. Argo CD 2.14.2 → 3.4.12

• v3.0.23 → v3.1.16 → v3.2.12 → v3.3.18 → v3.4.12. Label tracking is pinned in argocd-cm; our own resource.exclusions replaces the 3.0 defaults, so the home-assistant EndpointSlice stays managed. argocd-folder syncs with server-side apply (needed from 3.3)
• Switch to annotation tracking. Done 2026-10-24: the pin removed from argocd-cm, then every app synced once without prune. A sync started by patching .operation doesn't inherit the app's syncOptions, so the four server-side-apply apps fell back to client-side apply and failed on their big CRDs (annotation over 256 KB); passing the app's syncOptions in the operation fixed it. Six Certificates still carried the app.kubernetes.io/instance label that cert-manager's ingress-shim had copied over, owned by ingress-shim, which kept argocd-folder OutOfSync; removed

6. Calico 3.25.1 → 3.29.3

• Applied microk8s 1.33's manifest (canonical/microk8s branch 1.33, upgrade-scripts/000-switch-to-calico/resources/calico.yaml) with kubectl apply --server-side --force-conflicts. Same calico-node settings as before. The manifest is also in /var/snap/microk8s/current/args/cni-network/cni.yaml; the 3.25.1 one is saved as ~/cni-3.25.1-backup.yaml on opti1

7. microk8s 1.32 → 1.33

• First fix the k8s-dqlite write stalls, see below
• Fresh Velero backup (pre-microk8s-1-33-202610230712) and sudo microk8s dbctl backup (/root/dqlite-pre-1.33-*)
• sudo snap refresh microk8s --channel=1.33/stable: v1.33.13, snap revision 9067, done 2026-10-23
• sudo microk8s addons repo update core: branch 1.33, ingress-nginx v1.13.8
• If kubelite panics with "Missing actuated resource record" (#5057): sudo snap revert microk8s. No panic. The message does show up in the log without consequences, and kubelite has not restarted since the refresh. The dqlite snapshot tuning survived the refresh.
• Argo CD 3.x puts pre-delete finalizers on helm-longhorn, so they are now in git as well

8. After 1.33

• cert-manager 1.21.2. The chart dropped the unused tokenrequest Role and RoleBinding; Argo CD pruned them
• Envoy Gateway v1.9 instead of ingress-nginx, Longhorn behind authentik OIDC
  • Envoy Gateway v1.9.2 next to nginx. Gateway homelab in envoy-gateway-system, Envoy on host ports 10080/10443 (gateway/)
  • One app (gatus, s2.rubenhensen.nl) on an HTTPRoute, reusing its existing certificate through a ReferenceGrant; test with curl --resolve <host>:10443:192.168.1.204
  • Longhorn behind authentik OIDC with a SecurityPolicy, client secret in Vault at kv/longhorn-oidc. Two snags: authentik only picks up a new blueprint file at its hourly discovery (or blueprints_discovery.send() in ak shell), and Envoy Gateway does not retry a failed OIDC discovery until something changes, so the controller needed a restart. While the policy is invalid Envoy answers 500, so Longhorn stays closed
  • The other apps. Envoy cuts requests off after 15s by default, so the routes set timeouts.request: 0s. All hosts answer the same through Envoy as through nginx; WordPress only redirects on the test port because the Host header carries :10443
  • Argo CD on server.insecure, so both nginx and Envoy talk plain HTTP to argocd-server
  • cert-manager config.gatewayAPI.enabled
  • Switch over: microk8s disable ingress, Envoy on 80/443, and a gatewayHTTPRoute HTTP-01 solver on the ClusterIssuer. About a minute without HTTP(S) on 2026-10-23; node001's tunnel needed no change
  • Replace the Ingress objects with Certificate resources (the ingress-shim Certificates go with their Ingress, and gateway-shim can't write to the app namespaces) and drop the nginx annotations. Same names and specs, so nothing was reissued except immich, which dropped mbg.hensen.io; that reissue went through the Gateway's HTTP-01 solver. The Ingresses were pruned with PrunePropagationPolicy=orphan so the Certificates stayed. Gatus now expects a 302 from Longhorn
  • Retries (gateway/retry-policy.yaml, 2026-10-24). When a backend closes a kept-alive connection just as Envoy sends a request over it, Envoy answers 503 (flag UC) and, unlike nginx, does not retry. Behind sol-auth's Apache, which recycled its processes after 30 connections, that cost immich chunks during page loads ("error loading dynamically imported module"). A BackendTrafficPolicy on the Gateway retries twice on connect failures and resets

9. immich: PostgreSQL 14 with pgvecto.rs → 17 with VectorChord

PostgreSQL 14 reaches end of life on 2026-11-12, and immich 3.0 drops pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:

• Safety net: Longhorn snapshot and NAS backup immich-pg14-pre-vchord of the old volume, plus pg_dump -Fc (447 MB) in ~/immich-pg14-pre-vchord.dump on opti1
• immich server to 0 replicas (through git)
• On the old cluster, immich's manual migration in one transaction: drop clip_index and face_index, embedding columns to real[], DROP EXTENSION vectors, DROP SCHEMA vectors, reset the database's search_path (it included vectors)
• New cluster immich-postgres-17 (immich/postgresql-cluster-17.yaml): cloudnative-vectorchord:17.11-1.1.1 (pgvector 0.8.6, VectorChord 1.1.1), filled by a CNPG microservice import, then vchord created and the columns back to vector(512). An in-place pg_upgrade wasn't possible: the pgvecto.rs image is Debian bullseye, the VectorChord ones bookworm. Tables, indexes and row counts matched the old database
• immich pointed at immich-postgres-17-rw and its own app secret. v2.4.1 refused to start (it only accepts VectorChord <0.6); v2.5.0 and up accept <2, so immich went to v2.7.5, the last 2.x. It rebuilt both indexes as vchordrq in about 1.5 minutes
• After a few days without problems: delete the old cluster (immich/postgresql-cluster.yaml), the snapshot/backup and the dump. Done 2026-10-28 (PR #61), with the PG14 volume's Longhorn backups. Rolling back means reverting the old volume to the snapshot, since its embeddings are now real[] without pgvecto.rs
• immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on 2026-10-24, about three minutes of downtime (two of them pulling the image). Charts after 0.12.0 are only in the OCI registry (ghcr.io/immich-app/immich-charts), and since 0.13.0 the values schema rejects size, accessMode and storageClass on emptyDir volumes. None of the env variables v3 removed were set; the ML image needs x86-64-v2, and the i5-8600T is v3. The migrations took 16s, both vector indexes stayed, and immich-admin schema-check reports no drift (the geodata_places_pkey warning at startup came from the geodata import running at the same time). Safety net, since immich can't be downgraded: Longhorn snapshot immich-pg17-pre-v3 and ~/immich-pg17-pre-v3.dump (440 MB) on opti1
• Optional: MACHINE_LEARNING_MODEL_REVISION=v2 on the ML container (v3.3.0), for faster ML with less memory; numerically the same models, so no jobs to rerun. The model cache is an emptyDir, so the one-off preparation repeats on every restart
• After a few days without problems: delete the immich-pg17-pre-v3 snapshot and dump together with the PG14 leftovers above (2026-10-28)

k8s-dqlite write stalls (found 2026-10-22)

Lease updates regularly take more than 10s (169 of ~162k since the API server started), so controllers using leader election keep losing it and restarting: snapshot-controller over 1000 times, the CNPG operator, Longhorn's CSI sidecars.

Cause: the SQLite file behind dqlite is ~709 MB (173k pages of 4 KB) but 164k pages are free. The live data is ~35 MB: 3,700 kine rows for 3,006 keys, so compaction works. SQLite never returns free pages without VACUUM, and dqlite writes the whole file into a snapshot (~150 MB compressed) every 1,024 commits, i.e. every few minutes. Each snapshot takes about 6s of writing on the Apacer AS350 (DRAM-less SATA SSD, which also holds every Longhorn volume). Commits queue behind it. Outside snapshots fsync takes about 1.2 ms.

Constraints:

• microk8s 1.32 and 1.33 ship k8s-dqlite v1.3.5 (dqlite 1.17 LTS). VACUUM needs dqlite 1.18.2+ (microk8s 1.34 and up). dbctl restore writes into the running DB, so it doesn't shrink anything.
• microk8s 1.34+ ships k8s-dqlite 1.8.x with a reworked write path; that is where Canonical fixed this class of problem (canonical/microk8s#5153).

Plan (needs sudo; the API is down for the length of a dqlite restart, workloads keep running). ~/dqlite-tuning.sh and ~/tuning.yaml on opti1 do the first three steps: ssh -t ruben@192.168.1.204 ./dqlite-tuning.sh

• sudo microk8s dbctl backup -o /root/dqlite-pre-tuning-<date>
• Install backend/tuning.yaml with snapshot: {threshold: 8192, trailing: 16384}: a snapshot every ~25 minutes instead of every ~3. Never set trailing alone (threshold then becomes 0). Leaving out the kine intervals keeps their defaults. Costs ~0.5 GB more raft log on disk.
• sudo snap restart microk8s.daemon-k8s-dqlite, then check snapshot spacing in the backend dir and that lease updates stay under 10s. Measured over the first 40 minutes: snapshots 24.7 minutes apart (were ~3), 0 of 9,445 lease updates over 4s, no controller restarts
• Optional: Longhorn backupstore poll interval 300s -> 1800s (backupvolume updates are ~8% of all writes). Done 2026-10-24
• Later: microk8s 1.34+ for the new write path and VACUUM, see "On to 1.36" below.

--cgroups-per-qos=false, the likely trigger of the 1.32->1.33 kubelite panic (#5057), is not set in args/kubelet.

Still stalling after the VACUUM (2026-10-24)

The smaller database fixed the snapshot stalls, but controllers kept losing their leases a few times a day: since the API server started, 256 of ~278k lease updates took over 5s and 44 over 10s. dqlite's log shows short complete stops (at 10:00:42 even its own health query hit its deadline). Every lease loss that day lined up with minutes of heavy writes to the one SSD, which holds root, dqlite, every Longhorn replica, image layers and emptyDirs:

Lease losses	Writing at the time
00:02	Longhorn backups of every volume (from 00:00)
01:09	Velero backup (kopia reading and caching 133 GB of photos)
09:22	full kopia maintenance of mbgwp (6 min) and argocd (11 min)
10:00:43	full kopia maintenance of vault (9 min)
10:31, 10:38	immich upgrade: image pulls, pg_dump, migrations

Velero ran maintenance every hour for each of the 15 kopia repositories (15 pods an hour), without resource limits and with the kopia cache in an emptyDir; about once a day a run is a full maintenance of minutes, at whatever time of day the hourly schedule had drifted to.

• Velero: defaultRepoMaintainFrequency: 24h, the 15 existing BackupRepositories patched to 24h by hand, and maintenance jobs capped at 500m CPU (2Gi memory)
• Leases of 60s/40s/5s instead of 15s/10s/2s (duration, renew deadline, retry period) for the snapshot-controller (kustomize patch in its Argo CD app), the CNPG operator (additionalArgs) and Envoy Gateway (provider.kubernetes.leaderElection). Longhorn's CSI sidecars and global manager have no setting for it
• ~/iosample.sh on opti1 logs sda's counters and io pressure every 2s for 26h (from 2026-10-24 12:35 UTC) into ~/iosample.log, to put numbers on the disk latency during tonight's backups. Read on 2026-10-25 at 02:35 UTC, per 10 minutes:
  • Quiet: ~50 writes/s, 0.5-2 ms per write on average, "full" IO pressure under 15%
  • 14:30-23:15: 250-440 writes/s and 200-400 reads/s, 6-20 ms per write, pressure 45-75%. Longhorn merging immich-claim's deleted snapshots (docs/longhorn-second-ssd.md)
  • Longhorn backups 00:00-00:15: up to 1,220 reads/s, pressure 69%; the Velero backup at 01:00 was incremental and short (pressure 29%)
  • The authentik and Vault upgrades 01:20-01:40 (image pulls, migrations): 19-33 ms per write, single 2s windows near 2 s per write, pressure 78%. The CSI sidecars and Longhorn's global manager lost their leases at 01:31 Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25 11:30, leaving out the CNPG range cancels): all 67 minutes with errors had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and "full" pressure at 45-78%, against a median minute peak of 24 ms. The worst minute after the merges was the rollout of eight merged PRs at 2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller, CNPG and Envoy Gateway held their 60s leases through all of it Ubuntu's sysstat already keeps per-device history every 10 minutes without a script: sar -d -f /var/log/sysstat/sa24 showed sda at ~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
• Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with DRAM for Longhorn's data (replicas move over live by evicting the old disk) would leave the SATA SSD to dqlite and the system

On to 1.36

microk8s goes up one minor per refresh. 1.34 brings k8s-dqlite 1.8.x with the reworked write path and dqlite 1.18.2, which can VACUUM the mostly empty database. Before every hop: fresh Velero backup, no deprecated APIs in use (apiserver_requested_deprecated_apis metric empty), and the Calico manifest of the target version (canonical/microk8s branch <version>, upgrade-scripts/000-switch-to-calico/resources/calico.yaml).

Findings from 2026-10-23:

• microk8s 1.34, 1.35 and 1.36 all ship k8s-dqlite v1.8.1, which is built against dqlite v1.18.2 (1.33: v1.3.5 with dqlite v1.17.1). So VACUUM is possible from 1.34 on; 1.35 and 1.36 are for staying in support (Kubernetes 1.34 reaches end of life on 2026-10-27).
• Calico is v3.29.3 in all four branches, so no CNI change.
• containerd stays 1.7.29 on 1.34 but jumps to 2.1 on 1.35 and 2.2 on 1.36. Check args/containerd-template.toml before 1.35: it still has the old pause:3.7 sandbox image.
• backend/tuning.yaml still works in k8s-dqlite 1.8.1 (defaults there: threshold 512, trailing 4096, dynamic strategy).
• The 1.34 configure hook adds ListFromCacheSnapshot=false,SizeBasedListCostEstimate=false,DetectCacheInconsistency=false to the existing --feature-gates=WatchList=false line for kine clusters. WatchList is still beta in 1.34, so it can stay off.
• certs/front-proxy-client.crt expired on 2025-09-13 and microk8s never renews it (canonical/microk8s#5576). Nothing uses the aggregation layer here yet, but it gets renewed with the refresh.

microk8s 1.34

• Read canonical/microk8s#5568: 1.34 locks ConsistentListFromCache on, and k8s-dqlite's watch progress notifications lag under load, so LISTs can wait up to 3s before falling back to storage. Latency, not correctness; worst on busy HA clusters. Still open
• Velero backup: the full pre-microk8s-1-33-202610230712 from that morning; nothing that isn't in git changed since
• ssh -t ruben@192.168.1.204 ./microk8s-1.34.sh: dbctl backup, an offline copy of the backend (k8s-dqlite 1.8 migrates the schema on first start, so 1.33 can't read it afterwards), snap refresh to 1.34/stable, the addons repo, and the front-proxy certificate. Done 2026-10-23: v1.34.9, snap revision 9063. Backups in /root/backend-1.33-20261023-1938.tar and /root/dqlite-pre-1.34-20261023-1938.tar.gz. The tuning.yaml snapshot settings carried over; front-proxy-client.crt now runs to 2027
• Afterwards: LIST latency (apiserver_watch_cache_read_wait_seconds), lease update times and controller restarts. LISTs as fast as on 1.33 (92-234 ms), no watch-cache wait over 0.1s and no storage fallbacks, so #5568 doesn't show on this single node. The CNPG operator lost its lease during the refresh and restarted once, as expected

Components for 1.35 and 1.36

• Longhorn 1.11.2 → 1.12.1 → 1.13.0 (one minor at a time, now allowed on 1.34); Renovate hold lifted. A system backup before each step (pre-upgrade-v1-11-2-20261023, pre-upgrade-v1-12-1-20261023); all 19 volumes moved to the new engine image within ten minutes per step, and the unused engine images were deleted. Kept off during the upgrades: 1.12's internal NetworkPolicies (Calico enforces them, and the 1.12 instance-manager policy only admits Longhorn pods while iSCSI comes from the host). Turned on afterwards on 1.13.0, which opens 3260 to any source: a pod outside Longhorn can no longer reach the longhorn-manager API (it could before), the UI still can, and a new volume attached over iSCSI, took writes and backed up to the NAS. Also kept off: 1.13's cluster-wide Secret read for the CSI controller (no StorageClass uses provisioner secrets). The global manager runs one replica. Checked on 1.13.0: a manual backup to the NAS, provisioning a new PVC, and the UI behind authentik
• Argo CD 3.4 → 3.5.6 (3.4 is tested up to 1.35, 3.5 on 1.32-1.36). 3.5 renders with Helm 4.2.1 instead of 3.19; all thirteen Helm apps rendered the same with both beforehand (alpine/helm images, live app inputs), and after the upgrade all 35 apps stayed Synced/Healthy without a sync
• Velero 1.16 → 1.17.1 (chart 11.4.0, AWS plugin v1.13.2) → 1.18.2 (chart 12.2.1, AWS plugin v1.14.4). Every pod volume backup and repository was kopia already. 1.17 moved --keep-latest-maintenance-jobs into the velero-repo-maintenance ConfigMap, which the chart renders. A gatus test backup completed on both versions
• Recheck CNPG, External Secrets, the Vault chart and Envoy Gateway against 1.36. Done 2026-10-23: CNPG 1.30 supports 1.34-1.36, ESO 2.13 up to 1.36, Envoy Gateway 1.9 1.33-1.36, the Vault chart >= 1.20

microk8s 1.35 and 1.36

Checked for 1.35 on 2026-10-23: default args and the configure hook are the same as 1.34's; the four feature gates set to false (WatchList and the three kine ones) stay beta through 1.36, so the API server still starts; the containerd template is the same apart from our older pause:3.7, which containerd 2.1 accepts. Every component supports 1.35 (Velero 1.16 only "expected"). args/certs.d/docker.io/hosts.toml had a stray y after the bitnamilegacy rewrite, so containerd logged "failed to decode hosts.toml" on every docker.io pull and fell back to defaults; fixed on opti1, the original is in ~/hosts.toml.docker.io.bak.

• ssh -t ruben@192.168.1.204 ./microk8s-1.35.sh: dbctl backup, snap refresh to 1.35/stable, addons repo (no schema migration, so snap revert is the way back). Done 2026-10-23: v1.35.6, snap revision 9072, containerd 2.1.6. containerd was down for 26s and logged no config warnings and no hosts.toml errors; all workloads, apps, certificates, volumes and both status pages green; LISTs 92-166 ms, no slow lease updates Checked for 1.36 on 2026-10-23: default args, containerd template and configure hook unchanged from 1.35; containerd 2.2 only drops registry mirrors/configs/auths (in 2.3), and the template uses config_path; every component supports 1.36; only the core/v1 Endpoints API is still in use as a deprecated API.

The microk8s stable channels lag upstream: 1.35/stable is v1.35.6 and 1.36/stable v1.36.2 (both from early July), while upstream is at v1.35.9 and v1.36.5. Only 1.36/edge has CVE-2026-2270 fixed (StatefulSet controller, cross-namespace pod creation; fixed in 1.35.9/1.36.5). Chosen: 1.36/stable with the snap held, so updates only happen on purpose (sudo snap refresh microk8s once stable passes v1.36.5). Before the hold, snapd refreshed microk8s within its channel four times a day.

• ssh -t ruben@192.168.1.204 ./microk8s-1.36.sh: dbctl backup, snap refresh to 1.36/stable, addons repo, snap refresh --hold microk8s. Done 2026-10-23: v1.36.2, snap revision 9092, containerd 2.2.3, snap held. All workloads, apps, certificates, volumes and both status pages green; LISTs 92-166 ms, no slow lease updates
• CVE-2026-2270: moved to 1.36/edge (v1.36.5, snap revision 9574) on 2026-10-23 with ~/microk8s-1.36.5.sh, still held. The edge build has the same containerd 2.2.3, runc 1.4.2 and k8s-dqlite 1.8.1 as the stable v1.36.2; only build and test fixes landed on the branch since. Everything green afterwards; one Velero maintenance job was cut off by the restart and runs again at the next interval
• Back to 1.36/stable once it is at v1.36.5 or later (sudo snap refresh microk8s --channel=1.36/stable); before that, the switch would install the older v1.36.2

Shrink the database

dqlite accepts VACUUM since v1.18.2 (canonical/dqlite#734). Its maintainer: "it potentially create a very big transaction which has effects both on the RAFT protocol and on resource usage. I would recommend not to run it unsupervised." The dqlite shell that ships with microk8s can't send it: it wraps every statement in a transaction, and SQLite refuses to VACUUM inside one. tools/dqlite-vacuum sends it through the go-dqlite driver as a plain statement, as user ruben (in the microk8s group, which can read cluster.crt, cluster.key and info.yaml); without -vacuum it only prints page statistics.

Rehearsed on 2026-10-23 on opti1 with the k8s-dqlite and dqlite binaries of microk8s 1.34, on a throwaway node in /dev/shm (not in Docker on the Mac: the amd64 emulation lacks io_setup, which raft needs): 657 MB with 616 MB free went to 41 MB with none free in 0.6s; a writer next to it saw no failures and no write over 1s; after a restart, and after a restart from the first snapshot (40 MB), integrity_check was ok and all rows were there.

Live database before: 709 MB, 666 MB free, ~4,400 kine rows; snapshots of ~150 MB every ~23 minutes; k8s-dqlite RSS 1,346 MB.

• ssh -t ruben@192.168.1.204 ./dqlite-vacuum.sh: dbctl backup, an offline copy of the backend (API down ~1 min), VACUUM, integrity check. How to go back is at the top of the script. Done 2026-10-23 20:10 UTC: 709 MB with 666 MB free became 44 MB with none free; integrity_check ok. Backups in /root/backend-pre-vacuum-20261023-*.tar and /root/dqlite-pre-vacuum-20261023-*.tar.gz
• Afterwards: the first snapshot (20:21) is 8.3 MB instead of ~150 MB, and k8s-dqlite's memory went from 1,346 MB to 403 MB. The seven leader-elected controllers (snapshot-controller, Longhorn's CSI sidecars, CNPG, Envoy Gateway) restarted once while dqlite was stopped for the offline copy, not during the VACUUM