k8scd/docs/upgrade-k8s-1.33.md

# Upgrade to Kubernetes 1.33

The cluster runs microk8s 1.32, which is out of upstream support, and Envoy
Gateway v1.9 (the planned ingress-nginx replacement) needs 1.33 or newer.
Several components are too old for 1.33 and have to move first, most of
them one minor version at a time. Everything up to step 7 happens on 1.32.

Compatibility research from 2026-10-22. Recheck release notes before each
step; versions move.

## 1. Preparation

- [x] Every Longhorn volume on engine image v1.7.3, old engine images deleted
- [x] Longhorn system backup (`pre-upgrade-v1-7-3-20261022`)
- [x] Renovate held at Longhorn 1.11.x (1.11.3 and 1.13 need k8s 1.34)

## 2. Longhorn 1.7.3 → 1.11.2

One minor per step, engine images upgraded after each one. Check volumes,
the NFS backup target, the recurring backup job and the Velero `bak`
snapshots after every step. Fresh Velero backup before starting.

- [x] 1.8.2: `backup-target` settings move to a `default` BackupTarget
  ([notes](https://longhorn.io/docs/archives/1.8.2/important-notes/))
- [x] 1.9.2
- [x] Migrate every Longhorn CRD to `v1beta2` only in `storedVersions`.
  Already the case after 1.8.2/1.9.2; recheck right before 1.10.
  Mandatory before 1.10; skipping it leaves only a forced downgrade
  ([notes](https://longhorn.io/docs/archives/1.10.2/important-notes/))
- [x] 1.10.2: v1beta1 API removed, settings move to a v1/v2 JSON format.
  Engine images are now `docker.io/longhornio/...`
- [x] 1.11.2. Not 1.11.3 or 1.12.x: those need k8s 1.34

## 3. Smaller components

- [x] CloudNativePG 1.25.1 → 1.26 → 1.27 → 1.28 → 1.29 → 1.30.1 (1.27-1.29 are
  EOL already). In-place instance manager updates are on; the databases
  restarted once, on the first hop
- [x] Velero 1.15.2 → 1.16.2 (chart 10.1.3), plugin-for-aws v1.12.2
- [x] Vault chart 0.28.1 → 0.34.1 (vault-k8s 1.7.6), server pinned to 1.17.2.
  Upgrading the Vault server itself needs the unseal keys
- [x] cert-manager 1.15.5 → 1.16.5 → 1.17.4 → 1.18.6 → 1.19.6 → 1.20.4
  (1.18 changes the `rotationPolicy` default to Always)

## 4. External Secrets 0.10.7 → 2.13.0

- [x] Every ExternalSecret and the ClusterSecretStore on `external-secrets.io/v1`
- [x] One minor at a time, 28 hops. The ESO app syncs with server-side apply.
  At 0.16.2 the old conversion webhook blocked the new CRDs; setting
  `spec.conversion` to `None` by hand on the three CRDs fixed it
- [x] Once every object is stored as v1, drop v1beta1 from the CRDs'
  `status.storedVersions` (needed before ESO removes v1beta1 entirely).
  Done 2026-10-23: the 20 ExternalSecrets and the ClusterSecretStore
  rewritten (checked in kine that the latest revision of each is v1), then
  `storedVersions: ["v1"]` on externalsecrets, clusterexternalsecrets,
  secretstores and clustersecretstores. Two old v1beta1 managedFields
  entries on vault-backend removed. Pitfall: `kubectl get -o json | kubectl
  replace -f -` rewrites an existing `last-applied-configuration` with the
  whole object (resourceVersion, uid, finalizers), after which Argo CD's
  apply failed on vault-folder ("resourceVersion must be specified") and
  would have stripped ESO's finalizer. Removing the annotation let Argo CD
  write a clean one

## 5. Argo CD 2.14.2 → 3.4.12

- [x] v3.0.23 → v3.1.16 → v3.2.12 → v3.3.18 → v3.4.12. Label tracking is
  pinned in `argocd-cm`; our own `resource.exclusions` replaces the 3.0
  defaults, so the home-assistant EndpointSlice stays managed.
  `argocd-folder` syncs with server-side apply (needed from 3.3)
- [x] Switch to annotation tracking. Done 2026-10-24: the pin removed
  from `argocd-cm`, then every app synced once without prune. A sync
  started by patching `.operation` doesn't inherit the app's syncOptions,
  so the four server-side-apply apps fell back to client-side apply and
  failed on their big CRDs (annotation over 256 KB); passing the app's
  syncOptions in the operation fixed it. Six Certificates still carried
  the `app.kubernetes.io/instance` label that cert-manager's ingress-shim
  had copied over, owned by ingress-shim, which kept argocd-folder
  OutOfSync; removed

## 6. Calico 3.25.1 → 3.29.3

- [x] Applied microk8s 1.33's manifest (canonical/microk8s branch 1.33,
  `upgrade-scripts/000-switch-to-calico/resources/calico.yaml`) with
  `kubectl apply --server-side --force-conflicts`. Same calico-node settings
  as before. The manifest is also in
  `/var/snap/microk8s/current/args/cni-network/cni.yaml`; the 3.25.1 one is
  saved as `~/cni-3.25.1-backup.yaml` on opti1

## 7. microk8s 1.32 → 1.33

- [x] First fix the k8s-dqlite write stalls, see below
- [x] Fresh Velero backup (`pre-microk8s-1-33-202610230712`) and
  `sudo microk8s dbctl backup` (`/root/dqlite-pre-1.33-*`)
- [x] `sudo snap refresh microk8s --channel=1.33/stable`: v1.33.13, snap
  revision 9067, done 2026-10-23
- [x] `sudo microk8s addons repo update core`: branch 1.33, ingress-nginx
  v1.13.8
- [x] If kubelite panics with "Missing actuated resource record"
  ([#5057](https://github.com/canonical/microk8s/issues/5057)):
  `sudo snap revert microk8s`. No panic. The message does show up in the log
  without consequences, and kubelite has not restarted since the refresh.
  The dqlite snapshot tuning survived the refresh.
- [x] Argo CD 3.x puts pre-delete finalizers on helm-longhorn, so they are
  now in git as well

## 8. After 1.33

- [x] cert-manager 1.21.2. The chart dropped the unused tokenrequest Role and
  RoleBinding; Argo CD pruned them
- [x] Envoy Gateway v1.9 instead of ingress-nginx, Longhorn behind authentik
  OIDC
  - [x] Envoy Gateway v1.9.2 next to nginx. Gateway `homelab` in
    `envoy-gateway-system`, Envoy on host ports 10080/10443 (`gateway/`)
  - [x] One app (gatus, s2.rubenhensen.nl) on an HTTPRoute, reusing its
    existing certificate through a ReferenceGrant; test with
    `curl --resolve <host>:10443:192.168.1.204`
  - [x] Longhorn behind authentik OIDC with a SecurityPolicy, client secret
    in Vault at `kv/longhorn-oidc`. Two snags: authentik only picks up a
    new blueprint file at its hourly discovery (or
    `blueprints_discovery.send()` in `ak shell`), and Envoy Gateway does
    not retry a failed OIDC discovery until something changes, so the
    controller needed a restart. While the policy is invalid Envoy answers
    500, so Longhorn stays closed
  - [x] The other apps. Envoy cuts requests off after 15s by default, so the
    routes set `timeouts.request: 0s`. All hosts answer the same through
    Envoy as through nginx; WordPress only redirects on the test port
    because the Host header carries `:10443`
  - [x] Argo CD on `server.insecure`, so both nginx and Envoy talk plain
    HTTP to argocd-server
  - [x] cert-manager `config.gatewayAPI.enabled`
  - [x] Switch over: `microk8s disable ingress`, Envoy on 80/443, and a
    `gatewayHTTPRoute` HTTP-01 solver on the ClusterIssuer. About a minute
    without HTTP(S) on 2026-10-23; node001's tunnel needed no change
  - [x] Replace the Ingress objects with Certificate resources (the
    ingress-shim Certificates go with their Ingress, and gateway-shim
    can't write to the app namespaces) and drop the nginx annotations.
    Same names and specs, so nothing was reissued except immich, which
    dropped mbg.hensen.io; that reissue went through the Gateway's HTTP-01
    solver. The Ingresses were pruned with `PrunePropagationPolicy=orphan`
    so the Certificates stayed. Gatus now expects a 302 from Longhorn
  - [x] Retries (`gateway/retry-policy.yaml`, 2026-10-24). When a backend
    closes a kept-alive connection just as Envoy sends a request over it,
    Envoy answers 503 (flag UC) and, unlike nginx, does not retry. Behind
    sol-auth's Apache, which recycled its processes after 30 connections,
    that cost immich chunks during page loads ("error loading dynamically
    imported module"). A BackendTrafficPolicy on the Gateway retries twice
    on connect failures and resets

## 9. immich: PostgreSQL 14 with pgvecto.rs → 17 with VectorChord

PostgreSQL 14 reaches end of life on 2026-11-12, and immich 3.0 drops
pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:

- [x] Safety net: Longhorn snapshot and NAS backup `immich-pg14-pre-vchord`
  of the old volume, plus `pg_dump -Fc` (447 MB) in
  `~/immich-pg14-pre-vchord.dump` on opti1
- [x] immich server to 0 replicas (through git)
- [x] On the old cluster, immich's manual migration in one transaction:
  drop `clip_index` and `face_index`, `embedding` columns to `real[]`,
  `DROP EXTENSION vectors`, `DROP SCHEMA vectors`, reset the database's
  `search_path` (it included `vectors`)
- [x] New cluster `immich-postgres-17` (`immich/postgresql-cluster-17.yaml`):
  `cloudnative-vectorchord:17.11-1.1.1` (pgvector 0.8.6, VectorChord
  1.1.1), filled by a CNPG microservice import, then `vchord` created and
  the columns back to `vector(512)`. An in-place `pg_upgrade` wasn't
  possible: the pgvecto.rs image is Debian bullseye, the VectorChord ones
  bookworm. Tables, indexes and row counts matched the old database
- [x] immich pointed at `immich-postgres-17-rw` and its own app secret.
  v2.4.1 refused to start (it only accepts VectorChord <0.6); v2.5.0 and up
  accept <2, so immich went to v2.7.5, the last 2.x. It rebuilt both
  indexes as `vchordrq` in about 1.5 minutes
- [x] After a few days without problems: delete the old cluster
  (`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump.
  Done 2026-10-28 (PR #61), with the PG14 volume's Longhorn backups.
  Rolling back means reverting the old volume to the snapshot, since its
  embeddings are now `real[]` without pgvecto.rs
- [x] immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on
  2026-10-24, about three minutes of downtime (two of them pulling the
  image). Charts after 0.12.0 are only in the OCI registry
  (`ghcr.io/immich-app/immich-charts`), and since 0.13.0 the values
  schema rejects size, accessMode and storageClass on emptyDir volumes.
  None of the env variables v3 removed were set; the ML image needs
  x86-64-v2, and the i5-8600T is v3. The migrations took 16s, both vector
  indexes stayed, and `immich-admin schema-check` reports no drift (the
  geodata_places_pkey warning at startup came from the geodata import
  running at the same time). Safety net, since immich can't be
  downgraded: Longhorn snapshot `immich-pg17-pre-v3` and
  `~/immich-pg17-pre-v3.dump` (440 MB) on opti1
- [ ] Optional: `MACHINE_LEARNING_MODEL_REVISION=v2` on the ML container
  (v3.3.0), for faster ML with less memory; numerically the same models,
  so no jobs to rerun. The model cache is an emptyDir, so the one-off
  preparation repeats on every restart
- [x] After a few days without problems: delete the `immich-pg17-pre-v3`
  snapshot and dump together with the PG14 leftovers above (2026-10-28)

## k8s-dqlite write stalls (found 2026-10-22)

Lease updates regularly take more than 10s (169 of ~162k since the API
server started), so controllers using leader election keep losing it and
restarting: snapshot-controller over 1000 times, the CNPG operator,
Longhorn's CSI sidecars.

Cause: the SQLite file behind dqlite is ~709 MB (173k pages of 4 KB) but
164k pages are free. The live data is ~35 MB: 3,700 kine rows for 3,006
keys, so compaction works. SQLite never returns free pages without VACUUM,
and dqlite writes the whole file into a snapshot (~150 MB compressed)
every 1,024 commits, i.e. every few minutes. Each snapshot takes about 6s
of writing on the Apacer AS350 (DRAM-less SATA SSD, which also holds every
Longhorn volume). Commits queue behind it. Outside snapshots fsync takes
about 1.2 ms.

Constraints:
- microk8s 1.32 and 1.33 ship k8s-dqlite v1.3.5 (dqlite 1.17 LTS). VACUUM
  needs dqlite 1.18.2+ (microk8s 1.34 and up). `dbctl restore` writes into the
  running DB, so it doesn't shrink anything.
- microk8s 1.34+ ships k8s-dqlite 1.8.x with a reworked write path; that is
  where Canonical fixed this class of problem (canonical/microk8s#5153).

Plan (needs sudo; the API is down for the length of a dqlite restart,
workloads keep running). `~/dqlite-tuning.sh` and `~/tuning.yaml` on opti1
do the first three steps: `ssh -t ruben@192.168.1.204 ./dqlite-tuning.sh`
- [x] `sudo microk8s dbctl backup -o /root/dqlite-pre-tuning-<date>`
- [x] Install `backend/tuning.yaml` with
  `snapshot: {threshold: 8192, trailing: 16384}`: a snapshot every ~25
  minutes instead of every ~3. Never set trailing alone (threshold then
  becomes 0). Leaving out the kine intervals keeps their defaults. Costs
  ~0.5 GB more raft log on disk.
- [x] `sudo snap restart microk8s.daemon-k8s-dqlite`, then check snapshot
  spacing in the backend dir and that lease updates stay under 10s.
  Measured over the first 40 minutes: snapshots 24.7 minutes apart
  (were ~3), 0 of 9,445 lease updates over 4s, no controller restarts
- [x] Optional: Longhorn backupstore poll interval 300s -> 1800s
  (backupvolume updates are ~8% of all writes). Done 2026-10-24
- Later: microk8s 1.34+ for the new write path and VACUUM, see
  "On to 1.36" below.

`--cgroups-per-qos=false`, the likely trigger of the 1.32->1.33 kubelite
panic (#5057), is not set in `args/kubelet`.

### Still stalling after the VACUUM (2026-10-24)

The smaller database fixed the snapshot stalls, but controllers kept
losing their leases a few times a day: since the API server started,
256 of ~278k lease updates took over 5s and 44 over 10s. dqlite's log
shows short complete stops (at 10:00:42 even its own health query hit
its deadline). Every lease loss that day lined up with minutes of heavy
writes to the one SSD, which holds root, dqlite, every Longhorn replica,
image layers and emptyDirs:

| Lease losses | Writing at the time |
|---|---|
| 00:02 | Longhorn backups of every volume (from 00:00) |
| 01:09 | Velero backup (kopia reading and caching 133 GB of photos) |
| 09:22 | full kopia maintenance of mbgwp (6 min) and argocd (11 min) |
| 10:00:43 | full kopia maintenance of vault (9 min) |
| 10:31, 10:38 | immich upgrade: image pulls, pg_dump, migrations |

Velero ran maintenance every hour for each of the 15 kopia repositories
(15 pods an hour), without resource limits and with the kopia cache in
an emptyDir; about once a day a run is a full maintenance of minutes, at
whatever time of day the hourly schedule had drifted to.

- [x] Velero: `defaultRepoMaintainFrequency: 24h`, the 15 existing
  BackupRepositories patched to 24h by hand, and maintenance jobs capped
  at 500m CPU (2Gi memory)
- [x] Leases of 60s/40s/5s instead of 15s/10s/2s (duration, renew deadline,
  retry period) for the snapshot-controller (kustomize patch in its Argo
  CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway
  (`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and
  global manager have no setting for it
- [x] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
  for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
  numbers on the disk latency during tonight's backups. Read on 2026-10-25
  at 02:35 UTC, per 10 minutes:
  - Quiet: ~50 writes/s, 0.5-2 ms per write on average, "full" IO
    pressure under 15%
  - 14:30-23:15: 250-440 writes/s and 200-400 reads/s, 6-20 ms per write,
    pressure 45-75%. Longhorn merging immich-claim's deleted snapshots
    (docs/longhorn-second-ssd.md)
  - Longhorn backups 00:00-00:15: up to 1,220 reads/s, pressure 69%; the
    Velero backup at 01:00 was incremental and short (pressure 29%)
  - The authentik and Vault upgrades 01:20-01:40 (image pulls,
    migrations): 19-33 ms per write, single 2s windows near 2 s per write,
    pressure 78%. The CSI sidecars and Longhorn's global manager lost
    their leases at 01:31
  Per minute against dqlite's own errors (2026-10-24 12:35 to 2026-10-25
  11:30, leaving out the CNPG range cancels): all 67 minutes with errors
  had sda ~100% busy, single 2s windows at 200 ms to 2.2 s per write and
  "full" pressure at 45-78%, against a median minute peak of 24 ms. The
  worst minute after the merges was the rollout of eight merged PRs at
  2026-10-25 11:25-11:27 (77 errors in one minute). The snapshot-controller,
  CNPG and Envoy Gateway held their 60s leases through all of it
  Ubuntu's sysstat already keeps per-device history every 10 minutes
  without a script: `sar -d -f /var/log/sysstat/sa24` showed sda at
  ~26 MB/s read and ~27 MB/s written, 9 ms await, during the merging
- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
  i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
  present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with
  DRAM for Longhorn's data (replicas move over live by evicting the old
  disk) would leave the SATA SSD to dqlite and the system

## On to 1.36

microk8s goes up one minor per refresh. 1.34 brings k8s-dqlite 1.8.x with
the reworked write path and dqlite 1.18.2, which can VACUUM the mostly
empty database. Before every hop: fresh Velero backup, no
deprecated APIs in use (`apiserver_requested_deprecated_apis` metric
empty), and the Calico manifest of the target version (canonical/microk8s
branch `<version>`, `upgrade-scripts/000-switch-to-calico/resources/calico.yaml`).

Findings from 2026-10-23:
- microk8s 1.34, 1.35 and 1.36 all ship k8s-dqlite v1.8.1, which is built
  against dqlite v1.18.2 (1.33: v1.3.5 with dqlite v1.17.1). So VACUUM is
  possible from 1.34 on; 1.35 and 1.36 are for staying in support
  (Kubernetes 1.34 reaches end of life on 2026-10-27).
- Calico is v3.29.3 in all four branches, so no CNI change.
- containerd stays 1.7.29 on 1.34 but jumps to 2.1 on 1.35 and 2.2 on 1.36.
  Check `args/containerd-template.toml` before 1.35: it still has the old
  `pause:3.7` sandbox image.
- `backend/tuning.yaml` still works in k8s-dqlite 1.8.1 (defaults there:
  threshold 512, trailing 4096, dynamic strategy).
- The 1.34 configure hook adds
  `ListFromCacheSnapshot=false,SizeBasedListCostEstimate=false,DetectCacheInconsistency=false`
  to the existing `--feature-gates=WatchList=false` line for kine
  clusters. WatchList is still beta in 1.34, so it can stay off.
- `certs/front-proxy-client.crt` expired on 2025-09-13 and microk8s never
  renews it (canonical/microk8s#5576). Nothing uses the aggregation layer
  here yet, but it gets renewed with the refresh.

### microk8s 1.34
- [x] Read canonical/microk8s#5568: 1.34 locks ConsistentListFromCache on,
  and k8s-dqlite's watch progress notifications lag under load, so LISTs
  can wait up to 3s before falling back to storage. Latency, not
  correctness; worst on busy HA clusters. Still open
- [x] Velero backup: the full `pre-microk8s-1-33-202610230712` from that
  morning; nothing that isn't in git changed since
- [x] `ssh -t ruben@192.168.1.204 ./microk8s-1.34.sh`: dbctl backup, an
  offline copy of the backend (k8s-dqlite 1.8 migrates the schema on first
  start, so 1.33 can't read it afterwards), `snap refresh` to 1.34/stable,
  the addons repo, and the front-proxy certificate. Done 2026-10-23:
  v1.34.9, snap revision 9063. Backups in `/root/backend-1.33-20261023-1938.tar`
  and `/root/dqlite-pre-1.34-20261023-1938.tar.gz`. The tuning.yaml
  snapshot settings carried over; front-proxy-client.crt now runs to 2027
- [x] Afterwards: LIST latency (`apiserver_watch_cache_read_wait_seconds`),
  lease update times and controller restarts. LISTs as fast as on 1.33
  (92-234 ms), no watch-cache wait over 0.1s and no storage fallbacks, so
  #5568 doesn't show on this single node. The CNPG operator lost its lease
  during the refresh and restarted once, as expected

### Components for 1.35 and 1.36
- [x] Longhorn 1.11.2 → 1.12.1 → 1.13.0 (one minor at a time, now allowed
  on 1.34); Renovate hold lifted. A system backup before each step
  (`pre-upgrade-v1-11-2-20261023`, `pre-upgrade-v1-12-1-20261023`); all
  19 volumes moved to the new engine image within ten minutes per step,
  and the unused engine images were deleted. Kept off during the
  upgrades: 1.12's internal NetworkPolicies (Calico enforces them, and the
  1.12 instance-manager policy only admits Longhorn pods while iSCSI comes
  from the host). Turned on afterwards on 1.13.0, which opens 3260 to any
  source: a pod outside Longhorn can no longer reach the longhorn-manager
  API (it could before), the UI still can, and a new volume attached over
  iSCSI, took writes and backed up to the NAS. Also kept off: 1.13's
  cluster-wide Secret read for the CSI controller (no StorageClass uses
  provisioner secrets). The global manager runs one replica. Checked on
  1.13.0: a manual backup to the NAS, provisioning a new PVC, and the UI
  behind authentik
- [x] Argo CD 3.4 → 3.5.6 (3.4 is tested up to 1.35, 3.5 on 1.32-1.36).
  3.5 renders with Helm 4.2.1 instead of 3.19; all thirteen Helm apps
  rendered the same with both beforehand (`alpine/helm` images, live app
  inputs), and after the upgrade all 35 apps stayed Synced/Healthy without
  a sync
- [x] Velero 1.16 → 1.17.1 (chart 11.4.0, AWS plugin v1.13.2) → 1.18.2
  (chart 12.2.1, AWS plugin v1.14.4). Every pod volume backup and
  repository was kopia already. 1.17 moved `--keep-latest-maintenance-jobs`
  into the `velero-repo-maintenance` ConfigMap, which the chart renders.
  A gatus test backup completed on both versions
- [x] Recheck CNPG, External Secrets, the Vault chart and Envoy Gateway
  against 1.36. Done 2026-10-23: CNPG 1.30 supports 1.34-1.36, ESO 2.13 up to
  1.36, Envoy Gateway 1.9 1.33-1.36, the Vault chart >= 1.20

### microk8s 1.35 and 1.36

Checked for 1.35 on 2026-10-23: default args and the configure hook are
the same as 1.34's; the four feature gates set to false (WatchList and the
three kine ones) stay beta through 1.36, so the API server still starts;
the containerd template is the same apart from our older `pause:3.7`, which
containerd 2.1 accepts. Every component supports 1.35 (Velero 1.16 only
"expected"). `args/certs.d/docker.io/hosts.toml` had a stray `y` after the
bitnamilegacy rewrite, so containerd logged "failed to decode hosts.toml"
on every docker.io pull and fell back to defaults; fixed on opti1, the
original is in `~/hosts.toml.docker.io.bak`.

- [x] `ssh -t ruben@192.168.1.204 ./microk8s-1.35.sh`: dbctl backup, `snap
  refresh` to 1.35/stable, addons repo (no schema migration, so `snap
  revert` is the way back). Done 2026-10-23: v1.35.6, snap revision 9072,
  containerd 2.1.6. containerd was down for 26s and logged no config
  warnings and no hosts.toml errors; all workloads, apps, certificates,
  volumes and both status pages green; LISTs 92-166 ms, no slow lease
  updates
Checked for 1.36 on 2026-10-23: default args, containerd template and
configure hook unchanged from 1.35; containerd 2.2 only drops registry
`mirrors`/`configs`/`auths` (in 2.3), and the template uses `config_path`;
every component supports 1.36; only the core/v1 Endpoints API is still in
use as a deprecated API.

The microk8s stable channels lag upstream: 1.35/stable is v1.35.6 and
1.36/stable v1.36.2 (both from early July), while upstream is at v1.35.9
and v1.36.5. Only 1.36/edge has CVE-2026-2270 fixed (StatefulSet
controller, cross-namespace pod creation; fixed in 1.35.9/1.36.5). Chosen:
1.36/stable with the snap held, so updates only happen on purpose
(`sudo snap refresh microk8s` once stable passes v1.36.5). Before the hold,
snapd refreshed microk8s within its channel four times a day.

- [x] `ssh -t ruben@192.168.1.204 ./microk8s-1.36.sh`: dbctl backup, `snap
  refresh` to 1.36/stable, addons repo, `snap refresh --hold microk8s`.
  Done 2026-10-23: v1.36.2, snap revision 9092, containerd 2.2.3, snap
  held. All workloads, apps, certificates, volumes and both status pages
  green; LISTs 92-166 ms, no slow lease updates
- [x] CVE-2026-2270: moved to 1.36/edge (v1.36.5, snap revision 9574) on
  2026-10-23 with `~/microk8s-1.36.5.sh`, still held. The edge build has
  the same containerd 2.2.3, runc 1.4.2 and k8s-dqlite 1.8.1 as the
  stable v1.36.2; only build and test fixes landed on the branch since.
  Everything green afterwards; one Velero maintenance job was cut off by
  the restart and runs again at the next interval
- [ ] Back to 1.36/stable once it is at v1.36.5 or later
  (`sudo snap refresh microk8s --channel=1.36/stable`); before that, the
  switch would install the older v1.36.2

### Shrink the database

dqlite accepts VACUUM since v1.18.2 (canonical/dqlite#734). Its maintainer:
"it potentially create a very big transaction which has effects both on the
RAFT protocol *and* on resource usage. I would recommend not to run it
unsupervised." The dqlite shell that ships with microk8s can't send it: it
wraps every statement in a transaction, and SQLite refuses to VACUUM inside
one. `tools/dqlite-vacuum` sends it through the go-dqlite driver as a plain
statement, as user `ruben` (in the `microk8s` group, which can read
`cluster.crt`, `cluster.key` and `info.yaml`); without `-vacuum` it only
prints page statistics.

Rehearsed on 2026-10-23 on opti1 with the k8s-dqlite and dqlite binaries
of microk8s 1.34, on a throwaway node in `/dev/shm` (not in Docker on the
Mac: the amd64 emulation lacks `io_setup`, which raft needs): 657 MB with
616 MB free went to 41 MB with none free in 0.6s; a writer next to it saw
no failures and no write over 1s; after a restart, and after a restart from
the first snapshot (40 MB), `integrity_check` was ok and all rows were
there.

Live database before: 709 MB, 666 MB free, ~4,400 kine rows; snapshots of
~150 MB every ~23 minutes; k8s-dqlite RSS 1,346 MB.

- [x] `ssh -t ruben@192.168.1.204 ./dqlite-vacuum.sh`: dbctl backup, an
  offline copy of the backend (API down ~1 min), VACUUM, integrity check.
  How to go back is at the top of the script. Done 2026-10-23 20:10 UTC:
  709 MB with 666 MB free became 44 MB with none free; integrity_check ok.
  Backups in `/root/backend-pre-vacuum-20261023-*.tar` and
  `/root/dqlite-pre-vacuum-20261023-*.tar.gz`
- [x] Afterwards: the first snapshot (20:21) is 8.3 MB instead of ~150 MB,
  and k8s-dqlite's memory went from 1,346 MB to 403 MB. The seven
  leader-elected controllers (snapshot-controller, Longhorn's CSI sidecars,
  CNPG, Envoy Gateway) restarted once while dqlite was stopped for the
  offline copy, not during the VACUUM