Commit Graph
1369 Commits
Author SHA1 Message Date
Ruben Hensen 6517c21937 mbgwp: remove the unused wordpress-external NodePort
It selected pods of the old mbgwp-helm release, so since the move to the official chart it had no endpoints and only held node port 30989 open.
2026-10-24 00:08:19 +02:00
Ruben Hensen 8357d31f9c docs: record the Longhorn NetworkPolicies and poll interval 2026-10-24 00:05:44 +02:00
Ruben Hensen 5208379310 longhorn: enable the internal NetworkPolicies
With them only Longhorn's own components can reach the longhorn-manager API, which any pod could otherwise call without authentication to manage volumes. 1.12 was kept off because its instance-manager policy left the host-side iSCSI initiator out; 1.13 opens port 3260 to any source while v1DataEngineInitiatorSourceCIDRs is empty.
2026-10-24 00:02:32 +02:00
Ruben Hensen f2936e69ec docs: record the switch to annotation tracking 2026-10-24 00:01:16 +02:00
Ruben Hensen 7c8cd28f71 argocd: track resources by annotation
Drops the label tracking pin kept through the 3.x upgrades, so Argo CD uses its default argocd.argoproj.io/tracking-id annotation. Labels can be copied by other tools (cert-manager's ingress-shim copied app.kubernetes.io/instance onto Certificates); the annotation names the exact resource. Every app gets synced once right after, so resources removed later are still recognised for pruning.
2026-10-23 23:54:06 +02:00
Ruben Hensen f901beea4c docs: record the ESO storedVersions cleanup 2026-10-23 23:53:16 +02:00
Ruben Hensen 04a2a2aed2 gatus: let Argo CD prune old resources
configMapGenerator gives every config change a new gatus-config-<hash> ConfigMap, and without prune the old one stayed behind and left the app OutOfSync.
2026-10-23 23:42:58 +02:00
Ruben Hensen 947a705016 longhorn: poll the backup store every 30 minutes
Each poll updates the backupvolume objects, which made up about 8% of the k8s-dqlite writes at the 300s interval. The recurring backup runs once a night, so 30 minutes is plenty.
2026-10-23 23:39:49 +02:00
Ruben Hensen 416356b4d5 Remove the disabled apps and a stray Argo CD patch copy
Nothing deploys disable/: the apps in it (ente-photos, minio, mist, owntracks, traccar, ttrss, whisper and others) are gone from the cluster apart from some old TLS secrets. git history keeps them. The argocd-cm patch copy was an old leftover that kustomize never used.
2026-10-23 23:39:49 +02:00
Ruben Hensen d747530f52 docs: tick off the 1.36 component recheck 2026-10-23 23:37:17 +02:00
Ruben Hensen f0519d1b70 docs: record the CVE-2026-2270 patch
microk8s now runs 1.36/edge (v1.36.5) and stays held; it goes back to 1.36/stable once that channel catches up.
2026-10-23 23:35:32 +02:00
Ruben Hensen acc3d7eabb docs: tick off microk8s 1.36
v1.36.2 with containerd 2.2.3, and the snap is held. The CVE-2026-2270 patch release is still to come to the stable channel.
2026-10-23 23:24:06 +02:00
Ruben Hensen 96870af92f docs: prepare microk8s 1.36
Nothing changes in microk8s between 1.35 and 1.36 for this setup. The stable channels lag upstream patches, including the CVE-2026-2270 fix, so the snap gets held and patch updates become a deliberate step.
2026-10-23 23:18:51 +02:00
Ruben Hensen a0fcf9ec3f docs: tick off Longhorn 1.13 2026-10-23 23:15:04 +02:00
Ruben Hensen b19c776728 longhorn: upgrade to 1.13.0
1.13 needs Kubernetes 1.34 or later. It adds a longhorn-global-manager Deployment, set to one replica on this single node, and by default lets the CSI controller read every Secret for provisioner-secret StorageClass parameters; neither StorageClass uses those, so that grant stays off.
2026-10-23 23:06:08 +02:00
Ruben Hensen 514fcd8dd4 longhorn: upgrade to 1.12.1
First step from 1.11.2 to 1.13 now that the cluster runs Kubernetes 1.35. The internal NetworkPolicies that 1.12 enables by default stay off: Calico would enforce them, and the instance-manager policy only admits Longhorn pods while the iSCSI initiator connects from the host.

Also lifts the Renovate hold below 1.11.3, which only existed for Kubernetes 1.33.
2026-10-23 22:58:29 +02:00
Ruben Hensen bd8643c00b docs: tick off Velero 1.18 2026-10-23 22:48:57 +02:00
Ruben Hensen 1926ccc42a velero: upgrade to 1.18.2 (chart 12.2.1)
No breaking changes for this setup: the chart renders the same resources and server flags, and chart 12 only dropped the Bitnami kubectl image from the CRD hook, which stays off because Argo CD applies the CRDs. The AWS plugin goes to v1.14.4, the line for Velero 1.18.
2026-10-23 22:47:27 +02:00
Ruben Hensen e3074b43ad velero: upgrade to 1.17.1 (chart 11.4.0)
1.17 drops restic, which nothing here uses (all 411 pod volume backups and 20 repositories are kopia), and moves the maintenance job settings from server flags into the velero-repo-maintenance ConfigMap, which the chart now renders with keepLatestMaintenanceJobs: 3. The AWS plugin goes to v1.13.2, the line for Velero 1.17.
2026-10-23 22:45:00 +02:00
Ruben Hensen 04ac64f956 docs: tick off Argo CD 3.5 2026-10-23 22:42:00 +02:00
Ruben Hensen e490c79e78 argocd: upgrade to v3.5.6
3.5 is tested on Kubernetes 1.32 to 1.36; 3.4 stops at 1.35. It renders Helm charts with Helm 4.2.1 instead of 3.19; rendering all thirteen Helm apps with both gave the same manifests apart from whitespace, comments and document order.
2026-10-23 22:35:00 +02:00
Ruben Hensen e95d78e160 docs: tick off microk8s 1.35
v1.35.6 with containerd 2.1.6. containerd converted the old config without warnings, and the fixed hosts.toml now parses.
2026-10-23 22:29:32 +02:00
Ruben Hensen b60d3b5589 docs: record the VACUUM and prepare microk8s 1.35
The database went from 709 MB to 44 MB, the first snapshot after it is 8.3 MB instead of 150 MB, and k8s-dqlite uses 403 MB instead of 1,346 MB.
2026-10-23 22:21:46 +02:00
Ruben Hensen 175769d36c Add a tool to VACUUM the k8s-dqlite database
dqlite supports VACUUM from v1.18.2, which microk8s 1.34 ships, but the dqlite shell wraps every statement in a transaction and SQLite refuses to VACUUM inside one. dqlite-vacuum sends it through the go-dqlite driver instead and otherwise just prints page statistics. dqlite-vacuum.sh makes the backups first.

Rehearsed on opti1 on a throwaway node in /dev/shm: 657 MB to 41 MB in 0.6s, with concurrent writes unaffected and the data intact after restarts.
2026-10-23 22:03:30 +02:00
Ruben Hensen c55b7ceab5 docs: tick off microk8s 1.34
v1.34.9 runs with the dqlite snapshot tuning intact. LIST latency is unchanged from 1.33, so the ConsistentListFromCache stalls of canonical/microk8s#5568 don't show here.
2026-10-23 21:42:44 +02:00
Ruben Hensen 67abb0b972 docs: prepare the move to microk8s 1.34
k8s-dqlite v1.8.1 in microk8s 1.34 already bundles dqlite 1.18.2, so the database can be vacuumed from 1.34 on. Also notes the containerd 2 jump at 1.35, the feature gates the 1.34 hook adds for kine, and the front-proxy certificate that expired in 2025.
2026-10-23 21:35:16 +02:00
Ruben Hensen a07ef664cb docs: tick off the Envoy Gateway migration 2026-10-23 21:29:37 +02:00
Ruben Hensen 0a810817b6 gatus: expect the authentik redirect for Longhorn
Longhorn now sits behind an authentik login on the Gateway instead of nginx basic auth, so the check expects a 302 without following it. Same change for s1 on node001.
2026-10-23 21:24:00 +02:00
Ruben Hensen 1f04831a97 Replace the Ingresses with Certificate resources
ingress-nginx is gone and every app is served by the Envoy Gateway. The certificates were created by cert-manager's ingress-shim and are owned by the Ingresses, so each app now declares its Certificate itself, with the same name, secret and usages, so nothing gets reissued. gateway-shim can't take over because the certificates live in the app namespaces.

immich drops mbg.hensen.io, which has no route, so that one certificate is reissued through the Gateway's HTTP-01 solver.

Also removes the Longhorn test-port redirect URI and updates comments that still described nginx.
2026-10-23 21:20:25 +02:00
Ruben Hensen 48f5c7d894 docs: tick off the switch to Envoy Gateway and the Longhorn login 2026-10-23 21:17:04 +02:00
Ruben Hensen 910b792452 longhorn: log in through authentik on the Gateway
Longhorn has no users of its own; nginx guarded it with basic auth using apr1 hashes, which Envoy Gateway does not support. An Envoy Gateway SecurityPolicy now sends visitors through an authentik OIDC application that only admins may use. The client secret lives in Vault at kv/longhorn-oidc and reaches both authentik and Envoy through External Secrets.
2026-10-23 21:07:31 +02:00
Ruben Hensen 009b8ce510 gateway: take over ports 80 and 443 from ingress-nginx
Envoy moves from the test ports to 80 and 443, and Let's Encrypt HTTP-01 challenges go through an HTTPRoute on the Gateway. Push only after sudo microk8s disable ingress, since both want the same host ports.
2026-10-23 21:02:52 +02:00
Ruben Hensen 23aaf5ab26 docs: tick off the routes and note what the switch-over needs 2026-10-23 21:02:10 +02:00
Ruben Hensen b360483210 cert-manager: enable the Gateway API integration
Needed for HTTP-01 challenges through the Gateway after the switch from ingress-nginx. Nothing changes until an issuer uses a gatewayHTTPRoute solver.
2026-10-23 21:01:17 +02:00
Ruben Hensen 134774509d argocd: serve plain HTTP behind the ingress and the Gateway
With server.insecure, TLS ends at nginx or Envoy and argocd-server serves plain HTTP, which is how Argo CD documents the Gateway API setup. The nginx Ingress now talks HTTP to the http port; the ssl-passthrough annotation did nothing without the controller flag. argocd-folder diffs server-side for the new HTTPRoute.
2026-10-23 17:44:08 +02:00
Ruben Hensen 69a0661d09 Add HTTPRoutes for the apps behind ingress-nginx
Each app gets an HTTPS listener on the Gateway with its existing certificate, an HTTPRoute, and a ReferenceGrant for that certificate. The routes drop Envoy's 15s request timeout, as nginx has no overall limit either. Argo CD and Longhorn follow separately; the Ingresses stay until the switch-over.
2026-10-23 17:39:04 +02:00
Ruben Hensen d1efb91a4b gateway: diff the Gateway resources server-side
The API server fills in Gateway API defaults such as the parentRef group and kind and a PathPrefix / match, so Argo CD's client-side diff kept gateway-folder OutOfSync.
2026-10-23 17:36:14 +02:00
Ruben Hensen f4377616e6 gatus: serve s2.rubenhensen.nl through the Gateway as well
First app on the Gateway API: an HTTPS listener for s2.rubenhensen.nl using the existing certificate in the gatus namespace, an HTTPRoute, and a ReferenceGrant for the certificate. Plain HTTP on the Gateway now redirects to HTTPS. The Ingress stays until the switch-over.
2026-10-23 17:30:42 +02:00
Ruben Hensen 8e508080d7 docs: plan the Envoy Gateway phases
Phase 1 is done: Envoy Gateway runs next to ingress-nginx on test ports.
2026-10-23 17:27:49 +02:00
Ruben Hensen 7475199ca5 gateway: add the Envoy GatewayClass and Gateway
Envoy runs as a DaemonSet on host ports 10080 and 10443 for now, so it can be tested next to ingress-nginx, which keeps 80 and 443. The Gateway has a plain HTTP listener and no routes yet.
2026-10-23 17:27:02 +02:00
Ruben Hensen a20808b752 envoy-gateway: install v1.9.2 next to ingress-nginx
First step of the move from the retired ingress-nginx to the Gateway API. This installs the controller with the Gateway API and Envoy Gateway CRDs. No Gateway exists yet, so no traffic changes.
2026-10-23 17:25:20 +02:00
Ruben Hensen 3a250b65cc docs: tick off cert-manager 1.21.2 2026-10-23 17:22:22 +02:00
Ruben Hensen 43a62ff48f cert-manager: upgrade to v1.21.2
1.21 supports Kubernetes 1.33 to 1.36. The chart no longer creates the tokenrequest Role and RoleBinding; nothing uses them, as the only issuer is ACME with HTTP-01 and no serviceAccountRef.
2026-10-23 17:20:15 +02:00
Ruben Hensen 2c4164ad22 velero: skip emptyDir volumes in scheduled backups
The nightly backup of 2026-10-23 ended PartiallyFailed because a repository maintenance job pod finished while Velero was reading its emptyDir volumes. emptyDir holds nothing worth restoring, so a volume policy now skips it for both schedules.
2026-10-23 17:12:12 +02:00
Ruben Hensen 336c37ca50 docs: tick off microk8s 1.33
The node runs v1.33.13 (snap revision 9067). kubelite logs the #5057 message but has not panicked or restarted, and the dqlite snapshot tuning carried over.
2026-10-23 17:06:27 +02:00
Ruben Hensen 5427666ff9 longhorn: list Argo CD's pre-delete finalizers
Argo CD 3.x adds pre-delete-finalizer.argocd.argoproj.io and its /cleanup variant to Applications with PreDelete hooks, which the Longhorn chart has. With only resources-finalizer in git, the apps app-of-apps stayed OutOfSync on helm-longhorn.
2026-10-23 17:03:32 +02:00
Ruben Hensen c3830cd9d7 docs: tick off the dqlite snapshot tuning
Over the first 40 minutes after the restart: snapshots 24.7 minutes apart instead of ~3, none of 9,445 lease updates took over 4s, and neither the CNPG operator nor the snapshot-controller restarted.
2026-10-23 00:53:11 +02:00
Ruben Hensen 6a782ff174 docs: plan the route to Kubernetes 1.36
1.34 brings the reworked k8s-dqlite write path, 1.36 a dqlite that can VACUUM the mostly empty datastore. Lists the per-hop checks, the one-way schema migration at 1.34, the components that still need to move for 1.35/1.36, and the staged dqlite tuning script on opti1.
2026-10-23 00:11:27 +02:00
Ruben Hensen 6508335914 docs: diagnose the k8s-dqlite write stalls
The dqlite SQLite file is ~709 MB with 95% free pages, so every 1,024 commits dqlite writes a ~150 MB snapshot that stalls commits for seconds and makes controllers lose leader election. Plan: larger snapshot threshold in tuning.yaml now, microk8s 1.34+ for the real fix.
2026-10-23 00:04:41 +02:00
Ruben Hensen 69edd7ef4c docs: tick off Calico 3.29.3, note the dqlite latency 2026-10-22 23:49:05 +02:00