docs: the roadmap as of 2026-10-28

This commit is contained in:
Ruben Hensen committed 2026-10-28 16:01:16 +01:00
1 parent 0589bb3e08
commit a4b09fc51e
1 file changed
+47 -20
+47 -20
View File
@@ -1,22 +1,27 @@
# Roadmap # Roadmap
Where the cluster stands and what is left, as of 2026-10-25. Takes over Where the cluster stands and what is left, as of 2026-10-28. Takes over
from wishlist.txt (January 2026, removed); its items are sorted in at the from wishlist.txt (January 2026, removed); its items are sorted in at the
bottom. bottom.
## State ## State
One node: opti1, an OptiPlex 5060 with an i5-8600T, 32 GB and one 1 TB One node: opti1, an OptiPlex 5060 with an i5-8600T, 32 GB and one 1 TB
SATA SSD that is 72% full. microk8s v1.36.5 from 1.36/edge, snap held. SATA SSD that is 54% full. microk8s v1.36.5 from 1.36/edge, snap held.
35 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn 39 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn
volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the volume has one replica. authentik 2026.8.5, Vault 2.2.1, immich 3.3.5 on
NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to Garage PostgreSQL 17, MariaDB 11.4.13. Memory requests and limits on everything
(MinIO until 2026-10-26) at 01:00 (daily kept 30 days, weekly 90 days). but Longhorn, the Vault servers, calico and the LDAP outpost. Backups go
to the NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to
Garage (MinIO until 2026-10-26) at 01:00 (daily kept 30 days, weekly 90
days), and CNPG to Garage (a base backup at 03:00, the WAL every 5
minutes, 30 days).
Gatus on s1 (node001, Hetzner) and s2 (home) watch each other and alert Gatus on s1 (node001, Hetzner) and s2 (home) watch each other and alert
through ntfy; s2 also gets an hourly report from inside the cluster on the through ntfy; s2 also gets an hourly report from inside the cluster on the
backups, Vault's seal, the Argo CD apps and the CNPG clusters. backups, Vault's seal, the Argo CD apps and the CNPG clusters.
Every app namespace has NetworkPolicies, and CI renders and validates all Every namespace with pods but kube-system has NetworkPolicies, and CI
apps on every PR. node001 runs mail and tunnels the public hosts home. renders and validates all apps on every PR. node001 runs mail and tunnels
the public hosts home.
## Follow-ups from this week ## Follow-ups from this week
@@ -34,7 +39,14 @@ docs/longhorn-second-ssd.md:
`immich-pg17-pre-v3` only ever was a snapshot `immich-pg17-pre-v3` only ever was a snapshot
- [ ] The upgrade's dqlite and backend copies in `/root/` on opti1 - [ ] The upgrade's dqlite and backend copies in `/root/` on opti1
(`backend-*.tar`, `dqlite-pre-*`) (`backend-*.tar`, `dqlite-pre-*`)
- [ ] Log in to immich as a jeugdlid (quota 0) - [ ] Log in to immich as a jeugdlid (quota 0): no uploads, albums as
viewer, people searchable but not renamable
- [x] Searching by face: people are per user in immich, and a partner's
library doesn't bring its people along, so since the personal accounts
only the group account could. share-sync now puts everyone in the
group account's cluster group and shares its people (immich 3.3's
person sharing; PR #59, 2026-10-28): 4430 people, 169 named, for each
of the 6 accounts, with write access for those who may upload
- [ ] Back to 1.36/stable once it is at v1.36.5 or later (still v1.36.2 on - [ ] Back to 1.36/stable once it is at v1.36.5 or later (still v1.36.2 on
2026-10-25). The snap is held, so microk8s gets no updates at all until 2026-10-25). The snap is held, so microk8s gets no updates at all until
someone refreshes it someone refreshes it
@@ -216,7 +228,8 @@ Findings from 2026-10-25:
22:48, since the old pod waited for its connections to close. The first 22:48, since the old pod waited for its connections to close. The first
base backups: 46 MB in 29 s and 776 MB in 3.5 minutes. immich's base backups: 46 MB in 29 s and 776 MB in 3.5 minutes. immich's
`immediate` backup ran before the plugin was up and failed; a manual one `immediate` backup ran before the plugin was up and failed; a manual one
(`immich-postgres-17-first`) went through (`immich-postgres-17-first`) went through. The 03:00 base backups of
10-27 and 10-28 completed for both; the failed backup is deleted
- [ ] Restore one of them into a new Cluster (`bootstrap.recovery` from - [ ] Restore one of them into a new Cluster (`bootstrap.recovery` from
the ObjectStore, to a point in time) to see that it works. After that, the ObjectStore, to a point in time) to see that it works. After that,
decide whether Velero still needs to copy the CNPG data directories decide whether Velero still needs to copy the CNPG data directories
@@ -242,11 +255,13 @@ Findings from 2026-10-25:
MinIO still has the kopia repositories of namespaces that are gone MinIO still has the kopia repositories of namespaces that are gone
(~156 GB: nextcloud, ttrss, whisper-priv, whisper-webui, blog, ghost, (~156 GB: nextcloud, ttrss, whisper-priv, whisper-webui, blog, ghost,
mbg-wordpress, sogo, sogo-mail, outrun-game, owntracks); decide first mbg-wordpress, sogo, sogo-mail, outrun-game, owntracks); decide first
whether any of them is worth keeping whether any of them is worth keeping. The dailies of 10-27 and 10-28
on Garage completed without errors
- [ ] Garage's SQLite WAL (`meta/db.sqlite-wal`) was 1.7 GB on - [ ] Garage's SQLite WAL (`meta/db.sqlite-wal`) was 1.7 GB on
2026-10-26 next to a 270 MB database, and going by the folders in 2026-10-26 next to a 270 MB database, and going by the folders in
`meta/snapshots`, each 6-hourly metadata snapshot takes about 70 `meta/snapshots`, each 6-hourly metadata snapshot takes about 70
minutes. Not growing; check again after a week of normal use minutes. Not growing; check again after a week of normal use. 1.8 GB
on 2026-10-28
### 6. Monitoring ### 6. Monitoring
@@ -315,13 +330,14 @@ Counted 2026-10-25 over the 73 running containers:
OIDC logins of Vault and FreshRSS reach authentik, all 24 OIDC logins of Vault and FreshRSS reach authentik, all 24
ExternalSecrets refresh, raft intact; Gitea no longer reaches the NAS ExternalSecrets refresh, raft intact; Gitea no longer reaches the NAS
or Vault, and a pod in `default` no longer reaches Vault or Gitea or Vault, and a pod in `default` no longer reaches Vault or Gitea
- [ ] Certificates couldn't renew in those 11 namespaces: Envoy may only - [x] Certificates couldn't renew in those 11 namespaces: Envoy may only
reach each app's own port, and cert-manager's HTTP-01 solver listens on reach each app's own port, and cert-manager's HTTP-01 solver listens on
8089. ynab's renewal hung from 2026-10-26 17:35 UTC; a policy for the 8089. ynab's renewal hung from 2026-10-26 17:35 UTC; a policy for the
solver pods, applied by hand in actual-budget that evening, let it solver pods, applied by hand in actual-budget that evening, let it
through within 20 seconds (now valid until 2027-01-24). PR #51 adds it through within 20 seconds (now valid until 2027-01-24). PR #51 adds it
to all 11. Without it the next renewals hang too: gitea on 11-15, to all 11 (merged 2026-10-26 22:27 UTC). Without it the next renewals
bolletjestrui on 11-18, the rest from 12-03 would hang too; the first real test is gitea's on 11-15, then
bolletjestrui on 11-18 and the rest from 12-03
- [x] The infrastructure namespaces (cert-manager, cnpg-system, - [x] The infrastructure namespaces (cert-manager, cnpg-system,
external-secrets, envoy-gateway-system, velero, mariadb-system, dns, external-secrets, envoy-gateway-system, velero, mariadb-system, dns,
default), merged 2026-10-26: PR #54 for envoy-gateway-system (22:31 default), merged 2026-10-26: PR #54 for envoy-gateway-system (22:31
@@ -349,6 +365,12 @@ Counted 2026-10-25 over the 73 running containers:
sat idle all day (240Mi), so on 2026-10-28 a test image went through sat idle all day (240Mi), so on 2026-10-28 a test image went through
CLIP, face recognition and OCR inside the container: 884Mi with all CLIP, face recognition and OCR inside the container: 884Mi with all
three loaded, limit 2Gi three loaded, limit 2Gi
- [ ] authentik's server fails its liveness probe while its database is
away (`/-/health/live/` answers 500 when PostgreSQL is down), so each
database restart gets the server killed too, as on 2026-10-27 for
PostgreSQL 17.11. The login only comes back once the server has
started again. A liveness probe that doesn't touch the database would
leave it at not ready
- [x] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on - [x] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on
2026-10-25, its third restart: a crawler asked for 88 pages and assets 2026-10-25, its third restart: a crawler asked for 88 pages and assets
within a few seconds (normally ~6 a minute), Apache started a PHP worker within a few seconds (normally ~6 a minute), Apache started a PHP worker
@@ -414,10 +436,15 @@ every PR.
fail the run; tested with a planted Secret and an untagged image), plus fail the run; tested with a planted Secret and an untagged image), plus
PR annotations for containers without requests or probes. freshrss's PR annotations for containers without requests or probes. freshrss's
`alpine/git` init container had no tag; now `v2.52.0` `alpine/git` init container had no tag; now `v2.52.0`
- [ ] Automerge patch versions of low-risk apps now that CI runs: PR #52 - [x] Automerge patch versions of low-risk apps now that CI runs: PR #52,
(ntfy, gatus, freshrss, serpbear, actual-budget and CI's own tools; merged 2026-10-26 (ntfy, gatus, freshrss, serpbear, actual-budget and
three days after the release, only on weekday evenings and at the CI's own tools; three days after the release, only on weekday evenings
weekend, when the cluster can be reached) and at the weekend, when the cluster can be reached). Nothing has
merged itself yet by 2026-10-28
- [x] MariaDB stays on 11.4 (PR #60, 2026-10-28): Renovate offered 11.8.9
as a minor update (#42, closed), but 11.8 can't be rolled back and has
community support until June 2028, against May 2029 for 11.4. Renovate
now only offers 11.4 patches; the next LTS step, 12.3, is by hand
- [x] Renovate listed external-snapshotter twice (#15 and #24): its - [x] Renovate listed external-snapshotter twice (#15 and #24): its
`argocd` manager and a custom `github-tags` regex both matched. The `argocd` manager and a custom `github-tags` regex both matched. The
regex is gone; Renovate should close #15 by itself regex is gone; Renovate should close #15 by itself
@@ -510,7 +537,7 @@ critical / high:
| Image | Crit | High | What to do | | Image | Crit | High | What to do |
|---|---|---|---| |---|---|---|---|
| tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; deleted 2026-10-28 | | tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; deleted 2026-10-28 |
| cloudnative-pg/postgresql:17.4 | 27 | 260 | authentik's database, a Debian 11 image. Renovate now offers 17.7; a `17.x-standard-trixie` image is more current | | cloudnative-pg/postgresql:17.4 | 27 | 260 | authentik's database, a Debian 11 image. 17.11 since PR #40 (2026-10-27); not scanned again, a `17.x-standard-trixie` image is more current |
| tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag | | tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag |
| longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below | | longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below |
| actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it | | actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it |