Needed from ESO 0.19, whose CRDs are too big for client-side apply. Turned on now, on 0.10.7, so Argo CD migrates the managed fields before the upgrade starts.
Pin authentik-postgres to the image CNPG filled in at creation
(postgresql:17.4). Later operators default to PostgreSQL 18, and since
1.26 a higher major in the image triggers an offline in-place major
upgrade. immich-postgres already pins its pgvecto.rs image.
Turn on ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES so the five operator hops
(1.25 -> 1.30, one minor at a time as the docs recommend) swap the
instance manager inside the running pods instead of restarting both
single-instance databases each time.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
One minor at a time from v1.15.5 to v1.20.4 on the way to Kubernetes 1.33, see docs/upgrade-k8s-1.33.md. The chart renders the current values unchanged.
Last hop of the Longhorn upgrade for Kubernetes 1.33. Stays on 1.11.2:
1.11.3 ships csi-provisioner v6.3.0 and needs Kubernetes 1.34. The
1.11.2 chart still declares kubeVersion >=1.25 and ships csi-provisioner
v5.3.0, and renders our values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-10-2-202610221449 is Ready. PV
nodeAffinity recorded to compare after the upgrade (1.11.0 had a
regression there, fixed in 1.11.2).
Third hop towards 1.11.2. 1.10 removes the longhorn.io/v1beta1 API, which
requires every Longhorn CRD to store only v1beta2; all 22 already do (the
check from the 1.10 important notes). Settings move to a per-data-engine
JSON format, but the chart renders our defaultSettings and
defaultBackupStore values unchanged.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, system backup pre-upgrade-v1-9-2-202610221421 is Ready.
Second hop towards 1.11.2. 1.9 stops serving longhorn.io/v1beta1 (nothing
in this repo uses it) and renames orphan-auto-deletion (false here) to
orphan-resource-auto-deletion during the upgrade. Chart values are
unchanged from 1.8.2.
Checked beforehand: 17 healthy volumes plus 2 detached, no backing
images, every Longhorn CRD already stores only v1beta2, and system backup
pre-upgrade-v1-8-2-202610221408 is Ready.
First hop from 1.7.3 to 1.11.2 (docs/upgrade-k8s-1.33.md). 1.8 drops the
backup-target settings in favour of a "default" BackupTarget resource, so
the NFS target and poll interval move into defaultBackupStore. Until now
they only lived in the cluster.
Checked beforehand: no Faulted volumes, no backing images, all volumes on
engine v1.7.3, system backup pre-upgrade-v1-7-3-20261022 and Velero
backup pre-longhorn-1-8-202610221326 both complete.
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
Neither is used any more: the blog moves to a self-written site, and
whishper needs more compute than this setup has. Ghost also ran Node 18,
which won't get the eMudhra root store fix.
Drops both apps, the blog DNS record, Gatus check and node001 tunnel
entry, their ca-bundle namespaces, and the leftover ExternalSecret for the
old hensen.io blog namespace. Final Velero backups keep the data:
ghost-blog-final-202610220955 until 22 October 2027 and
whisper-final-202610220955 until 20 January 2027.
Neither host has a DNS record, so both were unreachable and cert-manager
has been failing to renew their certificates since 6 April. DSM stays
reachable on the LAN at 192.168.1.99. The nas app only held that ingress
and its ExternalName service, so it goes entirely.
Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
Nine more namespaces need patched-ca-bundle, the bundle from Alpine
ca-certificates-bundle 20261020-r0 without the roots above the compromised
eMudhra intermediates. Rather than another nine ~190KB copies, a small
chart renders the ConfigMap from one ca-certificates.crt into every
namespace in values.yaml. argocd, authentik and actual-budget keep their
own copies for now.
lingo-api ran PHP 8.1.34 (php:8.1-apache) and was reachable at
https://lingo.rubenhensen.nl/api/. PHP 8.1 is EOL and gets no fix for
CVE-2026-109618 and CVE-2026-109619, so it was exposed to the PHP2Shell
RCE. It is no longer used, so drop it entirely: the app, its Gatus
check, the DNS record and the node001 tunnel entry.
wordpress:7.1.3-php8.3-apache was rebuilt at 15:57 UTC on PHP 8.3.36,
which fixes CVE-2026-109618 and CVE-2026-109619. WordPress core lives on
the volume, so the tag only moves PHP (8.3.30 -> 8.3.36) and Apache. Its
ca-certificates (20250419+deb13u1) no longer carries the roots above the
compromised eMudhra intermediates either.
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
Every run has failed with "Permission denied (publickey)" on
root@46.224.26.65, the last successful job was 219 days ago, and one pod
had been hanging for 138 days. The home IP is pinned in
nix-infra-machine/nodes/node001.nix instead, so nothing used it.
It also ran alpine:3.19, which is EOL and still trusts the compromised
eMudhra intermediates.
CERT-EU 2026-018 (CVE-2026-109618, CVE-2026-109619, CVSS 9.2): possible
unauthenticated RCE in PHP 5 and later. Both are internet-facing and are
the only PHP workloads (FreshRSS on PHP 8.2.29, WordPress on 8.3.30), so
scale them to 0 until their images ship a fixed PHP. PVCs and the MariaDB
cluster stay as they are.
Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at
the home IP like s2. s2 now alerts on mail and on s1 through it. The
home services are left to s1, which sees them from outside.
ntfy denies everything except the provisioned user. The user, its
password hash and the Gatus token come from Vault kv/ntfy.
Gatus with sqlite history, checking every public service plus its twin
s1.rubenhensen.nl on Hetzner. Replaces the GitHub-hosted Upptime page.
DNS gets s1 (Hetzner) and s2. s2 points straight at the home IP rather
than through the Hetzner tunnel, so it stays reachable when Hetzner is
down.
gatus/README.md explains how to read the two pages and writes down the
plan for a single self-hosted failover URL (PowerDNS LUA records).
Authentik's default-authentication-login stage ships with
session_duration: seconds=0, so the SSO session died with the browser.
New blueprint raises it to 30 days for every app on that flow.
FreshRSS runs OIDC through Apache mod_auth_openidc, whose defaults are a
5 minute inactivity timeout and a ~7.5 hour max duration in a non-persistent
cookie. Bumped to a 30 day sliding inactivity window (refreshed on every
request) with a 90 day hard cap, and made the cookie persistent so it
survives a browser restart.
Deploys ghcr.io/rubenhensen/bolletjestrui at vic.rubenhensen.nl:
- bolletjestrui/ folder: deployment (SQLite on a PVC, migrate-on-start,
ORIGIN + raised BODY_SIZE_LIMIT for GPX uploads), service, nginx
ingress with prod-cluster-issuer TLS, PVC, and a Vault ExternalSecret
(kv/bolletjestrui: site_password, admin_password, key_secret).
- apps/templates/bolletjestrui-folder.yaml: ArgoCD Application.
- dns: vic A record -> 46.224.26.65 (ingress LB).
- renovate: watch bolletjestrui/ for image tag bumps.
Committed locally, not pushed: needs the Vault secret created and the
ghcr package made public first.
Single-pod sqlite install (simplest path for a home git host). HTTPS-only,
no SSH listener for now. Wired into the Hetzner tunnel like the other
rubenhensen.nl services.
First visitor to git.rubenhensen.nl can register; Gitea auto-promotes the
first user to admin.
Ruben Hensen