First hop from 1.7.3 to 1.11.2 (docs/upgrade-k8s-1.33.md). 1.8 drops the
backup-target settings in favour of a "default" BackupTarget resource, so
the NFS target and poll interval move into defaultBackupStore. Until now
they only lived in the cluster.
Checked beforehand: no Faulted volumes, no backing images, all volumes on
engine v1.7.3, system backup pre-upgrade-v1-7-3-20261022 and Velero
backup pre-longhorn-1-8-202610221326 both complete.
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
Envoy Gateway v1.9 needs Kubernetes 1.33, and microk8s 1.32 is out of
upstream support. docs/upgrade-k8s-1.33.md lists what has to move first
and in which order. Renovate is held below Longhorn 1.11.3, since that
and newer releases need Kubernetes 1.34.
After the microk8s ingress addon went from ingress-nginx 1.8.0 to 1.13.8,
ha.rubenhensen.nl returned 503: the controller now resolves externalName
as a DNS name and "192.168.1.202" fails the lookup. A selector-less
Service with an EndpointSlice pointing at the IP is the supported way to
route to an address outside the cluster.
Neither is used any more: the blog moves to a self-written site, and
whishper needs more compute than this setup has. Ghost also ran Node 18,
which won't get the eMudhra root store fix.
Drops both apps, the blog DNS record, Gatus check and node001 tunnel
entry, their ca-bundle namespaces, and the leftover ExternalSecret for the
old hensen.io blog namespace. Final Velero backups keep the data:
ghost-blog-final-202610220955 until 22 October 2027 and
whisper-final-202610220955 until 20 January 2027.
Neither host has a DNS record, so both were unreachable and cert-manager
has been failing to renew their certificates since 6 April. DSM stays
reachable on the LAN at 192.168.1.99. The nas app only held that ingress
and its ExternalName service, so it goes entirely.
Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
Velero 1.15 builds kopia repository maintenance jobs from a Deployment
called "velero" in its namespace. Ours was velero-helm, so every
BackupRepository reported 'error to build maintenance job: Deployment.apps
"velero" not found' and the repos on the NAS were never cleaned up, some
not since March 2025. fullnameOverride renames the Deployment, Service,
ServiceAccount and RBAC; the old velero-helm objects get deleted by hand
since this app doesn't prune.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). The server gets --use-openssl-ca for Node, machine-learning points requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). whishper also gets --use-openssl-ca for Node, and both point requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Nine more namespaces need patched-ca-bundle, the bundle from Alpine
ca-certificates-bundle 20261020-r0 without the roots above the compromised
eMudhra intermediates. Rather than another nine ~190KB copies, a small
chart renders the ConfigMap from one ca-certificates.crt into every
namespace in values.yaml. argocd, authentik and actual-budget keep their
own copies for now.
lingo-api ran PHP 8.1.34 (php:8.1-apache) and was reachable at
https://lingo.rubenhensen.nl/api/. PHP 8.1 is EOL and gets no fix for
CVE-2026-109618 and CVE-2026-109619, so it was exposed to the PHP2Shell
RCE. It is no longer used, so drop it entirely: the app, its Gatus
check, the DNS record and the node001 tunnel entry.
wordpress:7.1.3-php8.3-apache was rebuilt at 15:57 UTC on PHP 8.3.36,
which fixes CVE-2026-109618 and CVE-2026-109619. WordPress core lives on
the volume, so the tag only moves PHP (8.3.30 -> 8.3.36) and Apache. Its
ca-certificates (20250419+deb13u1) no longer carries the roots above the
compromised eMudhra intermediates either.
freshrss/freshrss:1.30.3, released today, runs PHP 8.4.27 from Debian
trixie security (8.4.27-1~deb13u1), which fixes CVE-2026-109618 and
CVE-2026-109619. Pin it instead of :latest so the running version is the
checked one. Its ca-certificates (20250419+deb13u1) also no longer
carries the roots above the compromised eMudhra intermediates.
Reverts 8c2bd89. The NAS had fallen back to a DHCP address (.10), which
also moved the UniFi controller it hosts away from where the gateway
and access points inform, and left the Longhorn backup target at
nfs://192.168.1.99 unreachable. It now has 192.168.1.99 set statically
in DSM again, so everything that pointed at .99 works without changes.
At 512Mi the server has been OOMKilled 314 times. With the bucket
reachable again it died 23 seconds into a manual backup of ~14.7k items,
which then got marked Failed on restart. opti1 has 32GB.
The hook's init container pulls docker.io/bitnami/kubectl:1.32, which
Bitnami has removed, so the sync for the new NAS address hung on
ImagePullBackOff. The chart stays at 8.4.0 and the CRDs are already in
place.
The NAS moved from 192.168.1.99 to .10, so the backup storage location
has been Unavailable ("no route to host"). Every backup since at least
26 July failed, and 154 expired ones are stuck in Deleting because Velero
can't reach the bucket to remove them. MinIO answers on .10:9000 and DSM
on .10:5000.
Node uses its built-in roots, which still include emSign ECC Root CA - G3,
and the Node fix is still pending. Switch to OpenSSL's store with
--use-openssl-ca and point it at patched-ca-bundle, mounted over
/etc/ssl/certs. That covers bank sync and the authentik OIDC calls.
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
The Argo CD v2.14.2 image (Ubuntu 24.04, ca-certificates 20240203) and the
dex image still trust the eMudhra intermediates whose keys leaked
(https://emudhra.we-have-a-problem.com), and the repo server fetches git
and Helm repos over that trust store. A forged certificate there means
arbitrary manifests in the cluster.
patched-ca-bundle holds the bundle from Alpine ca-certificates-bundle
20261020-r0 and replaces /etc/ssl/certs in every Argo CD container except
redis. The whole directory goes, since OpenSSL and Go also read the
per-root files next to ca-certificates.crt. It uses server-side apply to
keep the bundle out of the last-applied annotation.
Every run has failed with "Permission denied (publickey)" on
root@46.224.26.65, the last successful job was 219 days ago, and one pod
had been hanging for 138 days. The home IP is pinned in
nix-infra-machine/nodes/node001.nix instead, so nothing used it.
It also ran alpine:3.19, which is EOL and still trusts the compromised
eMudhra intermediates.
The private keys of eight eMudhra intermediates are recoverable
(https://emudhra.we-have-a-problem.com). node001 runs NixOS 25.11 from
March with nss-cacert 3.121, and curl there still accepted a chain via
emSign ECC Root CA - G3. Blacklist that root and Comodo AAA Services root
until nixpkgs ships NSS 3.131.1. Everything node001 talks to chains to
ISRG Root X1.
Already deployed with ./cli update node001, and Gatus, ntfy and Stalwart
were restarted so they reload the trust store.
CERT-EU 2026-018 (CVE-2026-109618, CVE-2026-109619, CVSS 9.2): possible
unauthenticated RCE in PHP 5 and later. Both are internet-facing and are
the only PHP workloads (FreshRSS on PHP 8.2.29, WordPress on 8.3.30), so
scale them to 0 until their images ship a fixed PHP. PVCs and the MariaDB
cluster stay as they are.
The store went through https://vault.rubenhensen.nl, so via Hetzner and
the home internet connection. When home internet dropped, the store
validation failed and External Secrets backed off long enough that it
needed a controller restart after the connection came back.
vault-active.vault.svc talks to the active Vault pod directly. It's plain
HTTP, same as the ingress → Vault hop already is.
ntfy listens on 127.0.0.1:2586 behind the same SNI-to-8444 setup as s1.
Users and tokens come from encrypted nix-infra secrets, exported as env
vars by a small wrapper since ntfy and Gatus only read them from the
environment.
s1 alerts on the home services, Scouting MBG and s2. It stays quiet on
mail, which lives on this box and is covered by s2.
Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at
the home IP like s2. s2 now alerts on mail and on s1 through it. The
home services are left to s1, which sees them from outside.
ntfy denies everything except the provisioned user. The user, its
password hash and the Gatus token come from Vault kv/ntfy.
Gatus listens on 127.0.0.1:8090. nginx gets s1 in the SNI map, sends it
to a local TLS vhost on 8444 with its own ACME cert, and proxies to
Gatus from there. The endpoint list is built from tunneledHosts plus
mail, Scouting MBG and s2, matching gatus/config.yaml.
Gatus with sqlite history, checking every public service plus its twin
s1.rubenhensen.nl on Hetzner. Replaces the GitHub-hosted Upptime page.
DNS gets s1 (Hetzner) and s2. s2 points straight at the home IP rather
than through the Hetzner tunnel, so it stays reachable when Hetzner is
down.
gatus/README.md explains how to read the two pages and writes down the
plan for a single self-hosted failover URL (PowerDNS LUA records).
Authentik's default-authentication-login stage ships with
session_duration: seconds=0, so the SSO session died with the browser.
New blueprint raises it to 30 days for every app on that flow.
FreshRSS runs OIDC through Apache mod_auth_openidc, whose defaults are a
5 minute inactivity timeout and a ~7.5 hour max duration in a non-persistent
cookie. Bumped to a 30 day sliding inactivity window (refreshed on every
request) with a 90 day hard cap, and made the cookie persistent so it
survives a browser restart.
Adds the VAPID keypair for web push and raises the ingress body limit to 25m to
match the app's BODY_SIZE_LIMIT, since a bulk route import posts every GPX in one
request.
The VAPID env refs are optional:true on purpose — a missing key in a secretKeyRef
puts the pod in CreateContainerConfigError, and the app already handles absent
keys by sending no notifications.
Ruben Hensen