Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
Velero 1.15 builds kopia repository maintenance jobs from a Deployment
called "velero" in its namespace. Ours was velero-helm, so every
BackupRepository reported 'error to build maintenance job: Deployment.apps
"velero" not found' and the repos on the NAS were never cleaned up, some
not since March 2025. fullnameOverride renames the Deployment, Service,
ServiceAccount and RBAC; the old velero-helm objects get deleted by hand
since this app doesn't prune.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). The server gets --use-openssl-ca for Node, machine-learning points requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). whishper also gets --use-openssl-ca for Node, and both point requests at the bundle instead of certifi.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Nine more namespaces need patched-ca-bundle, the bundle from Alpine
ca-certificates-bundle 20261020-r0 without the roots above the compromised
eMudhra intermediates. Rather than another nine ~190KB copies, a small
chart renders the ConfigMap from one ca-certificates.crt into every
namespace in values.yaml. argocd, authentik and actual-budget keep their
own copies for now.
lingo-api ran PHP 8.1.34 (php:8.1-apache) and was reachable at
https://lingo.rubenhensen.nl/api/. PHP 8.1 is EOL and gets no fix for
CVE-2026-109618 and CVE-2026-109619, so it was exposed to the PHP2Shell
RCE. It is no longer used, so drop it entirely: the app, its Gatus
check, the DNS record and the node001 tunnel entry.
wordpress:7.1.3-php8.3-apache was rebuilt at 15:57 UTC on PHP 8.3.36,
which fixes CVE-2026-109618 and CVE-2026-109619. WordPress core lives on
the volume, so the tag only moves PHP (8.3.30 -> 8.3.36) and Apache. Its
ca-certificates (20250419+deb13u1) no longer carries the roots above the
compromised eMudhra intermediates either.
freshrss/freshrss:1.30.3, released today, runs PHP 8.4.27 from Debian
trixie security (8.4.27-1~deb13u1), which fixes CVE-2026-109618 and
CVE-2026-109619. Pin it instead of :latest so the running version is the
checked one. Its ca-certificates (20250419+deb13u1) also no longer
carries the roots above the compromised eMudhra intermediates.
Reverts 8c2bd89. The NAS had fallen back to a DHCP address (.10), which
also moved the UniFi controller it hosts away from where the gateway
and access points inform, and left the Longhorn backup target at
nfs://192.168.1.99 unreachable. It now has 192.168.1.99 set statically
in DSM again, so everything that pointed at .99 works without changes.
At 512Mi the server has been OOMKilled 314 times. With the bucket
reachable again it died 23 seconds into a manual backup of ~14.7k items,
which then got marked Failed on restart. opti1 has 32GB.
The hook's init container pulls docker.io/bitnami/kubectl:1.32, which
Bitnami has removed, so the sync for the new NAS address hung on
ImagePullBackOff. The chart stays at 8.4.0 and the CRDs are already in
place.
The NAS moved from 192.168.1.99 to .10, so the backup storage location
has been Unavailable ("no route to host"). Every backup since at least
26 July failed, and 154 expired ones are stuck in Deleting because Velero
can't reach the bucket to remove them. MinIO answers on .10:9000 and DSM
on .10:5000.
Node uses its built-in roots, which still include emSign ECC Root CA - G3,
and the Node fix is still pending. Switch to OpenSSL's store with
--use-openssl-ca and point it at patched-ca-bundle, mounted over
/etc/ssl/certs. That covers bank sync and the authentik OIDC calls.
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
The Argo CD v2.14.2 image (Ubuntu 24.04, ca-certificates 20240203) and the
dex image still trust the eMudhra intermediates whose keys leaked
(https://emudhra.we-have-a-problem.com), and the repo server fetches git
and Helm repos over that trust store. A forged certificate there means
arbitrary manifests in the cluster.
patched-ca-bundle holds the bundle from Alpine ca-certificates-bundle
20261020-r0 and replaces /etc/ssl/certs in every Argo CD container except
redis. The whole directory goes, since OpenSSL and Go also read the
per-root files next to ca-certificates.crt. It uses server-side apply to
keep the bundle out of the last-applied annotation.
Every run has failed with "Permission denied (publickey)" on
root@46.224.26.65, the last successful job was 219 days ago, and one pod
had been hanging for 138 days. The home IP is pinned in
nix-infra-machine/nodes/node001.nix instead, so nothing used it.
It also ran alpine:3.19, which is EOL and still trusts the compromised
eMudhra intermediates.
The private keys of eight eMudhra intermediates are recoverable
(https://emudhra.we-have-a-problem.com). node001 runs NixOS 25.11 from
March with nss-cacert 3.121, and curl there still accepted a chain via
emSign ECC Root CA - G3. Blacklist that root and Comodo AAA Services root
until nixpkgs ships NSS 3.131.1. Everything node001 talks to chains to
ISRG Root X1.
Already deployed with ./cli update node001, and Gatus, ntfy and Stalwart
were restarted so they reload the trust store.
CERT-EU 2026-018 (CVE-2026-109618, CVE-2026-109619, CVSS 9.2): possible
unauthenticated RCE in PHP 5 and later. Both are internet-facing and are
the only PHP workloads (FreshRSS on PHP 8.2.29, WordPress on 8.3.30), so
scale them to 0 until their images ship a fixed PHP. PVCs and the MariaDB
cluster stay as they are.
The store went through https://vault.rubenhensen.nl, so via Hetzner and
the home internet connection. When home internet dropped, the store
validation failed and External Secrets backed off long enough that it
needed a controller restart after the connection came back.
vault-active.vault.svc talks to the active Vault pod directly. It's plain
HTTP, same as the ingress → Vault hop already is.
ntfy listens on 127.0.0.1:2586 behind the same SNI-to-8444 setup as s1.
Users and tokens come from encrypted nix-infra secrets, exported as env
vars by a small wrapper since ntfy and Gatus only read them from the
environment.
s1 alerts on the home services, Scouting MBG and s2. It stays quiet on
mail, which lives on this box and is covered by s2.
Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at
the home IP like s2. s2 now alerts on mail and on s1 through it. The
home services are left to s1, which sees them from outside.
ntfy denies everything except the provisioned user. The user, its
password hash and the Gatus token come from Vault kv/ntfy.
Gatus listens on 127.0.0.1:8090. nginx gets s1 in the SNI map, sends it
to a local TLS vhost on 8444 with its own ACME cert, and proxies to
Gatus from there. The endpoint list is built from tunneledHosts plus
mail, Scouting MBG and s2, matching gatus/config.yaml.
Gatus with sqlite history, checking every public service plus its twin
s1.rubenhensen.nl on Hetzner. Replaces the GitHub-hosted Upptime page.
DNS gets s1 (Hetzner) and s2. s2 points straight at the home IP rather
than through the Hetzner tunnel, so it stays reachable when Hetzner is
down.
gatus/README.md explains how to read the two pages and writes down the
plan for a single self-hosted failover URL (PowerDNS LUA records).
Authentik's default-authentication-login stage ships with
session_duration: seconds=0, so the SSO session died with the browser.
New blueprint raises it to 30 days for every app on that flow.
FreshRSS runs OIDC through Apache mod_auth_openidc, whose defaults are a
5 minute inactivity timeout and a ~7.5 hour max duration in a non-persistent
cookie. Bumped to a 30 day sliding inactivity window (refreshed on every
request) with a 90 day hard cap, and made the cookie persistent so it
survives a browser restart.
Adds the VAPID keypair for web push and raises the ingress body limit to 25m to
match the app's BODY_SIZE_LIMIT, since a bulk route import posts every GPX in one
request.
The VAPID env refs are optional:true on purpose — a missing key in a secretKeyRef
puts the pod in CreateContainerConfigError, and the app already handles absent
keys by sending no notifications.
Add vic.rubenhensen.nl to tunneledHosts so the edge box does SNI
passthrough on :443 and the :80 ACME/redirect vhost for it, matching
the DNS record. Without this, vic falls through to Stalwart on :8443
and cert-manager's HTTP-01 challenge can't be reached.
Deploys ghcr.io/rubenhensen/bolletjestrui at vic.rubenhensen.nl:
- bolletjestrui/ folder: deployment (SQLite on a PVC, migrate-on-start,
ORIGIN + raised BODY_SIZE_LIMIT for GPX uploads), service, nginx
ingress with prod-cluster-issuer TLS, PVC, and a Vault ExternalSecret
(kv/bolletjestrui: site_password, admin_password, key_secret).
- apps/templates/bolletjestrui-folder.yaml: ArgoCD Application.
- dns: vic A record -> 46.224.26.65 (ingress LB).
- renovate: watch bolletjestrui/ for image tag bumps.
Committed locally, not pushed: needs the Vault secret created and the
ghcr package made public first.
Ruben Hensen