Compare commits

...
10 Commits
Author SHA1 Message Date
Ruben Hensen 6d9e405126 Point Velero and the NAS services back at 192.168.1.99
Reverts 8c2bd89. The NAS had fallen back to a DHCP address (.10), which
also moved the UniFi controller it hosts away from where the gateway
and access points inform, and left the Longhorn backup target at
nfs://192.168.1.99 unreachable. It now has 192.168.1.99 set statically
in DSM again, so everything that pointed at .99 works without changes.
2026-10-21 15:25:16 +02:00
Ruben Hensen dfdc4451fb velero: raise the server memory limit to 2Gi
At 512Mi the server has been OOMKilled 314 times. With the bucket
reachable again it died 23 seconds into a manual backup of ~14.7k items,
which then got marked Failed on restart. opti1 has 32GB.
2026-10-21 12:33:31 +02:00
Ruben Hensen e19f4b8db5 velero: skip the CRD upgrade hook
The hook's init container pulls docker.io/bitnami/kubectl:1.32, which
Bitnami has removed, so the sync for the new NAS address hung on
ImagePullBackOff. The chart stays at 8.4.0 and the CRDs are already in
place.
2026-10-21 12:22:57 +02:00
Ruben Hensen 8c2bd89444 Point Velero and the NAS services at 192.168.1.10
The NAS moved from 192.168.1.99 to .10, so the backup storage location
has been Unavailable ("no route to host"). Every backup since at least
26 July failed, and 154 expired ones are stuck in Deleting because Velero
can't reach the bucket to remove them. MinIO answers on .10:9000 and DSM
on .10:5000.
2026-10-21 12:16:19 +02:00
Ruben Hensen 400d46c462 actual-budget: use a CA bundle without the compromised eMudhra roots
Node uses its built-in roots, which still include emSign ECC Root CA - G3,
and the Node fix is still pending. Switch to OpenSSL's store with
--use-openssl-ca and point it at patched-ca-bundle, mounted over
/etc/ssl/certs. That covers bank sync and the authentik OIDC calls.
2026-10-21 12:09:41 +02:00
Ruben Hensen 25a6bf3a05 authentik: mount a CA bundle without the compromised eMudhra roots
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
2026-10-21 12:09:41 +02:00
Ruben Hensen a507282103 argocd: mount a CA bundle without the compromised eMudhra roots
The Argo CD v2.14.2 image (Ubuntu 24.04, ca-certificates 20240203) and the
dex image still trust the eMudhra intermediates whose keys leaked
(https://emudhra.we-have-a-problem.com), and the repo server fetches git
and Helm repos over that trust store. A forged certificate there means
arbitrary manifests in the cluster.

patched-ca-bundle holds the bundle from Alpine ca-certificates-bundle
20261020-r0 and replaces /etc/ssl/certs in every Argo CD container except
redis. The whole directory goes, since OpenSSL and Go also read the
per-root files next to ca-certificates.crt. It uses server-side apply to
keep the bundle out of the last-applied annotation.
2026-10-21 12:09:41 +02:00
Ruben Hensen 19eacd35f7 Remove ip-updater
Every run has failed with "Permission denied (publickey)" on
root@46.224.26.65, the last successful job was 219 days ago, and one pod
had been hanging for 138 days. The home IP is pinned in
nix-infra-machine/nodes/node001.nix instead, so nothing used it.

It also ran alpine:3.19, which is EOL and still trusts the compromised
eMudhra intermediates.
2026-10-21 11:49:28 +02:00
Ruben Hensen bb7e2f2bac node001: distrust the roots above the compromised eMudhra intermediates
The private keys of eight eMudhra intermediates are recoverable
(https://emudhra.we-have-a-problem.com). node001 runs NixOS 25.11 from
March with nss-cacert 3.121, and curl there still accepted a chain via
emSign ECC Root CA - G3. Blacklist that root and Comodo AAA Services root
until nixpkgs ships NSS 3.131.1. Everything node001 talks to chains to
ISRG Root X1.

Already deployed with ./cli update node001, and Gatus, ntfy and Stalwart
were restarted so they reload the trust store.
2026-10-21 11:49:28 +02:00
Ruben Hensen 1efa132f9d Stop FreshRSS + MBG WordPress until PHP is patched
CERT-EU 2026-018 (CVE-2026-109618, CVE-2026-109619, CVSS 9.2): possible
unauthenticated RCE in PHP 5 and later. Both are internet-facing and are
the only PHP workloads (FreshRSS on PHP 8.2.29, WordPress on 8.3.30), so
scale them to 0 until their images ship a fixed PHP. PVCs and the MariaDB
cluster stay as they are.
2026-10-21 09:18:24 +02:00
14 changed files with 9390 additions and 82 deletions

No files matched your search

+14
View File
@@ -36,6 +36,20 @@ spec:
extraEnvVars:
ACTUAL_OPENID_SERVER_HOSTNAME: https://ynab.rubenhensen.nl
# Node trusts its built-in roots, which still include the ones
# above the compromised eMudhra intermediates. Use patched-ca-bundle
# through OpenSSL instead until Node ships a fix.
NODE_OPTIONS: --use-openssl-ca
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
# Service configuration
service:
File diff suppressed because it is too large. Load diff
+19
View File
@@ -86,8 +86,18 @@ spec:
secretKeyRef:
name: actualbudget-oidc-client-secret
key: client_secret
# Point OpenSSL and requests (which uses certifi otherwise) at
# patched-ca-bundle, mounted below until the image drops the
# compromised eMudhra intermediates.
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca-certificates.crt
- name: REQUESTS_CA_BUNDLE
value: /etc/ssl/certs/ca-certificates.crt
server:
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
- name: blueprint-vault
configMap:
name: authentik-blueprint-vault
@@ -131,6 +141,9 @@ spec:
- name: blueprint-session
mountPath: /blueprints/custom/session-duration.yaml
subPath: session-duration.yaml
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
ingress:
enabled: true
ingressClassName: nginx
@@ -144,6 +157,9 @@ spec:
- authentik.rubenhensen.nl
worker:
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
- name: blueprint-vault
configMap:
name: authentik-blueprint-vault
@@ -187,6 +203,9 @@ spec:
- name: blueprint-session
mountPath: /blueprints/custom/session-duration.yaml
subPath: session-duration.yaml
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
postgresql:
enabled: false
redis:
-22
View File
@@ -1,22 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ip-updater
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: ip-updater
destination:
server: https://kubernetes.default.svc
namespace: ip-updater
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
prune: true
+4
View File
@@ -16,6 +16,10 @@ spec:
chart: wordpress
helm:
values: |
# Stopped until PHP is patched (CERT-EU 2026-018, CVE-2026-109618,
# CVE-2026-109619). Set back to 1 once the image ships a fixed PHP.
replicaCount: 0
# Official WordPress image configuration
image:
registry: docker.io
+3 -1
View File
@@ -9,6 +9,7 @@ resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/refs/tags/v2.14.2/manifests/install.yaml
- ingress.yaml
- oidc-secret.yaml
- patched-ca-bundle.yaml
patches:
- target:
@@ -18,4 +19,5 @@ patches:
name: argocd-notifications-controller
path: overlays/argocd-notifications-controller-patch.yaml
- path: overlays/argocd-cm.patch.yaml
- path: overlays/argocd-rbac-cm.patch.yaml
- path: overlays/argocd-rbac-cm.patch.yaml
- path: overlays/patched-ca-bundle.patch.yaml
@@ -0,0 +1,111 @@
# Mount patched-ca-bundle over /etc/ssl/certs so git, helm and the Go
# binaries stop trusting the compromised eMudhra intermediates. The whole
# directory is replaced because OpenSSL and Go also read the per-root files
# next to ca-certificates.crt. Drop this once Argo CD ships a fixed image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-server
spec:
template:
spec:
containers:
- name: argocd-server
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: argocd-repo-server
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: argocd-application-controller
spec:
template:
spec:
containers:
- name: argocd-application-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-applicationset-controller
spec:
template:
spec:
containers:
- name: argocd-applicationset-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-notifications-controller
spec:
template:
spec:
containers:
- name: argocd-notifications-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-dex-server
spec:
template:
spec:
containers:
- name: dex
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+3 -1
View File
@@ -8,7 +8,9 @@ metadata:
io.kompose.service: freshrss
name: freshrss
spec:
replicas: 1
# Stopped until PHP is patched (CERT-EU 2026-018, CVE-2026-109618,
# CVE-2026-109619). Set back to 1 once the image ships a fixed PHP.
replicas: 0
selector:
matchLabels:
io.kompose.service: freshrss
-39
View File
@@ -1,39 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: ip-updater
spec:
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
containers:
- name: ip-updater
image: alpine:3.19
command: ["/bin/sh", "-c"]
args:
- |
apk add --no-cache openssh-client curl > /dev/null 2>&1
cp /ssh-key/private_key /tmp/ssh_key
chmod 600 /tmp/ssh_key
IP=$(curl -s --max-time 10 https://api.ipify.org)
if [ -z "$IP" ]; then
echo "Failed to get external IP"
exit 1
fi
echo "Current external IP: $IP"
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-i /tmp/ssh_key root@46.224.26.65 "$IP"
volumeMounts:
- name: ssh-key
mountPath: /ssh-key
readOnly: true
volumes:
- name: ssh-key
secret:
secretName: ip-updater-ssh-key
defaultMode: 0400
restartPolicy: OnFailure
-17
View File
@@ -1,17 +0,0 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ip-updater-ssh-key
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: ip-updater-ssh-key
creationPolicy: Owner
data:
- secretKey: private_key
remoteRef:
key: kv/ip-updater
property: ssh_private_key
+11
View File
@@ -122,6 +122,17 @@ in
80 # HTTP (ACME)
];
# ──────────────────────────────────────────────
# Trust store
# ──────────────────────────────────────────────
# Distrust the roots above the compromised eMudhra intermediates until
# nixpkgs ships NSS 3.131.1, see https://emudhra.we-have-a-problem.com.
# Everything this box talks to chains to ISRG Root X1.
security.pki.caCertificateBlacklist = [
"Comodo AAA Services root"
"emSign ECC Root CA - G3"
];
# ──────────────────────────────────────────────
# ACME / Let's Encrypt
# ──────────────────────────────────────────────
+9 -2
View File
@@ -11,6 +11,11 @@ spec:
targetRevision: 8.4.0
helm:
values: |
# The CRD upgrade hook pulls docker.io/bitnami/kubectl, which Bitnami
# has removed, so every sync hangs on it. The CRDs only change with
# the chart version; turn this back on (with another kubectl image)
# when bumping targetRevision.
upgradeCRDs: false
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:v1.11.1
@@ -43,13 +48,15 @@ spec:
nodeAgent:
containerSecurityContext:
privileged: true
# 512Mi got the server OOMKilled 314 times, including halfway
# through a backup of ~14.7k items.
resources:
requests:
cpu: 100m
memory: 256Mi
memory: 512Mi
limits:
cpu: 500m
memory: 512Mi
memory: 2Gi
destination:
server: https://kubernetes.default.svc