Compare commits

..
39 Commits
Author SHA1 Message Date
renovate[bot] ba6e53db5f Update helm chart longhorn to v1.11.2 2026-10-22 10:57:32 +00:00
Ruben Hensen 6da1b3e88e longhorn: upgrade volume engines automatically, one at a time
concurrentAutomaticEngineUpgradePerNodeLimit was 0, so volumes stayed on
the engine image of whatever Longhorn version created them: three on
1.7.1, nine on 1.7.2, seven on 1.7.3. Every hop of the upgrade to 1.11.2
needs all volumes on the current engine first.
2026-10-22 12:56:21 +02:00
Ruben Hensen 8aaf2ec8ec Plan the move to Kubernetes 1.33
Envoy Gateway v1.9 needs Kubernetes 1.33, and microk8s 1.32 is out of
upstream support. docs/upgrade-k8s-1.33.md lists what has to move first
and in which order. Renovate is held below Longhorn 1.11.3, since that
and newer releases need Kubernetes 1.34.
2026-10-22 12:56:07 +02:00
Ruben Hensen f2716e1169 home-assistant: mark the EndpointSlice endpoint ready
The API server fills in conditions on every endpoint, which left Argo CD showing the slice as OutOfSync. Spell it out so desired and live state match.
2026-10-22 11:07:42 +02:00
Ruben Hensen a10dbecc94 home-assistant: replace the ExternalName service with an EndpointSlice
After the microk8s ingress addon went from ingress-nginx 1.8.0 to 1.13.8,
ha.rubenhensen.nl returned 503: the controller now resolves externalName
as a DNS name and "192.168.1.202" fails the lookup. A selector-less
Service with an EndpointSlice pointing at the IP is the supported way to
route to an address outside the cluster.
2026-10-22 11:03:19 +02:00
Ruben Hensen 2ce9c33dbd Remove ghost-blog and whishper
Neither is used any more: the blog moves to a self-written site, and
whishper needs more compute than this setup has. Ghost also ran Node 18,
which won't get the eMudhra root store fix.

Drops both apps, the blog DNS record, Gatus check and node001 tunnel
entry, their ca-bundle namespaces, and the leftover ExternalSecret for the
old hensen.io blog namespace. Final Velero backups keep the data:
ghost-blog-final-202610220955 until 22 October 2027 and
whisper-final-202610220955 until 20 January 2027.
2026-10-22 10:11:06 +02:00
Ruben Hensen 8a66913a54 Drop the nas.rubenhensen.nl and alv.rubenhensen.nl ingresses
Neither host has a DNS record, so both were unreachable and cert-manager
has been failing to renew their certificates since 6 April. DSM stays
reachable on the LAN at 192.168.1.99. The nas app only held that ingress
and its ExternalName service, so it goes entirely.
2026-10-22 09:36:26 +02:00
Ruben Hensen 1f736cd4c7 Remove phocas-lustrum
The folder went in April (bb308a3) but the app and its DNS record stayed,
so the demo site kept serving at phocaslustrum.rubenhensen.nl.
2026-10-22 09:36:26 +02:00
Ruben Hensen e5d82b69fd Remove the old in-cluster Stalwart
Its folder went in March (44c31cd) but the Argo CD app stayed and kept the
old hensen.io mail server running, with a NodePort for SMTP/IMAP and a
10Gi volume. Mail runs on node001 now and this one logged nothing in three
days. A final Velero backup (stalwart-final-202610220935) keeps its data
until 20 January 2027. The app has no finalizer, so the namespace gets
deleted by hand.
2026-10-22 09:36:26 +02:00
Ruben Hensen fe678d4b4b ca-bundle: also serve argocd, authentik and actual-budget
Those three namespaces had their own ~190KB copy of patched-ca-bundle.
Let the ca-bundle chart render them too, so the bundle lives in one file.
2026-10-22 09:26:43 +02:00
Ruben Hensen ff73f99d43 velero: name the release resources "velero" so repo maintenance runs
Velero 1.15 builds kopia repository maintenance jobs from a Deployment
called "velero" in its namespace. Ours was velero-helm, so every
BackupRepository reported 'error to build maintenance job: Deployment.apps
"velero" not found' and the repos on the NAS were never cleaned up, some
not since March 2025. fullnameOverride renames the Deployment, Service,
ServiceAccount and RBAC; the old velero-helm objects get deleted by hand
since this app doesn't prune.
2026-10-22 09:23:51 +02:00
Ruben Hensen ea86611deb cert-manager: mount a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Only the controller, which talks ACME to Let's Encrypt.
2026-10-22 09:13:02 +02:00
Ruben Hensen 7820b8e0bf immich: use a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). The server gets --use-openssl-ca for Node, machine-learning points requests at the bundle instead of certifi.
2026-10-22 09:13:02 +02:00
Ruben Hensen de7ca770a0 whishper: use a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). whishper also gets --use-openssl-ca for Node, and both point requests at the bundle instead of certifi.
2026-10-22 09:13:02 +02:00
Ruben Hensen 767facc5e2 bolletjestrui: use a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
2026-10-22 09:13:02 +02:00
Ruben Hensen 746cd1283e ghost-blog: use a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
2026-10-22 09:13:02 +02:00
Ruben Hensen 345055e3e6 serpbear: use a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com). Node uses its built-in roots, so also switch it to OpenSSL's store with --use-openssl-ca.
2026-10-22 09:13:02 +02:00
Ruben Hensen f52b6973e1 gatus: mount a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
2026-10-22 09:13:01 +02:00
Ruben Hensen 177dbef15b ntfy: mount a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
2026-10-22 09:13:01 +02:00
Ruben Hensen 103aed46ed gitea: mount a CA bundle without the compromised eMudhra roots
Mount patched-ca-bundle (see ca-bundle/) over /etc/ssl/certs so it stops trusting the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
2026-10-22 09:13:01 +02:00
Ruben Hensen 9a984c724c Add a ca-bundle chart for the patched CA bundle
Nine more namespaces need patched-ca-bundle, the bundle from Alpine
ca-certificates-bundle 20261020-r0 without the roots above the compromised
eMudhra intermediates. Rather than another nine ~190KB copies, a small
chart renders the ConfigMap from one ca-certificates.crt into every
namespace in values.yaml. argocd, authentik and actual-budget keep their
own copies for now.
2026-10-22 09:11:01 +02:00
Ruben Hensen 3e45ddaf4a Remove lingo
lingo-api ran PHP 8.1.34 (php:8.1-apache) and was reachable at
https://lingo.rubenhensen.nl/api/. PHP 8.1 is EOL and gets no fix for
CVE-2026-109618 and CVE-2026-109619, so it was exposed to the PHP2Shell
RCE. It is no longer used, so drop it entirely: the app, its Gatus
check, the DNS record and the node001 tunnel entry.
2026-10-22 09:06:52 +02:00
Ruben Hensen 842f087da7 mbgwp: start again on PHP 8.3.36
wordpress:7.1.3-php8.3-apache was rebuilt at 15:57 UTC on PHP 8.3.36,
which fixes CVE-2026-109618 and CVE-2026-109619. WordPress core lives on
the volume, so the tag only moves PHP (8.3.30 -> 8.3.36) and Apache. Its
ca-certificates (20250419+deb13u1) no longer carries the roots above the
compromised eMudhra intermediates either.
2026-10-21 20:36:53 +02:00
Ruben Hensen 307bd23500 freshrss: start again on 1.30.3 with fixed PHP
freshrss/freshrss:1.30.3, released today, runs PHP 8.4.27 from Debian
trixie security (8.4.27-1~deb13u1), which fixes CVE-2026-109618 and
CVE-2026-109619. Pin it instead of :latest so the running version is the
checked one. Its ca-certificates (20250419+deb13u1) also no longer
carries the roots above the compromised eMudhra intermediates.
2026-10-21 17:43:08 +02:00
Ruben Hensen 6d9e405126 Point Velero and the NAS services back at 192.168.1.99
Reverts 8c2bd89. The NAS had fallen back to a DHCP address (.10), which
also moved the UniFi controller it hosts away from where the gateway
and access points inform, and left the Longhorn backup target at
nfs://192.168.1.99 unreachable. It now has 192.168.1.99 set statically
in DSM again, so everything that pointed at .99 works without changes.
2026-10-21 15:25:16 +02:00
Ruben Hensen dfdc4451fb velero: raise the server memory limit to 2Gi
At 512Mi the server has been OOMKilled 314 times. With the bucket
reachable again it died 23 seconds into a manual backup of ~14.7k items,
which then got marked Failed on restart. opti1 has 32GB.
2026-10-21 12:33:31 +02:00
Ruben Hensen e19f4b8db5 velero: skip the CRD upgrade hook
The hook's init container pulls docker.io/bitnami/kubectl:1.32, which
Bitnami has removed, so the sync for the new NAS address hung on
ImagePullBackOff. The chart stays at 8.4.0 and the CRDs are already in
place.
2026-10-21 12:22:57 +02:00
Ruben Hensen 8c2bd89444 Point Velero and the NAS services at 192.168.1.10
The NAS moved from 192.168.1.99 to .10, so the backup storage location
has been Unavailable ("no route to host"). Every backup since at least
26 July failed, and 154 expired ones are stuck in Deleting because Velero
can't reach the bucket to remove them. MinIO answers on .10:9000 and DSM
on .10:5000.
2026-10-21 12:16:19 +02:00
Ruben Hensen 400d46c462 actual-budget: use a CA bundle without the compromised eMudhra roots
Node uses its built-in roots, which still include emSign ECC Root CA - G3,
and the Node fix is still pending. Switch to OpenSSL's store with
--use-openssl-ca and point it at patched-ca-bundle, mounted over
/etc/ssl/certs. That covers bank sync and the authentik OIDC calls.
2026-10-21 12:09:41 +02:00
Ruben Hensen 25a6bf3a05 authentik: mount a CA bundle without the compromised eMudhra roots
Server and worker trusted the leaked eMudhra intermediates through the
Debian bundle (ca-certificates 20250419), Python's ssl module and certifi
2026.01.04. Mount patched-ca-bundle over /etc/ssl/certs and point
SSL_CERT_FILE and REQUESTS_CA_BUNDLE at it until an authentik image ships
a fixed bundle and certifi.
2026-10-21 12:09:41 +02:00
Ruben Hensen a507282103 argocd: mount a CA bundle without the compromised eMudhra roots
The Argo CD v2.14.2 image (Ubuntu 24.04, ca-certificates 20240203) and the
dex image still trust the eMudhra intermediates whose keys leaked
(https://emudhra.we-have-a-problem.com), and the repo server fetches git
and Helm repos over that trust store. A forged certificate there means
arbitrary manifests in the cluster.

patched-ca-bundle holds the bundle from Alpine ca-certificates-bundle
20261020-r0 and replaces /etc/ssl/certs in every Argo CD container except
redis. The whole directory goes, since OpenSSL and Go also read the
per-root files next to ca-certificates.crt. It uses server-side apply to
keep the bundle out of the last-applied annotation.
2026-10-21 12:09:41 +02:00
Ruben Hensen 19eacd35f7 Remove ip-updater
Every run has failed with "Permission denied (publickey)" on
root@46.224.26.65, the last successful job was 219 days ago, and one pod
had been hanging for 138 days. The home IP is pinned in
nix-infra-machine/nodes/node001.nix instead, so nothing used it.

It also ran alpine:3.19, which is EOL and still trusts the compromised
eMudhra intermediates.
2026-10-21 11:49:28 +02:00
Ruben Hensen bb7e2f2bac node001: distrust the roots above the compromised eMudhra intermediates
The private keys of eight eMudhra intermediates are recoverable
(https://emudhra.we-have-a-problem.com). node001 runs NixOS 25.11 from
March with nss-cacert 3.121, and curl there still accepted a chain via
emSign ECC Root CA - G3. Blacklist that root and Comodo AAA Services root
until nixpkgs ships NSS 3.131.1. Everything node001 talks to chains to
ISRG Root X1.

Already deployed with ./cli update node001, and Gatus, ntfy and Stalwart
were restarted so they reload the trust store.
2026-10-21 11:49:28 +02:00
Ruben Hensen 1efa132f9d Stop FreshRSS + MBG WordPress until PHP is patched
CERT-EU 2026-018 (CVE-2026-109618, CVE-2026-109619, CVSS 9.2): possible
unauthenticated RCE in PHP 5 and later. Both are internet-facing and are
the only PHP workloads (FreshRSS on PHP 8.2.29, WordPress on 8.3.30), so
scale them to 0 until their images ship a fixed PHP. PVCs and the MariaDB
cluster stay as they are.
2026-10-21 09:18:24 +02:00
Ruben Hensen 054117ad6d vault: point External Secrets at the in-cluster Vault service
The store went through https://vault.rubenhensen.nl, so via Hetzner and
the home internet connection. When home internet dropped, the store
validation failed and External Secrets backed off long enough that it
needed a controller restart after the connection came back.

vault-active.vault.svc talks to the active Vault pod directly. It's plain
HTTP, same as the ingress → Vault hop already is.
2026-10-20 15:45:55 +02:00
Ruben Hensen ac61c346af node001: run ntfy at ntfy1.rubenhensen.nl and alert from s1
ntfy listens on 127.0.0.1:2586 behind the same SNI-to-8444 setup as s1.
Users and tokens come from encrypted nix-infra secrets, exported as env
vars by a small wrapper since ntfy and Gatus only read them from the
environment.

s1 alerts on the home services, Scouting MBG and s2. It stays quiet on
mail, which lives on this box and is covered by s2.
2026-10-20 15:34:42 +02:00
Ruben Hensen e23779044f gatus: alert through self-hosted ntfy
Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at
the home IP like s2. s2 now alerts on mail and on s1 through it. The
home services are left to s1, which sees them from outside.

ntfy denies everything except the provisioned user. The user, its
password hash and the Gatus token come from Vault kv/ntfy.
2026-10-20 15:34:42 +02:00
Ruben Hensen 53a079dc22 node001: run Gatus at s1.rubenhensen.nl
Gatus listens on 127.0.0.1:8090. nginx gets s1 in the SNI map, sends it
to a local TLS vhost on 8444 with its own ACME cert, and proxies to
Gatus from there. The endpoint list is built from tunneledHosts plus
mail, Scouting MBG and s2, matching gatus/config.yaml.
2026-10-20 11:03:58 +02:00
Ruben Hensen 3ffb10b627 gatus: add s2.rubenhensen.nl status page on the home cluster
Gatus with sqlite history, checking every public service plus its twin
s1.rubenhensen.nl on Hetzner. Replaces the GitHub-hosted Upptime page.

DNS gets s1 (Hetzner) and s2. s2 points straight at the home IP rather
than through the Hetzner tunnel, so it stays reachable when Hetzner is
down.

gatus/README.md explains how to read the two pages and writes down the
plan for a single self-hosted failover URL (PowerDNS LUA records).
2026-10-20 11:03:58 +02:00
71 changed files with 4175 additions and 1241 deletions

No files matched your search

+14
View File
@@ -36,6 +36,20 @@ spec:
extraEnvVars:
ACTUAL_OPENID_SERVER_HOSTNAME: https://ynab.rubenhensen.nl
# Node trusts its built-in roots, which still include the ones
# above the compromised eMudhra intermediates. Use patched-ca-bundle
# through OpenSSL instead until Node ships a fix.
NODE_OPTIONS: --use-openssl-ca
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
# Service configuration
service:
+19
View File
@@ -86,8 +86,18 @@ spec:
secretKeyRef:
name: actualbudget-oidc-client-secret
key: client_secret
# Point OpenSSL and requests (which uses certifi otherwise) at
# patched-ca-bundle, mounted below until the image drops the
# compromised eMudhra intermediates.
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca-certificates.crt
- name: REQUESTS_CA_BUNDLE
value: /etc/ssl/certs/ca-certificates.crt
server:
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
- name: blueprint-vault
configMap:
name: authentik-blueprint-vault
@@ -131,6 +141,9 @@ spec:
- name: blueprint-session
mountPath: /blueprints/custom/session-duration.yaml
subPath: session-duration.yaml
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
ingress:
enabled: true
ingressClassName: nginx
@@ -144,6 +157,9 @@ spec:
- authentik.rubenhensen.nl
worker:
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
- name: blueprint-vault
configMap:
name: authentik-blueprint-vault
@@ -187,6 +203,9 @@ spec:
- name: blueprint-session
mountPath: /blueprints/custom/session-duration.yaml
subPath: session-duration.yaml
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
postgresql:
enabled: false
redis:
@@ -1,7 +1,7 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ip-updater
name: ca-bundle
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
@@ -10,13 +10,12 @@ spec:
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: ip-updater
path: ca-bundle
destination:
server: https://kubernetes.default.svc
namespace: ip-updater
# Every ConfigMap sets its own namespace.
namespace: argocd
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
prune: true
+11
View File
@@ -18,6 +18,17 @@ spec:
parameters:
- name: crds.enabled
value: 'true'
# The controller talks ACME to Let's Encrypt. Give it a trust store
# without the compromised eMudhra intermediates, see ca-bundle/.
values: |
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
syncPolicy:
syncOptions:
- CreateNamespace=true
@@ -1,7 +1,7 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ghost-blog
name: gatus-folder
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
@@ -10,10 +10,10 @@ spec:
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: ghost-blog
path: gatus
destination:
server: https://kubernetes.default.svc
namespace: ghost-blog
namespace: gatus
syncPolicy:
syncOptions:
- CreateNamespace=true
+5 -1
View File
@@ -16,13 +16,17 @@ spec:
sources:
- chart: longhorn
repoURL: https://charts.longhorn.io/
targetRevision: 1.13.0 # Replace with the Longhorn version you'd like to install or upgrade to
targetRevision: 1.11.2 # Replace with the Longhorn version you'd like to install or upgrade to
helm:
values: |
persistence:
defaultClassReplicaCount: 1
preUpgradeChecker:
jobEnabled: false
defaultSettings:
# Move volumes to the new engine image after each Longhorn
# upgrade, one at a time. At 0 they stayed behind on 1.7.1/1.7.2.
concurrentAutomaticEngineUpgradePerNodeLimit: 1
parameters:
- name: csi.kubeletRootDir
value: "/var/snap/microk8s/common/var/lib/kubelet"
+4 -2
View File
@@ -16,11 +16,13 @@ spec:
chart: wordpress
helm:
values: |
# Official WordPress image configuration
# Official WordPress image configuration. WordPress core lives on the
# volume, so the tag only sets PHP and Apache; 7.1.3-php8.3-apache
# ships PHP 8.3.36, which fixes CVE-2026-109618 and CVE-2026-109619.
image:
registry: docker.io
repository: wordpress
tag: "6.9.1-apache"
tag: "7.1.3-php8.3-apache"
pullPolicy: IfNotPresent
# Disable embedded MariaDB
-21
View File
@@ -1,21 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: nas
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: nas
destination:
server: https://kubernetes.default.svc
namespace: nas
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
@@ -1,7 +1,7 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: lingo
name: ntfy-folder
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
@@ -10,10 +10,10 @@ spec:
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: lingo
path: ntfy
destination:
server: https://kubernetes.default.svc
namespace: lingo
namespace: ntfy
syncPolicy:
syncOptions:
- CreateNamespace=true
-21
View File
@@ -1,21 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: phocas-lustrum
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: phocas-lustrum
destination:
server: https://kubernetes.default.svc
namespace: phocas-lustrum
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
-22
View File
@@ -1,22 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: stalwart
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: stalwart
destination:
server: https://kubernetes.default.svc
namespace: stalwart
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
-21
View File
@@ -1,21 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: whisper
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/rubenhensen/k8scd.git
targetRevision: HEAD
path: whishper
destination:
server: https://kubernetes.default.svc
namespace: whisper
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
+2 -1
View File
@@ -18,4 +18,5 @@ patches:
name: argocd-notifications-controller
path: overlays/argocd-notifications-controller-patch.yaml
- path: overlays/argocd-cm.patch.yaml
- path: overlays/argocd-rbac-cm.patch.yaml
- path: overlays/argocd-rbac-cm.patch.yaml
- path: overlays/patched-ca-bundle.patch.yaml
@@ -0,0 +1,112 @@
# Mount patched-ca-bundle (from ca-bundle/) over /etc/ssl/certs so git, helm
# and the Go binaries stop trusting the compromised eMudhra intermediates.
# The whole directory is replaced because OpenSSL and Go also read the
# per-root files next to ca-certificates.crt. Drop this once Argo CD ships a
# fixed image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-server
spec:
template:
spec:
containers:
- name: argocd-server
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: argocd-repo-server
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: argocd-application-controller
spec:
template:
spec:
containers:
- name: argocd-application-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-applicationset-controller
spec:
template:
spec:
containers:
- name: argocd-applicationset-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-notifications-controller
spec:
template:
spec:
containers:
- name: argocd-notifications-controller
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-dex-server
spec:
template:
spec:
containers:
- name: dex
volumeMounts:
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
@@ -69,6 +69,13 @@ spec:
optional: true
- name: VAPID_SUBJECT
value: mailto:ruben.hensen@protonmail.com
# Node trusts its built-in roots, which still include the ones
# above the compromised eMudhra intermediates. Use
# patched-ca-bundle through OpenSSL instead until Node ships a fix.
- name: NODE_OPTIONS
value: --use-openssl-ca
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca-certificates.crt
resources:
requests:
memory: "128Mi"
@@ -91,8 +98,14 @@ spec:
volumeMounts:
- mountPath: /app/data
name: data
- mountPath: /etc/ssl/certs
name: patched-ca-bundle
readOnly: true
restartPolicy: Always
volumes:
- name: data
persistentVolumeClaim:
claimName: bolletjestrui-data
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
+5
View File
@@ -0,0 +1,5 @@
apiVersion: v2
name: ca-bundle
description: Puts the patched-ca-bundle ConfigMap in every namespace listed in values.yaml
type: application
version: 0.1.0
File diff suppressed because it is too large. Load diff
+20
View File
@@ -0,0 +1,20 @@
{{- /*
Temporary CA bundle: ca-certificates.crt is /etc/ssl/certs/ca-certificates.crt
from Alpine ca-certificates-bundle 20261020-r0, which drops the roots above
the compromised eMudhra intermediates (https://emudhra.we-have-a-problem.com).
Workloads mount it over /etc/ssl/certs until their images ship a fixed
bundle. Server-side apply keeps the bundle out of the last-applied annotation.
*/}}
{{- range .Values.namespaces }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: patched-ca-bundle
namespace: {{ . }}
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
data:
ca-certificates.crt: |
{{ $.Files.Get "ca-certificates.crt" | indent 4 }}
{{- end }}
+12
View File
@@ -0,0 +1,12 @@
# Namespaces that get the patched-ca-bundle ConfigMap.
namespaces:
- actual-budget
- argocd
- authentik
- bolletjestrui
- cert-manager
- gatus
- gitea
- immich
- ntfy
- serpbear
+19 -12
View File
@@ -7,10 +7,6 @@ records:
expire: 300
type: A
content: "46.224.26.65"
- name: "phocaslustrum"
expire: 300
type: A
content: "62.41.86.27"
- name: "ynab"
expire: 300
type: A
@@ -43,14 +39,6 @@ records:
expire: 300
type: A
content: "46.224.26.65"
- name: "lingo"
expire: 300
type: A
content: "46.224.26.65"
- name: "blog"
expire: 300
type: A
content: "46.224.26.65"
- name: "serpbear"
expire: 300
type: A
@@ -63,6 +51,25 @@ records:
expire: 300
type: A
content: "46.224.26.65"
# Gatus status pages. s2 points straight at home, not through the
# Hetzner tunnel, so it stays reachable when Hetzner is down.
- name: "s1"
expire: 300
type: A
content: "46.224.26.65"
- name: "s2"
expire: 300
type: A
content: "62.41.86.27"
# ntfy for Gatus alerts, one next to each Gatus. Same reasoning as s2.
- name: "ntfy1"
expire: 300
type: A
content: "46.224.26.65"
- name: "ntfy2"
expire: 300
type: A
content: "62.41.86.27"
- name: "@"
expire: 300
type: MX
+78
View File
@@ -0,0 +1,78 @@
# Upgrade to Kubernetes 1.33
The cluster runs microk8s 1.32, which is out of upstream support, and Envoy
Gateway v1.9 (the planned ingress-nginx replacement) needs 1.33 or newer.
Several components are too old for 1.33 and have to move first, most of
them one minor version at a time. Everything up to step 7 happens on 1.32.
Compatibility research from 2026-10-22. Recheck release notes before each
step; versions move.
## 1. Preparation
- [ ] Every Longhorn volume on engine image v1.7.3, old engine images deleted
- [ ] Longhorn system backup
- [ ] Renovate held at Longhorn 1.11.x (1.11.3 and 1.13 need k8s 1.34)
## 2. Longhorn 1.7.3 → 1.11.2
One minor per step, engine images upgraded after each one. Check volumes,
the NFS backup target, the recurring backup job and the Velero `bak`
snapshots after every step. Fresh Velero backup before starting.
- [ ] 1.8.2: `backup-target` settings move to a `default` BackupTarget
([notes](https://longhorn.io/docs/1.8.2/important-notes/))
- [ ] 1.9.2
- [ ] Migrate every Longhorn CRD to `v1beta2` only in `storedVersions`.
Mandatory before 1.10; skipping it leaves only a forced downgrade
([notes](https://longhorn.io/docs/1.10.2/important-notes/))
- [ ] 1.10.2: v1beta1 API removed, settings move to a v1/v2 JSON format
- [ ] 1.11.2. Not 1.11.3 or 1.12.x: those need k8s 1.34
## 3. Smaller components
- [ ] CloudNativePG 1.25.1 → 1.26 → 1.27 → 1.28.4 (each step restarts the
single-instance databases)
- [ ] Velero 1.15.2 → 1.16.x (chart 10.x), plugin-for-aws v1.12
- [ ] Vault chart 0.28.1 → ≥0.30.1 (vault-k8s 1.7.0)
- [ ] cert-manager 1.15.5 → 1.16.5 → 1.17.4 → 1.18.6 → 1.19.6 → 1.20.4
(1.18 changes the `rotationPolicy` default to Always)
## 4. External Secrets 0.10.7 → ≥0.17
- [ ] Move every ExternalSecret and SecretStore to `external-secrets.io/v1`
- [ ] Upgrade one minor at a time; 0.16 needs v1alpha1 gone from
`storedVersions`, 0.17 stops serving v1beta1, 0.19 needs server-side apply
## 5. Argo CD 2.14.2 → 3.4.12
- [ ] Read every upgrade note from
[2.14→3.0](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.14-3.0/)
onward. 3.0 switches to annotation tracking and excludes Endpoints and
EndpointSlices by default (affects `home-assistant/service.yaml`); 3.3
needs `kubectl apply --server-side --force-conflicts`
## 6. Calico 3.25.1 → 3.29.3
- [ ] Apply the manifest microk8s 1.33 ships
(`/snap/microk8s/current/upgrade-scripts/000-switch-to-calico/resources/calico.yaml`
after the refresh, or the 1.33 branch of canonical/microk8s before it).
A snap refresh does not upgrade Calico.
## 7. microk8s 1.32 → 1.33
- [ ] Fresh Velero backup and `sudo microk8s dbctl backup`
- [ ] `sudo snap refresh microk8s --channel=1.33/stable`
- [ ] `sudo microk8s addons repo update core`
- [ ] If kubelite panics with "Missing actuated resource record"
([#5057](https://github.com/canonical/microk8s/issues/5057)):
`sudo snap revert microk8s`
## 8. After 1.33
- [ ] cert-manager 1.21.2
- [ ] Envoy Gateway v1.9 instead of ingress-nginx, Longhorn behind authentik
OIDC
Out of scope here but due: the immich PostgreSQL 14 cluster reaches end of
life on 2026-11-12.
+3 -1
View File
@@ -72,7 +72,9 @@ spec:
value: "server-cache:persistent"
- name: TZ
value: Europe/Amsterdam
image: freshrss/freshrss:latest
# 1.30.3 ships PHP 8.4.27, which fixes CVE-2026-109618 and
# CVE-2026-109619.
image: freshrss/freshrss:1.30.3
name: freshrss
ports:
- containerPort: 80
+105
View File
@@ -0,0 +1,105 @@
# Gatus
Two Gatus instances check the same endpoints from two places:
| Status page | Runs on | Config |
|---|---|---|
| https://s1.rubenhensen.nl | Hetzner (node001) | `nix-infra-machine/nodes/node001.nix` |
| https://s2.rubenhensen.nl | Home cluster | `gatus/config.yaml` (this folder) |
Each one also checks the other's `/health`. The s2 DNS record points straight
at the home IP instead of going through the Hetzner tunnel, so s2 stays
reachable when Hetzner is down.
The endpoint list exists twice, so a new service has to be added in both
places. s1 builds its list from `tunneledHosts`, so new tunneled hosts show up
there automatically. s2 needs them added by hand.
## Alerting
Each Gatus sends alerts to an ntfy instance on its own node:
`ntfy1.rubenhensen.nl` (Hetzner, node001.nix) and `ntfy2.rubenhensen.nl`
(home, `ntfy/` in this repo). Subscribe the phone app to topic `alerts` on
both servers, with the same login on each.
Only the watcher that sees a service from outside alerts on it:
- s1 alerts on everything that runs at home (all tunneled apps, Scouting MBG)
and on s2.
- s2 alerts on mail and on s1.
An outage on either node still reaches you, through the other node's ntfy,
and you get one notification per outage instead of two. An alert fires after
3 failed checks in a row (about 3 minutes), and you get another notification
when the service recovers.
Both ntfy servers deny everything except the provisioned user. The
credentials are the same on both:
- Home: Vault `kv/ntfy` with `auth_users`, `auth_tokens` and `gatus_token`,
pulled in by External Secrets.
- Hetzner: the nix-infra secrets `ntfy-auth-users`, `ntfy-auth-tokens` and
`ntfy-gatus-token`.
## Reading the two pages
- s1 sees the homelab from outside, so trust it for homelab services. s2 sees
Hetzner from outside, so trust it for mail and anything else on Hetzner.
- Every `*.rubenhensen.nl` app is tunneled Hetzner → home, so a Hetzner
outage shows all of them as down on both pages. That is accurate: nobody
can reach them then.
- If s1 and s2 disagree about an endpoint, the problem is the network path
between one watcher and that service, not the service itself.
- Neither page has history for the time its own node was down.
## Later: one URL with self-hosted DNS failover
This is the plan for replacing s1/s2 with a single `status.rubenhensen.nl`
that always points at a watcher that's up, with no third party involved.
Hand `status.rubenhensen.nl` over to two nameservers we run ourselves, one per
node, both PowerDNS Authoritative. PowerDNS LUA records health-check
addresses and only return the healthy ones:
```
@ 30 IN LUA A "ifurlup('https://status.rubenhensen.nl/health', {{'46.224.26.65'}, {'62.41.86.27'}})"
```
The double braces make two groups. PowerDNS returns the first group that has
a healthy address, so Hetzner is primary and home is the fallback. Each
nameserver runs the check from its own location.
| Situation | Hetzner NS | Home NS | Visitors end up on |
|---|---|---|---|
| Both up | answers Hetzner | answers Hetzner | Hetzner |
| Hetzner down | dead | Hetzner check fails, answers home | home |
| Home down / home internet down | answers Hetzner | dead | Hetzner |
| Hetzner up, its Gatus broken | own check fails, answers home | answers home | home |
A node that goes down takes its nameserver down with it, so DNS resolvers
switch to the other nameserver. The two nodes never have to agree with each
other: the answer depends on what the internet can reach. The 30s TTL caps
how long clients keep a stale answer.
Steps:
1. Parent zone (`dns/domains/rubenhensen.nl.yaml`, synced to TransIP): add
`status NS ns-hetzner.rubenhensen.nl.` and `status NS ns-home.rubenhensen.nl.`,
plus A records for `ns-hetzner` (46.224.26.65) and `ns-home` (62.41.86.27).
First check that TransIP accepts NS records for a subdomain.
2. Hetzner: run PowerDNS from node001.nix with `enable-lua-records=yes` and
the bind backend. Use the same zone file as home. Open 53/udp and 53/tcp.
3. Home: run PowerDNS in the cluster with the same zone file, on 53/udp and
53/tcp of the home IP (router port forward, and check how the cluster
exposes UDP).
4. Both Gatus instances also serve `status.rubenhensen.nl`. On Hetzner, add the
name to the SNI map, the same way as s1.
5. TLS: Hetzner gets its cert through HTTP-01 as usual. On Hetzner's :80 vhost
for `status`, serve local ACME challenges and proxy the rest to the home
cluster (`try_files $uri @home`). That way cert-manager's HTTP-01 also
works while DNS points at Hetzner. When Hetzner is down, DNS points home
and cert-manager reaches the home cluster directly.
6. Requirements: a static home IP (`homeIP` in node001.nix already assumes
one), and port 53 open on both nodes.
After that, s1/s2 can be removed, or kept as direct links to each instance.
+136
View File
@@ -0,0 +1,136 @@
# s2.rubenhensen.nl: Gatus on the home cluster.
# Its twin s1.rubenhensen.nl runs on the Hetzner box, see
# nix-infra-machine/nodes/node001.nix. Keep the endpoint lists in sync.
storage:
type: sqlite
path: /data/data.db
ui:
title: Status (homelab)
header: s2 · homelab
buttons:
- name: s1 (Hetzner)
link: https://s1.rubenhensen.nl
# NTFY_TOKEN comes from the gatus-ntfy secret (Vault kv/ntfy)
alerting:
ntfy:
url: http://ntfy.ntfy.svc.cluster.local
topic: alerts
token: ${NTFY_TOKEN}
priority: 4
click: https://s2.rubenhensen.nl
default-alert:
failure-threshold: 3
success-threshold: 2
send-on-resolved: true
# s2 is the outside view of Hetzner, so it alerts on mail and on s1. The
# home services are left to s1.
endpoints:
# Tunneled through Hetzner to the home cluster
- name: rss
group: rubenhensen.nl
url: https://rss.rubenhensen.nl
interval: 60s
# mod_auth_openidc answers 401 instead of redirecting to the login
# page unless the client asks for HTML
headers:
Accept: text/html
conditions: &http-ok
- "[STATUS] == 200"
- "[CERTIFICATE_EXPIRATION] > 168h"
- name: authentik
group: rubenhensen.nl
url: https://authentik.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: vault
group: rubenhensen.nl
url: https://vault.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: ynab
group: rubenhensen.nl
url: https://ynab.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: argocd
group: rubenhensen.nl
url: https://argocd.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: ha
group: rubenhensen.nl
url: https://ha.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: longhorn
group: rubenhensen.nl
url: https://longhorn.rubenhensen.nl
interval: 60s
# Behind basic auth, so 401 means it's up
conditions:
- "[STATUS] == 401"
- "[CERTIFICATE_EXPIRATION] > 168h"
- name: serpbear
group: rubenhensen.nl
url: https://serpbear.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: git
group: rubenhensen.nl
url: https://git.rubenhensen.nl
interval: 60s
conditions: *http-ok
- name: vic
group: rubenhensen.nl
url: https://vic.rubenhensen.nl
interval: 60s
conditions: *http-ok
# Stalwart on Hetzner
- name: webmail
group: mail
url: https://mail.rubenhensen.nl
interval: 60s
conditions: *http-ok
alerts: &ntfy
- type: ntfy
- name: smtp
group: mail
url: tcp://mail.rubenhensen.nl:25
interval: 60s
conditions:
- "[CONNECTED] == true"
alerts: *ntfy
- name: imaps
group: mail
url: tls://mail.rubenhensen.nl:993
interval: 60s
conditions:
- "[CONNECTED] == true"
- "[CERTIFICATE_EXPIRATION] > 168h"
alerts: *ntfy
# Served straight from the home IP
- name: scoutingmbg.nl
group: Scouting MBG
url: https://scoutingmbg.nl
interval: 60s
conditions: *http-ok
- name: foto.scoutingmbg.nl
group: Scouting MBG
url: https://foto.scoutingmbg.nl/_app
interval: 60s
conditions: *http-ok
# The other watcher
- name: s1 (Hetzner)
group: watchers
url: https://s1.rubenhensen.nl/health
interval: 60s
conditions:
- "[STATUS] == 200"
- "[BODY].status == UP"
alerts: *ntfy
@@ -1,12 +1,10 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
io.kompose.service: mongo-claim10
name: mongo-claim10
name: gatus-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storage: 1Gi
@@ -1,17 +1,17 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ip-updater-ssh-key
name: gatus-ntfy
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: ip-updater-ssh-key
name: gatus-ntfy
creationPolicy: Owner
data:
- secretKey: private_key
- secretKey: NTFY_TOKEN
remoteRef:
key: kv/ip-updater
property: ssh_private_key
key: kv/ntfy
property: gatus_token
+65
View File
@@ -0,0 +1,65 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: gatus
spec:
replicas: 1
selector:
matchLabels:
app: gatus
strategy:
type: Recreate
template:
metadata:
labels:
app: gatus
spec:
containers:
- name: gatus
image: twinproduction/gatus:v5.37.0
env:
- name: GATUS_CONFIG_PATH
value: /config/config.yaml
- name: TZ
value: Europe/Amsterdam
envFrom:
- secretRef:
name: gatus-ntfy
ports:
- containerPort: 8080
protocol: TCP
readinessProbe:
httpGet:
path: /health
port: 8080
livenessProbe:
httpGet:
path: /health
port: 8080
resources:
requests:
memory: "64Mi"
cpu: "20m"
limits:
memory: "256Mi"
volumeMounts:
- name: config
mountPath: /config
readOnly: true
- name: data
mountPath: /data
# Trust store without the compromised eMudhra intermediates,
# see ca-bundle/. Drop once the image ships a fixed bundle.
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: config
configMap:
name: gatus-config
- name: data
persistentVolumeClaim:
claimName: gatus-data
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Service
metadata:
name: gatus
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: gatus
+7 -11
View File
@@ -1,7 +1,7 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nas-ingress
name: gatus-ingress
annotations:
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
@@ -9,21 +9,17 @@ metadata:
spec:
ingressClassName: nginx
rules:
- host: nas.rubenhensen.nl
- host: s2.rubenhensen.nl
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nas-external
port:
number: 5000
# TLS configuration for HTTPS
name: gatus
port:
number: 8080
tls:
# TLS secretName used on ClusterIssuer
- secretName: letsencrypt-prod
- secretName: letsencrypt-prod
hosts:
- nas.rubenhensen.nl
- s2.rubenhensen.nl
+16
View File
@@ -0,0 +1,16 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: gatus
resources:
- external-secret.yaml
- data-persistentvolumeclaim.yaml
- gatus-deployment.yaml
- gatus-service.yaml
- ingress.yaml
configMapGenerator:
- name: gatus-config
files:
- config.yaml
-42
View File
@@ -1,42 +0,0 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ghost-mail-config
namespace: ghost-blog
spec:
refreshInterval: "15m"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: ghost-mail-config
creationPolicy: Owner
data:
- secretKey: mail__transport
remoteRef:
key: kv/ghost-blog/config
property: email_transport
- secretKey: mail__from
remoteRef:
key: kv/ghost-blog/config
property: email_from
- secretKey: mail__options__service
remoteRef:
key: kv/ghost-blog/config
property: email_service
- secretKey: mail__options__host
remoteRef:
key: kv/ghost-blog/config
property: email_host
- secretKey: mail__options__port
remoteRef:
key: kv/ghost-blog/config
property: email_port
- secretKey: mail__options__auth__user
remoteRef:
key: kv/ghost-blog/config
property: email_user
- secretKey: mail__options__auth__pass
remoteRef:
key: kv/ghost-blog/config
property: email_password
-165
View File
@@ -1,165 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ghost-blog
namespace: ghost-blog
labels:
app: ghost-blog
app.kubernetes.io/name: ghost-blog
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: ghost
app.kubernetes.io/part-of: ghost-blog
spec:
replicas: 1
selector:
matchLabels:
app: ghost-blog
minReadySeconds: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 3
revisionHistoryLimit: 4
progressDeadlineSeconds: 600
template:
metadata:
namespace: ghost-blog
labels:
app: ghost-blog
spec:
automountServiceAccountToken: false
volumes:
- name: ghost-content
persistentVolumeClaim:
claimName: ghost-content
- name: tmp
emptyDir:
sizeLimit: 64Mi
initContainers:
- name: permissions-fix
imagePullPolicy: IfNotPresent
image: docker.io/busybox:stable-musl
env:
- name: GHOST_CONTENT
value: /var/lib/ghost/content
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
command:
- /bin/sh
- '-c'
- |
set -e
export DIRS='files logs apps themes data public settings images media'
echo 'Check if base dirs exists, if not, create them'
echo "Directories to check: $DIRS"
for dir in $DIRS; do
if [ ! -d $GHOST_CONTENT/$dir ]; then
echo "Creating $GHOST_CONTENT/$dir directory"
mkdir -pv $GHOST_CONTENT/$dir || echo "Error creating $GHOST_CONTENT/$dir directory"
fi
chown -Rfv 1000:1000 $GHOST_CONTENT/$dir && echo "chown ok on $dir" || echo "Error changing ownership of $GHOST_CONTENT/$dir directory"
done
exit 0
volumeMounts:
- name: ghost-content
mountPath: /var/lib/ghost/content
readOnly: false
containers:
- name: ghost-blog
image: ghost:5.114.1-alpine
imagePullPolicy: Always
ports:
- name: ghost
containerPort: 2368
protocol: TCP
readinessProbe:
httpGet:
path: /ghost/api/v4/admin/site/
port: ghost
httpHeaders:
- name: X-Forwarded-Proto
value: https
- name: Host
value: blog.rubenhensen.nl
periodSeconds: 10
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
initialDelaySeconds: 10
livenessProbe:
httpGet:
path: /ghost/api/v4/admin/site/
port: ghost
httpHeaders:
- name: X-Forwarded-Proto
value: https
- name: Host
value: blog.rubenhensen.nl
periodSeconds: 300
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 1
initialDelaySeconds: 30
env:
- name: NODE_ENV
value: production
- name: database__client
value: mysql
- name: database__connection__host
value: ghost-mysql-service
- name: database__connection__user
valueFrom:
secretKeyRef:
name: ghost-mysql-env
key: MYSQL_USER
- name: database__connection__password
valueFrom:
secretKeyRef:
name: ghost-mysql-env
key: MYSQL_PASSWORD
- name: database__connection__database
valueFrom:
secretKeyRef:
name: ghost-mysql-env
key: MYSQL_DATABASE
- name: url
value: https://blog.rubenhensen.nl
envFrom:
- secretRef:
name: ghost-mail-config
resources:
limits:
cpu: 800m
memory: 800Mi
requests:
cpu: 200m
memory: 400Mi
volumeMounts:
- name: ghost-content
mountPath: /var/lib/ghost/content
readOnly: false
- name: tmp
mountPath: /tmp
readOnly: false
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
restartPolicy: Always
terminationGracePeriodSeconds: 15
dnsPolicy: ClusterFirst
securityContext:
seccompProfile:
type: RuntimeDefault
-33
View File
@@ -1,33 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ghost-blog-ingress
namespace: ghost-blog
labels:
app: ghost-blog
app.kubernetes.io/name: ghost-blog-ingress
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: ingress
app.kubernetes.io/part-of: ghost-blog
annotations:
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/proxy-body-size: 1G
argocd.argoproj.io/sync-wave: "7"
spec:
ingressClassName: nginx
tls:
- hosts:
- blog.rubenhensen.nl
secretName: ghost-tls-secret
rules:
- host: blog.rubenhensen.nl
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ghost-service
port:
name: ghost
-36
View File
@@ -1,36 +0,0 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ghost-mysql-env
namespace: ghost-blog
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
refreshInterval: "15m"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: ghost-mysql-env
creationPolicy: Owner
data:
- secretKey: MYSQL_DATABASE
remoteRef:
key: kv/ghost-blog/mysql
property: database
- secretKey: MYSQL_USER
remoteRef:
key: kv/ghost-blog/mysql
property: username
- secretKey: MYSQL_PASSWORD
remoteRef:
key: kv/ghost-blog/mysql
property: password
- secretKey: MYSQL_ROOT_PASSWORD
remoteRef:
key: kv/ghost-blog/mysql
property: root_password
# - secretKey: MYSQL_HOST
# remoteRef:
# key: kv/ghost-blog/mysql
# property: host
-114
View File
@@ -1,114 +0,0 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ghost-mysql
namespace: ghost-blog
labels:
app: ghost-mysql
app.kubernetes.io/name: ghost-mysql
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: database
app.kubernetes.io/part-of: ghost-blog
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
serviceName: ghost-mysql-service
replicas: 1
selector:
matchLabels:
app: ghost-mysql
template:
metadata:
labels:
app: ghost-mysql
spec:
initContainers:
- name: ghost-mysql-init
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
image: docker.io/busybox:stable-musl
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -e
echo 'Changing ownership of mysql mount directory to 65534:65534'
chown -Rfv 65534:65534 /mnt/mysql || echo 'Error changing ownership of mysql mount directory to 65534:65534'
echo 'Changing ownership of tmp mount directory to 65534:65534'
chown -Rfv 65534:65534 /mnt/tmp || echo 'Error changing ownership of tmp mount directory to 65534:65534'
echo 'Changing ownership of socket mount directory to 65534:65534'
chown -Rfv 65534:65534 /mnt/var/run/mysqld || echo 'Error changing ownership of socket mount directory to 65534:65534'
volumeMounts:
- name: ghost-mysql-volume
mountPath: /mnt/mysql
subPath: mysql-empty-subdir
readOnly: false
- name: ghost-mysql-tmp
mountPath: /mnt/tmp
readOnly: false
- name: ghost-mysql-socket
mountPath: /mnt/var/run/mysqld
readOnly: false
resources:
requests:
memory: 50Mi
cpu: 50m
limits:
memory: 100Mi
cpu: 100m
containers:
- name: ghost-mysql
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
image: docker.io/mysql:8.4
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: ghost-mysql-env
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 500m
ports:
- containerPort: 3306
protocol: TCP
name: mysql
volumeMounts:
- name: ghost-mysql-volume
mountPath: /var/lib/mysql
subPath: mysql-empty-subdir
readOnly: false
- name: ghost-mysql-tmp
mountPath: /tmp
readOnly: false
- name: ghost-mysql-socket
mountPath: /var/run/mysqld
readOnly: false
automountServiceAccountToken: false
securityContext:
seccompProfile:
type: RuntimeDefault
volumes:
- name: ghost-mysql-volume
persistentVolumeClaim:
claimName: ghost-mysql-pvc
- name: ghost-mysql-tmp
emptyDir:
sizeLimit: 128Mi
- name: ghost-mysql-socket
emptyDir:
sizeLimit: 128Mi
-41
View File
@@ -1,41 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ghost-content
namespace: ghost-blog
labels:
app: ghost-blog
app.kubernetes.io/name: ghost-content
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: storage
app.kubernetes.io/part-of: ghost-blog
spec:
storageClassName: "longhorn"
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ghost-mysql-pvc
namespace: ghost-blog
labels:
app: ghost-mysql
app.kubernetes.io/name: ghost-mysql-pvc
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: database-storage
app.kubernetes.io/part-of: ghost-blog
spec:
storageClassName: "longhorn"
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
-44
View File
@@ -1,44 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: ghost-service
namespace: ghost-blog
labels:
app: ghost-blog
app.kubernetes.io/name: ghost-service
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: service-frontend
app.kubernetes.io/part-of: ghost-blog
spec:
ports:
- port: 2368
protocol: TCP
targetPort: ghost
name: ghost
type: ClusterIP
selector:
app: ghost-blog
---
apiVersion: v1
kind: Service
metadata:
name: ghost-mysql-service
namespace: ghost-blog
labels:
app: ghost-mysql
app.kubernetes.io/name: ghost-mysql-service
app.kubernetes.io/instance: ghost-blog
app.kubernetes.io/version: '5.92'
app.kubernetes.io/component: service-database
app.kubernetes.io/part-of: ghost-blog
spec:
ports:
- port: 3306
protocol: TCP
targetPort: mysql
name: mysql
type: ClusterIP
selector:
app: ghost-mysql
+8
View File
@@ -53,8 +53,16 @@ spec:
volumeMounts:
- mountPath: /data
name: data
# Trust store without the compromised eMudhra intermediates,
# see ca-bundle/. Drop once the image ships a fixed bundle.
- mountPath: /etc/ssl/certs
name: patched-ca-bundle
readOnly: true
restartPolicy: Always
volumes:
- name: data
persistentVolumeClaim:
claimName: gitea-data
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
+24 -4
View File
@@ -1,10 +1,30 @@
# Home Assistant runs outside the cluster on 192.168.1.202. ingress-nginx
# 1.13 resolves an ExternalName as a DNS name and fails on a bare IP, so
# point a selector-less Service at it with a hand-written EndpointSlice.
apiVersion: v1
kind: Service
metadata:
name: home-assistant-external
spec:
type: ExternalName
externalName: 192.168.1.202
ports:
- port: 8123
protocol: TCP
- name: http
port: 8123
protocol: TCP
targetPort: 8123
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: home-assistant-external-1
labels:
kubernetes.io/service-name: home-assistant-external
addressType: IPv4
ports:
- name: http
port: 8123
protocol: TCP
endpoints:
- addresses:
- 192.168.1.202
conditions:
ready: true
+25
View File
@@ -91,6 +91,20 @@ spec:
image:
repository: ghcr.io/immich-app/immich-server
pullPolicy: IfNotPresent
# Node trusts its built-in roots, which still include the
# ones above the compromised eMudhra intermediates. Use
# patched-ca-bundle through OpenSSL until Node ships a fix.
env:
NODE_OPTIONS: --use-openssl-ca
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
persistence:
patched-ca-bundle:
enabled: true
type: configMap
name: patched-ca-bundle
globalMounts:
- path: /etc/ssl/certs
readOnly: true
ingress:
main:
enabled: true
@@ -135,7 +149,18 @@ spec:
TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet
MPLCONFIGDIR: /cache/matplotlib-config
# Trust store without the compromised eMudhra
# intermediates; requests would otherwise use certifi.
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
persistence:
patched-ca-bundle:
enabled: true
type: configMap
name: patched-ca-bundle
globalMounts:
- path: /etc/ssl/certs
readOnly: true
cache:
enabled: true
size: 10Gi
-39
View File
@@ -1,39 +0,0 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: ip-updater
spec:
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
containers:
- name: ip-updater
image: alpine:3.19
command: ["/bin/sh", "-c"]
args:
- |
apk add --no-cache openssh-client curl > /dev/null 2>&1
cp /ssh-key/private_key /tmp/ssh_key
chmod 600 /tmp/ssh_key
IP=$(curl -s --max-time 10 https://api.ipify.org)
if [ -z "$IP" ]; then
echo "Failed to get external IP"
exit 1
fi
echo "Current external IP: $IP"
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-i /tmp/ssh_key root@46.224.26.65 "$IP"
volumeMounts:
- name: ssh-key
mountPath: /ssh-key
readOnly: true
volumes:
- name: ssh-key
secret:
secretName: ip-updater-ssh-key
defaultMode: 0400
restartPolicy: OnFailure
-32
View File
@@ -1,32 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: lingo-api
name: lingo-api
spec:
replicas: 1
selector:
matchLabels:
app: lingo-api
strategy:
type: Recreate
template:
metadata:
labels:
app: lingo-api
spec:
containers:
- image: ghcr.io/rubenhensen/lingo-api:master-4404fe5
name: lingo-api
ports:
- containerPort: 80
protocol: TCP
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
restartPolicy: Always
-13
View File
@@ -1,13 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels:
app: lingo-api
name: lingo-api
spec:
ports:
- name: "api"
port: 8000
targetPort: 80
selector:
app: lingo-api
-32
View File
@@ -1,32 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: lingo-frontend
name: lingo-frontend
spec:
replicas: 1
selector:
matchLabels:
app: lingo-frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: lingo-frontend
spec:
containers:
- image: ghcr.io/rubenhensen/lingo-frontend:master-46bddb5
name: lingo-frontend
ports:
- containerPort: 80
protocol: TCP
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
restartPolicy: Always
-13
View File
@@ -1,13 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels:
app: lingo-frontend
name: lingo-frontend
spec:
ports:
- name: "frontend"
port: 8001
targetPort: 80
selector:
app: lingo-frontend
-50
View File
@@ -1,50 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lingo-api-ingress
annotations:
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
ingressClassName: nginx
rules:
- host: lingo.rubenhensen.nl
http:
paths:
- path: /api(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: lingo-api
port:
number: 8000
tls:
- secretName: letsencrypt-prod
hosts:
- lingo.rubenhensen.nl
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lingo-frontend-ingress
annotations:
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
spec:
ingressClassName: nginx
rules:
- host: lingo.rubenhensen.nl
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: lingo-frontend
port:
number: 8001
tls:
- secretName: letsencrypt-prod
hosts:
- lingo.rubenhensen.nl
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: nas-external
spec:
type: ExternalName
externalName: 192.168.1.99
ports:
- name: nas-5000
protocol: TCP
port: 5000
- name: nas-5001
protocol: TCP
port: 5001
+198 -2
View File
@@ -16,8 +16,6 @@ let
"argocd.rubenhensen.nl"
"ha.rubenhensen.nl"
"longhorn.rubenhensen.nl"
"lingo.rubenhensen.nl"
"blog.rubenhensen.nl"
"serpbear.rubenhensen.nl"
"git.rubenhensen.nl"
"vic.rubenhensen.nl"
@@ -38,6 +36,75 @@ let
locations."/".proxyPass = "http://${homeIP}";
};
}) tunneledHosts);
# Gatus endpoints for s1.rubenhensen.nl. Its twin s2.rubenhensen.nl runs
# on the home cluster with the same list, see gatus/config.yaml in k8scd.
# Keep the two in sync.
gatusHttp = group: name: url: {
inherit name group url;
interval = "60s";
conditions = [
"[STATUS] == 200"
"[CERTIFICATE_EXPIRATION] > 168h"
];
};
# Per-host tweaks on top of the default check.
gatusOverrides = {
# Behind basic auth, so 401 means it's up
"longhorn.rubenhensen.nl".conditions = [
"[STATUS] == 401"
"[CERTIFICATE_EXPIRATION] > 168h"
];
# mod_auth_openidc answers 401 instead of redirecting to the login
# page unless the client asks for HTML
"rss.rubenhensen.nl".headers.Accept = "text/html";
};
# s1 is the outside view of home, so it alerts on what runs there and on
# s2. Mail lives on this box, so s2 alerts on that.
gatusAlert = { alerts = [ { type = "ntfy"; } ]; };
gatusEndpoints =
map (h:
gatusHttp "rubenhensen.nl" (lib.removeSuffix ".rubenhensen.nl" h) "https://${h}"
// gatusAlert
// (gatusOverrides.${h} or { })) tunneledHosts
++ [
(gatusHttp "mail" "webmail" "https://mail.rubenhensen.nl")
{
name = "smtp";
group = "mail";
url = "tcp://mail.rubenhensen.nl:25";
interval = "60s";
conditions = [ "[CONNECTED] == true" ];
}
{
name = "imaps";
group = "mail";
url = "tls://mail.rubenhensen.nl:993";
interval = "60s";
conditions = [ "[CONNECTED] == true" "[CERTIFICATE_EXPIRATION] > 168h" ];
}
(gatusHttp "Scouting MBG" "scoutingmbg.nl" "https://scoutingmbg.nl" // gatusAlert)
(gatusHttp "Scouting MBG" "foto.scoutingmbg.nl" "https://foto.scoutingmbg.nl/_app" // gatusAlert)
({
name = "s2 (homelab)";
group = "watchers";
url = "https://s2.rubenhensen.nl/health";
interval = "60s";
conditions = [ "[STATUS] == 200" "[BODY].status == UP" ];
} // gatusAlert)
];
# Runs a service with secrets from LoadCredentialEncrypted exported as
# env vars, for services that only take secrets through the environment.
withCredentialEnv = name: vars: cmd: pkgs.writeShellScript "${name}-with-credentials" ''
${lib.concatStrings (lib.mapAttrsToList (var: cred: ''
export ${var}="$(cat "$CREDENTIALS_DIRECTORY/${cred}")"
'') vars)}
exec ${cmd}
'';
in
{
# ──────────────────────────────────────────────
@@ -53,6 +120,17 @@ in
80 # HTTP (ACME)
];
# ──────────────────────────────────────────────
# Trust store
# ──────────────────────────────────────────────
# Distrust the roots above the compromised eMudhra intermediates until
# nixpkgs ships NSS 3.131.1, see https://emudhra.we-have-a-problem.com.
# Everything this box talks to chains to ISRG Root X1.
security.pki.caCertificateBlacklist = [
"Comodo AAA Services root"
"emSign ECC Root CA - G3"
];
# ──────────────────────────────────────────────
# ACME / Let's Encrypt
# ──────────────────────────────────────────────
@@ -86,9 +164,12 @@ in
# SNI passthrough on 443:
# - mail.rubenhensen.nl (and anything else) → local stalwart on 8443
# - tunneled hosts → home cluster ingress on 443
# - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy)
streamConfig = ''
map $ssl_preread_server_name $tunnel_upstream {
${sniMapEntries}
s1.rubenhensen.nl 127.0.0.1:8444;
ntfy1.rubenhensen.nl 127.0.0.1:8444;
default 127.0.0.1:8443;
}
@@ -137,6 +218,121 @@ ${sniMapEntries}
locations."/.well-known/acme-challenge/".root = "/var/lib/acme/acme-challenge";
locations."/".return = "301 https://$host$request_uri";
};
# Gatus status page. The stream block above passes its SNI to
# 8444, where nginx terminates TLS with its own ACME cert.
"s1.rubenhensen.nl" = {
listen = [
{ addr = "0.0.0.0"; port = 80; }
{ addr = "[::]"; port = 80; }
{ addr = "127.0.0.1"; port = 8444; ssl = true; }
];
enableACME = true;
forceSSL = true;
locations."/".proxyPass = "http://127.0.0.1:8090";
};
# ntfy for Gatus alerts, same TLS setup as s1. Proxy settings follow
# the ntfy docs so subscriptions can stay open.
"ntfy1.rubenhensen.nl" = {
listen = [
{ addr = "0.0.0.0"; port = 80; }
{ addr = "[::]"; port = 80; }
{ addr = "127.0.0.1"; port = 8444; ssl = true; }
];
enableACME = true;
forceSSL = true;
locations."/" = {
proxyPass = "http://127.0.0.1:2586";
proxyWebsockets = true;
extraConfig = ''
proxy_buffering off;
proxy_request_buffering off;
proxy_connect_timeout 3m;
proxy_send_timeout 3m;
proxy_read_timeout 3m;
client_max_body_size 0;
'';
};
};
};
};
# ──────────────────────────────────────────────
# ntfy (ntfy1.rubenhensen.nl)
# ──────────────────────────────────────────────
# Users and tokens come from secrets, everything else is denied. The
# same login works on ntfy2.rubenhensen.nl at home.
services.ntfy-sh = {
enable = true;
settings = {
base-url = "https://ntfy1.rubenhensen.nl";
listen-http = "127.0.0.1:2586";
behind-proxy = true;
auth-file = "/var/lib/ntfy-sh/user.db";
auth-default-access = "deny-all";
cache-file = "/var/lib/ntfy-sh/cache.db";
# Needed for instant delivery on iOS. ntfy.sh only gets the message
# ID, never the content.
upstream-base-url = "https://ntfy.sh";
};
};
systemd.services.ntfy-sh.serviceConfig = {
LoadCredentialEncrypted = [
"ntfy-auth-users:/root/secrets/[%%secrets/ntfy-auth-users%%]"
"ntfy-auth-tokens:/root/secrets/[%%secrets/ntfy-auth-tokens%%]"
];
ExecStart = lib.mkForce (withCredentialEnv "ntfy-sh" {
NTFY_AUTH_USERS = "ntfy-auth-users";
NTFY_AUTH_TOKENS = "ntfy-auth-tokens";
} "${config.services.ntfy-sh.package}/bin/ntfy serve -c /etc/ntfy/server.yml");
};
# ──────────────────────────────────────────────
# Gatus (s1.rubenhensen.nl)
# ──────────────────────────────────────────────
services.gatus = {
enable = true;
settings = {
web = {
address = "127.0.0.1";
port = 8090;
};
storage = {
type = "sqlite";
path = "/var/lib/gatus/data.db";
};
ui = {
title = "Status (Hetzner)";
header = "s1 · Hetzner";
buttons = [
{ name = "s2 (homelab)"; link = "https://s2.rubenhensen.nl"; }
];
};
alerting.ntfy = {
url = "http://127.0.0.1:2586";
topic = "alerts";
token = "\${NTFY_TOKEN}";
priority = 4;
click = "https://s1.rubenhensen.nl";
default-alert = {
failure-threshold = 3;
success-threshold = 2;
send-on-resolved = true;
};
};
endpoints = gatusEndpoints;
};
};
systemd.services.gatus = {
after = [ "ntfy-sh.service" ];
serviceConfig = {
LoadCredentialEncrypted = [
"ntfy-gatus-token:/root/secrets/[%%secrets/ntfy-gatus-token%%]"
];
ExecStart = lib.mkForce (withCredentialEnv "gatus" {
NTFY_TOKEN = "ntfy-gatus-token";
} (lib.getExe config.services.gatus.package));
};
};
@@ -1,12 +1,10 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
io.kompose.service: whishper-claim0
name: whishper-claim0
name: ntfy-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storage: 1Gi
+21
View File
@@ -0,0 +1,21 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: ntfy-auth
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: ntfy-auth
creationPolicy: Owner
data:
- secretKey: NTFY_AUTH_USERS
remoteRef:
key: kv/ntfy
property: auth_users
- secretKey: NTFY_AUTH_TOKENS
remoteRef:
key: kv/ntfy
property: auth_tokens
+28
View File
@@ -0,0 +1,28 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ntfy-ingress
annotations:
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
# Keep subscriptions open, as in the ntfy proxy docs
nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
nginx.ingress.kubernetes.io/proxy-send-timeout: "180"
spec:
ingressClassName: nginx
rules:
- host: ntfy2.rubenhensen.nl
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ntfy
port:
number: 80
tls:
- secretName: letsencrypt-prod
hosts:
- ntfy2.rubenhensen.nl
+11
View File
@@ -0,0 +1,11 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ntfy
resources:
- external-secret.yaml
- data-persistentvolumeclaim.yaml
- ntfy-deployment.yaml
- ntfy-service.yaml
- ingress.yaml
+74
View File
@@ -0,0 +1,74 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ntfy
spec:
replicas: 1
selector:
matchLabels:
app: ntfy
strategy:
type: Recreate
template:
metadata:
labels:
app: ntfy
spec:
containers:
- name: ntfy
image: binwiederhier/ntfy:v2.28.0
args: ["serve"]
env:
- name: NTFY_BASE_URL
value: https://ntfy2.rubenhensen.nl
- name: NTFY_LISTEN_HTTP
value: ":80"
- name: NTFY_BEHIND_PROXY
value: "true"
- name: NTFY_AUTH_FILE
value: /var/lib/ntfy/user.db
- name: NTFY_AUTH_DEFAULT_ACCESS
value: deny-all
- name: NTFY_CACHE_FILE
value: /var/lib/ntfy/cache.db
# Needed for instant delivery on iOS. ntfy.sh only gets the
# message ID, never the content.
- name: NTFY_UPSTREAM_BASE_URL
value: https://ntfy.sh
- name: TZ
value: Europe/Amsterdam
envFrom:
- secretRef:
name: ntfy-auth
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
httpGet:
path: /v1/health
port: 80
livenessProbe:
httpGet:
path: /v1/health
port: 80
resources:
requests:
memory: "32Mi"
cpu: "10m"
limits:
memory: "256Mi"
volumeMounts:
- name: data
mountPath: /var/lib/ntfy
# Trust store without the compromised eMudhra intermediates,
# see ca-bundle/. Drop once the image ships a fixed bundle.
- name: patched-ca-bundle
mountPath: /etc/ssl/certs
readOnly: true
volumes:
- name: data
persistentVolumeClaim:
claimName: ntfy-data
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
+11
View File
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Service
metadata:
name: ntfy
spec:
ports:
- name: http
port: 80
targetPort: 80
selector:
app: ntfy
+10 -1
View File
@@ -18,7 +18,9 @@
"/apps/templates/.+\\.yaml$/",
"/serpbear/.+\\.yaml$/",
"/bolletjestrui/.+\\.yaml$/",
"/gitea/.+\\.yaml$/"
"/gitea/.+\\.yaml$/",
"/gatus/.+\\.yaml$/",
"/ntfy/.+\\.yaml$/"
]
},
"customManagers": [
@@ -44,6 +46,13 @@
}
],
"packageRules": [
{
"description": "Longhorn 1.11.3 and newer need Kubernetes 1.34, see docs/upgrade-k8s-1.33.md",
"matchPackageNames": [
"longhorn"
],
"allowedVersions": "<1.11.3"
},
{
"matchDatasources": [
"helm"
-33
View File
@@ -1,33 +0,0 @@
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: hensenio-ghost-secrets
namespace: blog
spec:
refreshInterval: "15m"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: ghost-es
creationPolicy: Owner
data:
- secretKey: mysql-root-password
remoteRef:
key: kv/ghost
property: ghostmsqlrootpw
- secretKey: mysql-password
remoteRef:
key: kv/ghost
property: ghostmsqlpw
- secretKey: ghost-password
remoteRef:
key: kv/ghost
property: ghostpw
- secretKey: smtp-password
remoteRef:
key: kv/hensenio/smtp
property: password
+13
View File
@@ -44,6 +44,13 @@ spec:
value: https://serpbear.rubenhensen.nl
- name: TZ
value: Europe/Amsterdam
# Node trusts its built-in roots, which still include the ones
# above the compromised eMudhra intermediates. Use
# patched-ca-bundle through OpenSSL instead until Node ships a fix.
- name: NODE_OPTIONS
value: --use-openssl-ca
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca-certificates.crt
image: towfiqi/serpbear:3.1.0
name: serpbear
ports:
@@ -52,8 +59,14 @@ spec:
volumeMounts:
- mountPath: /app/data
name: data
- mountPath: /etc/ssl/certs
name: patched-ca-bundle
readOnly: true
restartPolicy: Always
volumes:
- name: data
persistentVolumeClaim:
claimName: data
- name: patched-ca-bundle
configMap:
name: patched-ca-bundle
+3 -1
View File
@@ -5,7 +5,9 @@ metadata:
spec:
provider:
vault:
server: "https://vault.rubenhensen.nl"
# In-cluster, so syncing secrets doesn't depend on Hetzner or the home
# internet connection. Plain HTTP, like the ingress → Vault hop.
server: "http://vault-active.vault.svc:8200"
path: "kv"
# Version is the Vault KV secret engine version.
# This can be either "v1" or "v2", defaults to "v2"
+14 -2
View File
@@ -11,6 +11,16 @@ spec:
targetRevision: 8.4.0
helm:
values: |
# Velero builds its repository maintenance jobs from a Deployment
# named "velero". Under the default name (velero-helm) every
# maintenance run failed and the kopia repos on the NAS were never
# cleaned up.
fullnameOverride: velero
# The CRD upgrade hook pulls docker.io/bitnami/kubectl, which Bitnami
# has removed, so every sync hangs on it. The CRDs only change with
# the chart version; turn this back on (with another kubectl image)
# when bumping targetRevision.
upgradeCRDs: false
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:v1.11.1
@@ -43,13 +53,15 @@ spec:
nodeAgent:
containerSecurityContext:
privileged: true
# 512Mi got the server OOMKilled 314 times, including halfway
# through a backup of ~14.7k items.
resources:
requests:
cpu: 100m
memory: 256Mi
memory: 512Mi
limits:
cpu: 500m
memory: 512Mi
memory: 2Gi
destination:
server: https://kubernetes.default.svc
-12
View File
@@ -1,12 +0,0 @@
apiVersion: v1
data:
DB_PASS: whishper
DB_USER: whishper
LT_LOAD_ONLY: es,en,fr,nl
WHISHPER_HOST: http://127.0.0.1:8082
WHISPER_MODELS: tiny,small,large-v3
kind: ConfigMap
metadata:
labels:
io.kompose.service: mongo-env
name: env
@@ -1,12 +0,0 @@
# apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
# labels:
# io.kompose.service: mongo-claim1
# name: mongo-claim1
# spec:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 10Gi
-76
View File
@@ -1,76 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: mongo
name: mongo
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: mongo
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: mongo
spec:
containers:
# - args:
# - --logpath
# - /var/log/mongodb/mongod.log
- env:
- name: DB_PASS
valueFrom:
configMapKeyRef:
key: DB_PASS
name: env
- name: DB_USER
valueFrom:
configMapKeyRef:
key: DB_USER
name: env
- name: LT_LOAD_ONLY
valueFrom:
configMapKeyRef:
key: LT_LOAD_ONLY
name: env
- name: MONGO_INITDB_ROOT_PASSWORD
value: whishper
- name: MONGO_INITDB_ROOT_USERNAME
value: whishper
- name: WHISHPER_HOST
valueFrom:
configMapKeyRef:
key: WHISHPER_HOST
name: env
- name: WHISPER_MODELS
valueFrom:
configMapKeyRef:
key: WHISPER_MODELS
name: env
image: mongo
name: mongo
ports:
- containerPort: 27017
protocol: TCP
volumeMounts:
- mountPath: /data/db
name: mongo-claim0
# - mountPath: /var/log/mongodb
# name: mongo-claim1
restartPolicy: Always
volumes:
- name: mongo-claim0
persistentVolumeClaim:
claimName: mongo-claim10
# - name: mongo-claim1
# persistentVolumeClaim:
# claimName: mongo-claim1
-16
View File
@@ -1,16 +0,0 @@
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: mongo
name: mongo
spec:
ports:
- name: "27017"
port: 27017
targetPort: 27017
selector:
io.kompose.service: mongo
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
io.kompose.service: translate-claim0
name: translate-claim0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
io.kompose.service: translate-claim1
name: translate-claim1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
-85
View File
@@ -1,85 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: translate
name: translate
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: translate
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: translate
spec:
containers:
- image: libretranslate/libretranslate:latest
securityContext:
runAsUser: 0
runAsGroup: 0
runAsNonRoot: false
livenessProbe:
exec:
command:
- ./venv/bin/python scripts/healthcheck.py
failureThreshold: 5
periodSeconds: 2
timeoutSeconds: 3
name: whisper-libretranslate
ports:
- containerPort: 5000
protocol: TCP
tty: true
volumeMounts:
- mountPath: /home/libretranslate/.local/share
name: translate-claim0
- mountPath: /home/libretranslate/.local/cache
name: translate-claim1
env:
- name: DB_PASS
valueFrom:
configMapKeyRef:
key: DB_PASS
name: env
- name: DB_USER
valueFrom:
configMapKeyRef:
key: DB_USER
name: env
- name: LT_DISABLE_WEB_UI
value: "true"
- name: LT_LOAD_ONLY
valueFrom:
configMapKeyRef:
key: LT_LOAD_ONLY
name: env
- name: LT_UPDATE_MODELS
value: "true"
- name: WHISHPER_HOST
valueFrom:
configMapKeyRef:
key: WHISHPER_HOST
name: env
- name: WHISPER_MODELS
valueFrom:
configMapKeyRef:
key: WHISPER_MODELS
name: env
restartPolicy: Always
volumes:
- name: translate-claim0
persistentVolumeClaim:
claimName: translate-claim0
- name: translate-claim1
persistentVolumeClaim:
claimName: translate-claim1
-16
View File
@@ -1,16 +0,0 @@
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: translate
name: translate
spec:
ports:
- name: "5000"
port: 5000
targetPort: 5000
selector:
io.kompose.service: translate
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
io.kompose.service: whishper-claim1
name: whishper-claim1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
-81
View File
@@ -1,81 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: whishper
name: whishper
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: whishper
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: whishper
spec:
containers:
- env:
- name: CPU_THREADS
value: "4"
- name: DB_PASS
valueFrom:
configMapKeyRef:
key: DB_PASS
name: env
- name: DB_USER
valueFrom:
configMapKeyRef:
key: DB_USER
name: env
- name: LT_LOAD_ONLY
valueFrom:
configMapKeyRef:
key: LT_LOAD_ONLY
name: env
- name: PUBLIC_API_HOST
- name: PUBLIC_INTERNAL_API_HOST
value: http://127.0.0.1:80
- name: PUBLIC_TRANSLATION_API_HOST
- name: PUBLIC_WHISHPER_PROFILE
value: cpu
- name: UPLOAD_DIR
value: /app/uploads
- name: WHISHPER_HOST
valueFrom:
configMapKeyRef:
key: WHISHPER_HOST
name: env
- name: WHISPER_MODELS
valueFrom:
configMapKeyRef:
key: WHISPER_MODELS
name: env
- name: WHISPER_MODELS_DIR
value: /app/models
image: pluja/whishper:latest
name: whishper
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /app/uploads
name: whishper-claim0
- mountPath: /var/log/whishper
name: whishper-claim1
restartPolicy: Always
volumes:
- name: whishper-claim0
persistentVolumeClaim:
claimName: whishper-claim0
- name: whishper-claim1
persistentVolumeClaim:
claimName: whishper-claim1
-43
View File
@@ -1,43 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: whisper-ingress
annotations:
# nginx.ingress.kubernetes.io/rewrite-target: /
cert-manager.io/cluster-issuer: prod-cluster-issuer
nginx.ingress.kubernetes.io/ssl-passthrough: "false"
nginx.ingress.kubernetes.io/enable-cors: "true"
# type of authentication
nginx.ingress.kubernetes.io/auth-type: basic
# name of the secret that contains the user/password definitions
nginx.ingress.kubernetes.io/auth-secret: whisper-basic-auth
# message to display with an appropriate context why the authentication is required
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
nginx.ingress.kubernetes.io/proxy-body-size: "0"
# nginx.ingress.kubernetes.io/cors-allow-origin: "http://localhost:3000/"
# nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS, DELETE"
# If you encounter a redirect loop or are getting a 307 response code
# then you need to force the nginx ingress to connect to the backend using HTTPS.
#
# nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
# namespace: ente-photos
spec:
ingressClassName: nginx
rules:
- host: alv.rubenhensen.nl
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: whishper
port:
number: 8082
# TLS configuration for HTTPS
tls:
# TLS secretName used on ClusterIssuer
- secretName: letsencrypt-prod
hosts:
- alv.rubenhensen.nl
-16
View File
@@ -1,16 +0,0 @@
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.34.0 (cbf2835db)
labels:
io.kompose.service: whishper
name: whishper
spec:
ports:
- name: "8082"
port: 8082
targetPort: 80
selector:
io.kompose.service: whishper