Compare commits

...
19 Commits
Author SHA1 Message Date
renovate[bot] a36fc6f348 chore(deps): update python docker tag to v3.15 2026-10-26 22:28:58 +00:00
Ruben Hensen 66207de2dd Merge pull request #52 from rubenhensen/renovate-automerge
Renovate: automerge patch updates of low-risk apps
2026-10-26 23:28:09 +01:00
Ruben Hensen d0ce5ddbec Merge pull request #51 from rubenhensen/netpol-acme
NetworkPolicies: let Envoy reach cert-manager's HTTP-01 solver
2026-10-26 23:27:48 +01:00
Ruben Hensen 6c727346f6 renovate: patch updates of low-risk apps merge themselves 2026-10-26 23:23:25 +01:00
Ruben Hensen ed828b426f networkpolicy: let Envoy reach cert-manager's HTTP-01 solver 2026-10-26 23:19:28 +01:00
Ruben Hensen 0dc69d34f0 Merge pull request #50 from rubenhensen/proxy-protocol
Pass the visitor's address from node001 to the home cluster
2026-10-26 23:12:14 +01:00
Ruben Hensen 30540ea579 node001: send the visitor's address to the home cluster
The stream server on 443 now starts each connection with a PROXY header.
nginx sends it to every upstream or none, so Stalwart (8443) and the
local nginx for Gatus and ntfy (8444) get theirs through a hop on
8445/8446 that takes it off again; nothing changes for them.

Deploy after the Gateway accepts the header, or the tunneled hosts stop
answering until it does.
2026-10-26 23:11:38 +01:00
Ruben Hensen 53d74768ad gateway: take the visitor's address from a PROXY header
node001 passes TLS through without opening it, so every request from
the tunneled hosts arrives from 46.224.26.65. Wordfence locked that
address out after 20 failed logins on 2026-10-26, and with it everyone
who logs in to scoutingmbg.nl. With a PROXY header from node001, Envoy
puts the visitor's address in X-Forwarded-For instead.

The header is optional: s2 and ntfy2 point straight at home, and
node001's proxy on port 80 sends none either.
2026-10-26 23:11:38 +01:00
Ruben Hensen b6462fb3a9 Merge pull request #49 from rubenhensen/roadmap-garage
docs: Velero on Garage
2026-10-26 23:04:15 +01:00
Ruben Hensen bed68c70a8 velero: the switch to Garage was on 10-26 2026-10-26 22:57:35 +01:00
Ruben Hensen 3991a8d726 docs: Velero on Garage, and the off-site copy from the bucket 2026-10-26 22:57:35 +01:00
Ruben Hensen 648f99c505 Merge pull request #35 from rubenhensen/velero-garage
Velero backs up to Garage instead of MinIO
2026-10-26 22:52:42 +01:00
Ruben Hensen 20bd3a4bca Merge pull request #39 from rubenhensen/immich-ml-v2
immich: the optimized ML models, cached on a volume
2026-10-25 17:48:40 +01:00
Ruben Hensen 6cba1a9363 immich: the optimized ML models, cached on a volume 2026-10-25 17:46:03 +01:00
Ruben Hensen acfce6645e velero: skip PVCs in Longhorn's no-backup group 2026-10-25 17:46:03 +01:00
Ruben Hensen 0de86dd65c Merge pull request #36 from rubenhensen/renovate-re2
Renovate: skip the non-app folders without a lookahead
2026-10-25 17:41:32 +01:00
Ruben Hensen d7ffdc05f6 docs: Renovate rejected the lookahead (#32) 2026-10-25 17:33:39 +01:00
Ruben Hensen c0f8034e14 renovate: skip the non-app folders through ignorePaths, not a lookahead 2026-10-25 17:33:39 +01:00
Ruben Hensen a15a4f7c75 Merge pull request #34 from rubenhensen/velero-garage-secret
Garage on the NAS, and Velero's key for it
2026-10-25 17:23:33 +01:00
19 changed files with 459 additions and 44 deletions

No files matched your search

+26
View File
@@ -55,3 +55,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+26
View File
@@ -135,3 +135,29 @@ spec:
cidr: 192.168.1.204/32 cidr: 192.168.1.204/32
ports: ports:
- port: 16443 - port: 16443
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+26
View File
@@ -54,3 +54,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+1 -1
View File
@@ -13,7 +13,7 @@ spec:
restartPolicy: Never restartPolicy: Never
containers: containers:
- name: sync - name: sync
image: python:3.12-alpine image: python:3.15-alpine
command: command:
- sh - sh
- -c - -c
+67 -34
View File
@@ -10,11 +10,11 @@ One node: opti1, an OptiPlex 5060 with an i5-8600T, 32 GB and one 1 TB
SATA SSD that is 72% full. microk8s v1.36.5 from 1.36/edge, snap held. SATA SSD that is 72% full. microk8s v1.36.5 from 1.36/edge, snap held.
35 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn 35 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn
volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the
NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to MinIO at NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to Garage
01:00 (daily kept 30 days, weekly 90 days). Gatus on s1 (node001, Hetzner) (MinIO until 2026-10-26) at 01:00 (daily kept 30 days, weekly 90 days).
and s2 (home) watch each other and alert through ntfy; s2 also gets an Gatus on s1 (node001, Hetzner) and s2 (home) watch each other and alert
hourly report from inside the cluster on the backups, Vault's seal, the through ntfy; s2 also gets an hourly report from inside the cluster on the
Argo CD apps and the CNPG clusters. backups, Vault's seal, the Argo CD apps and the CNPG clusters.
Every app namespace has NetworkPolicies, and CI renders and validates all Every app namespace has NetworkPolicies, and CI renders and validates all
apps on every PR. node001 runs mail and tunnels the public hosts home. apps on every PR. node001 runs mail and tunnels the public hosts home.
@@ -63,24 +63,25 @@ belong to the group.
There is no off-site copy (2026-10-25). The plan: swap S3 buckets with a There is no off-site copy (2026-10-25). The plan: swap S3 buckets with a
friend. friend.
- [ ] Ours to his bucket: Hyper Backup on the Synology, destination "S3 - [ ] Ours to his bucket: `rclone sync` from Garage's `velero-backups`
Storage" with his endpoint as a custom server. Client-side encryption (~190 GB) to his bucket through an rclone `crypt` remote, in Docker on
on: the kopia repositories use Velero's default password the NAS like the copy from MinIO (see 5). Encrypted on our side: the
(`static-passw0rd`), so without it he could read everything, the kopia repositories use Velero's default password (`static-passw0rd`),
group's photos included. Keep the encryption key next to the Vault so without it he could read everything, the group's photos included.
unseal key. Source: the shared folder with MinIO's `velero-backups` Keep the crypt passwords next to the Vault unseal key. Daily after the
bucket (~150-200 GB). Daily after the Velero backup, Smart Recycle, Velero backup, ntfy on failure. Hyper Backup, the earlier plan, backs up
mail on failure shared folders; Garage keeps its objects as blocks plus a metadata
database, and a copy of that folder taken while Garage runs is not
consistent
- [ ] Needed from him: endpoint, which S3 server he runs, space (~200 GB - [ ] Needed from him: endpoint, which S3 server he runs, space (~200 GB
now, 500 GB to grow), and versioning or object lock on the bucket, so now, 500 GB to grow), and versioning or object lock on the bucket, so
whoever takes over the NAS can't delete the off-site copy as well whoever takes over the NAS can't delete the off-site copy as well
- [ ] His backups with us: Garage (not MinIO, see 5) in Container Manager - [ ] His backups with us: a bucket and key for him, with a quota, in the
on the Synology, in its own shared folder, with a bucket and key for him Garage on the NAS (`nas/garage`, see 5). Only the S3 API public, as
and a quota. Only the S3 API public, as `s3.rubenhensen.nl` through `s3.rubenhensen.nl` through node001 and Envoy with a cert-manager
node001 and Envoy with a cert-manager certificate; the admin API stays certificate; the admin API stays on the LAN. The cluster side (DNS,
on the LAN. The cluster side (DNS, node001 tunnel, Certificate, node001 tunnel, Certificate, HTTPRoute) goes in git. Needed: how much
HTTPRoute) goes in git. Needed: how much space he wants, free space on space he wants, free space on the NAS
the NAS
- [ ] Decide which copy of the immich library goes off-site (see 5) - [ ] Decide which copy of the immich library goes off-site (see 5)
- [ ] A restore test that runs on a schedule. `backup-testing/` predates - [ ] A restore test that runs on a schedule. `backup-testing/` predates
Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth. Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth.
@@ -108,9 +109,12 @@ friend.
MinIO runs region `minio`; the script now reads endpoint, region and MinIO runs region `minio`; the script now reads endpoint, region and
bucket from Velero's location. Copying freshrss's kopia repository took bucket from Velero's location. Copying freshrss's kopia repository took
14 of the ~17 minutes 14 of the ~17 minutes
- [ ] Run it on a schedule. It needs Docker and the MinIO key, so not in - [x] Ran it again on 2026-10-26 against the copy in Garage
the cluster itself: the Mac (launchd) or node001, ideally with a (daily-backup-20261026010028, same namespaces), before Velero switched
read-only MinIO user over (see 5): passed, in 5 minutes instead of 17
- [ ] Run it on a schedule. It needs Docker and an S3 key, so not in the
cluster itself: the Mac (launchd) or node001, ideally with a read-only
Garage key (`garage key create`, `garage bucket allow --read`)
### 3. authentik 2026.2.1 → 2026.8.5 ### 3. authentik 2026.2.1 → 2026.8.5
@@ -193,7 +197,7 @@ Findings from 2026-10-25:
- Velero copies the CNPG and MariaDB data directories file by file while - Velero copies the CNPG and MariaDB data directories file by file while
the databases run, which is not a consistent copy. The Longhorn backups the databases run, which is not a consistent copy. The Longhorn backups
start from a snapshot and are crash-consistent. CNPG can back up to start from a snapshot and are crash-consistent. CNPG can back up to
MinIO itself (barman cloud plugin), with point-in-time recovery. Garage itself (barman cloud plugin), with point-in-time recovery.
- The PartiallyFailed daily backup of 10-23 (emptyDirs of Velero's own - The PartiallyFailed daily backup of 10-23 (emptyDirs of Velero's own
maintenance job pods) was fixed that afternoon by maintenance job pods) was fixed that afternoon by
`velero/skip-emptydir-policy.yaml`. `velero/skip-emptydir-policy.yaml`.
@@ -201,10 +205,31 @@ Findings from 2026-10-25:
- [ ] Whether the immich library stays in Velero. It stays for now - [ ] Whether the immich library stays in Velero. It stays for now
(2026-10-25); decide again with the off-site copy (2) (2026-10-25); decide again with the off-site copy (2)
- [ ] CNPG backups through the barman cloud plugin - [ ] CNPG backups through the barman cloud plugin
- [ ] Velero's target, MinIO on the NAS, is no longer maintained: no - [x] Velero's target, MinIO on the NAS, is no longer maintained: no
community images since October 2025, maintenance mode since December community images since October 2025, maintenance mode since December
2025, and the repository is archived. Move Velero to the Garage from 2 2025, and the repository is archived. Since 2026-10-26 Velero backs up
and switch MinIO off to Garage on the same NAS (`nas/garage`, PR #35). The bucket was copied
with rclone (185 GiB in about 6 hours, then a catch-up sync just before
the switch), so the existing backups and kopia repositories carried on:
after the switch all 55 backups were still listed, and a manual backup
of ntfy found the existing repository and used the night's snapshot as
its parent. Two things made the NAS slow during the copy: Garage on
LMDB with `metadata_fsync`, ~270 disk writes/s while idle (now SQLite,
see `nas/garage/garage.toml`), and MinIO's scanner, which kept one disk
at 97% busy. The scanner is now at `slowest` (`mc admin config set local
scanner speed=slowest` in the MinIO container); it may have slowed
Velero's backups to MinIO as well
- [ ] Switch MinIO off once Velero has run on Garage for a week (from
2026-11-02): stop the container, then delete `/volume1/minio` (343 GiB)
and the hand-made Secret `velero/cloud-credentials` (MinIO's key). Only
MinIO still has the kopia repositories of namespaces that are gone
(~156 GB: nextcloud, ttrss, whisper-priv, whisper-webui, blog, ghost,
mbg-wordpress, sogo, sogo-mail, outrun-game, owntracks); decide first
whether any of them is worth keeping
- [ ] Garage's SQLite WAL (`meta/db.sqlite-wal`) was 1.7 GB on
2026-10-26 next to a 270 MB database, and going by the folders in
`meta/snapshots`, each 6-hourly metadata snapshot takes about 70
minutes. Not growing; check again after a week of normal use
### 6. Monitoring ### 6. Monitoring
@@ -323,8 +348,10 @@ Counted 2026-10-25 over the 73 running containers:
### 8. Renovate and CI ### 8. Renovate and CI
10 Renovate PRs are open, the oldest from February, and nothing checks a On 2026-10-25 10 Renovate PRs were open, the oldest from February, and
PR before Argo CD applies it. nothing checked a PR before Argo CD applied it. All of them have since been
merged or closed (below), none are open (checked 2026-10-25), and CI checks
every PR.
- [x] `.github/workflows/validate.yaml` runs `tools/ci/validate.py` on - [x] `.github/workflows/validate.yaml` runs `tools/ci/validate.py` on
every PR and push (2026-10-25): it renders all 36 apps the way Argo CD every PR and push (2026-10-25): it renders all 36 apps the way Argo CD
@@ -360,10 +387,16 @@ PR before Argo CD applies it.
cover every app folder, a regex manager reads `imageName` in cover every app folder, a regex manager reads `imageName` in
`*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off `*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off
(those need pg_upgrade or a dump). Expect a batch of new PRs (those need pg_upgrade or a dump). Expect a batch of new PRs
(2026-10-25; `renovate-config-validator` passes). They only come once (2026-10-25). They only come once fewer than 10 Renovate PRs are open
fewer than 10 Renovate PRs are open (`config:recommended`'s (`config:recommended`'s `prConcurrentLimit`); until then they wait
`prConcurrentLimit`); until then they wait under "Rate-Limited" on the under "Rate-Limited" on the Dependency Dashboard (#2)
Dependency Dashboard (#2) - The other folders were left out with a `(?!…)` lookahead. Renovate
compiles these patterns with RE2, which has no lookahead, so it
rejected the config and stopped opening PRs (#32). Each manager now
takes every folder and skips the others through its own
`ignorePaths`. Renovate 44.181.6 with RE2 loaded fails the old config
with #32's error and passes the new one, and its matcher picks the
same 166 files as the old pattern did
- [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager - [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager
v1.21.3 (security release; all three pods on v1.21.3), #24 v1.21.3 (security release; all three pods on v1.21.3), #24
external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as
@@ -440,7 +473,7 @@ critical / high:
| tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag | | tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag |
| longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below | | longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below |
| actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it | | actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it |
| gitea/gitea:1.27.3 | 8 | 43 | PR #26 (28.2.0) | | gitea/gitea:1.27.3 | 8 | 43 | now 28.2.0 (PR #26, 2026-10-25); not scanned again |
| immich-server / machine-learning v3.3.5 | 7, 7 | 113, 78 | a newer immich release | | immich-server / machine-learning v3.3.5 | 7, 7 | 113, 78 | a newer immich release |
| towfiqi/serpbear:3.1.0 | 6 | 77 | a newer release | | towfiqi/serpbear:3.1.0 | 6 | 77 | a newer release |
| rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image | | rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image |
+26
View File
@@ -56,3 +56,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+23
View File
@@ -0,0 +1,23 @@
# node001 passes TLS for the tunneled hosts through without opening it, so
# without this every request arrives from 46.224.26.65. Wordfence locks that
# address out after 20 failed logins (2026-10-26), and with it everyone who
# logs in to scoutingmbg.nl. node001 puts a PROXY header with the visitor's
# address in front of each connection; Envoy takes the address from there
# and adds it to X-Forwarded-For.
#
# Optional, because s2 and ntfy2 point straight at home and their clients
# send no header, and neither does node001's proxy on port 80. The price:
# whoever connects to the home address directly can send a header of their
# own and claim any address.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: proxy-protocol
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: homelab
proxyProtocol:
optional: true
+26
View File
@@ -83,3 +83,29 @@ spec:
cidr: 192.168.1.204/32 cidr: 192.168.1.204/32
ports: ports:
- port: 16443 - port: 16443
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+26
View File
@@ -55,3 +55,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+14 -3
View File
@@ -127,6 +127,9 @@ spec:
TRANSFORMERS_CACHE: /cache TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet HF_XET_CACHE: /cache/huggingface-xet
MPLCONFIGDIR: /cache/matplotlib-config MPLCONFIGDIR: /cache/matplotlib-config
# The optimized models of v3.3.0: faster, less memory,
# and the same results, so no jobs to run again.
MACHINE_LEARNING_MODEL_REVISION: v2
# Trust store without the compromised eMudhra # Trust store without the compromised eMudhra
# intermediates; requests would otherwise use certifi. # intermediates; requests would otherwise use certifi.
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
@@ -139,11 +142,19 @@ spec:
globalMounts: globalMounts:
- path: /etc/ssl/certs - path: /etc/ssl/certs
readOnly: true readOnly: true
# The models and their one-off preparation, kept across restarts.
# Downloaded again if lost, so no backups: the no-backup group
# keeps it out of Longhorn's default one (backup-daily), and
# Velero skips that label too (velero/skip-emptydir-policy.yaml).
cache: cache:
enabled: true enabled: true
# Optional: to avoid downloading the ML models every start, use type: persistentVolumeClaim
# a persistentVolumeClaim with size, accessMode and storageClass storageClass: longhorn
type: emptyDir accessMode: ReadWriteOnce
size: 10Gi
labels:
recurring-job.longhorn.io/source: enabled
recurring-job-group.longhorn.io/no-backup: enabled
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
+26
View File
@@ -100,3 +100,29 @@ spec:
cidr: 192.168.1.204/32 cidr: 192.168.1.204/32
ports: ports:
- port: 16443 - port: 16443
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+26
View File
@@ -85,3 +85,29 @@ spec:
kubernetes.io/metadata.name: mariadb-system kubernetes.io/metadata.name: mariadb-system
ports: ports:
- port: 3306 - port: 3306
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+20 -3
View File
@@ -173,12 +173,18 @@ in
# - mail.rubenhensen.nl (and anything else) → local stalwart on 8443 # - mail.rubenhensen.nl (and anything else) → local stalwart on 8443
# - tunneled hosts → home cluster ingress on 443 # - tunneled hosts → home cluster ingress on 443
# - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy) # - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy)
#
# Each connection starts with a PROXY header that carries the visitor's
# address, so the home cluster sees who is asking instead of this box
# (gateway/proxy-protocol.yaml in k8scd). nginx sends the header to
# every upstream or none, so the local ones go through a hop on
# 8445/8446 that takes it off again.
streamConfig = '' streamConfig = ''
map $ssl_preread_server_name $tunnel_upstream { map $ssl_preread_server_name $tunnel_upstream {
${sniMapEntries} ${sniMapEntries}
s1.rubenhensen.nl 127.0.0.1:8444; s1.rubenhensen.nl 127.0.0.1:8446;
ntfy1.rubenhensen.nl 127.0.0.1:8444; ntfy1.rubenhensen.nl 127.0.0.1:8446;
default 127.0.0.1:8443; default 127.0.0.1:8445;
} }
server { server {
@@ -186,6 +192,17 @@ ${sniMapEntries}
listen [::]:443; listen [::]:443;
proxy_pass $tunnel_upstream; proxy_pass $tunnel_upstream;
ssl_preread on; ssl_preread on;
proxy_protocol on;
}
server {
listen 127.0.0.1:8445 proxy_protocol;
proxy_pass 127.0.0.1:8443;
}
server {
listen 127.0.0.1:8446 proxy_protocol;
proxy_pass 127.0.0.1:8444;
} }
''; '';
+26
View File
@@ -65,3 +65,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+40 -2
View File
@@ -4,9 +4,18 @@
"config:recommended", "config:recommended",
"customManagers:githubActionsVersions" "customManagers:githubActionsVersions"
], ],
"timezone": "Europe/Amsterdam",
"argocd": { "argocd": {
"description": "Every app folder; Renovate's regexes are RE2, which has no lookahead, so the other folders are skipped through ignorePaths",
"managerFilePatterns": [ "managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/" "/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
] ]
}, },
"helm-values": { "helm-values": {
@@ -15,8 +24,16 @@
] ]
}, },
"kubernetes": { "kubernetes": {
"description": "Every app folder, as for argocd",
"managerFilePatterns": [ "managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/" "/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
] ]
}, },
"customManagers": [ "customManagers": [
@@ -72,6 +89,27 @@
"major" "major"
], ],
"enabled": false "enabled": false
},
{
"description": "Patch updates of apps that hold little state and are easy to roll back, and of CI's tools, merge themselves once CI passes: three days after the release, in the evening or at the weekend, when the cluster can be reached. Not the databases, operators, Vault, authentik, Longhorn, Velero, immich, Gitea or WordPress. Renovate merges itself; the repo has no branch protection for GitHub's auto-merge to wait on",
"matchFileNames": [
"ntfy/**",
"gatus/**",
"freshrss/**",
"serpbear/**",
"actual-budget/**",
".github/workflows/**"
],
"matchUpdateTypes": [
"patch"
],
"minimumReleaseAge": "3 days",
"automerge": true,
"platformAutomerge": false,
"automergeSchedule": [
"* 17-21 * * 1-5",
"* 10-21 * * 0,6"
]
} }
] ]
} }
+26
View File
@@ -54,3 +54,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+26
View File
@@ -66,3 +66,29 @@ spec:
- 169.254.0.0/16 - 169.254.0.0/16
- 172.16.0.0/12 - 172.16.0.0/12
- 192.168.0.0/16 - 192.168.0.0/16
---
# cert-manager's HTTP-01 solver, which runs only while a certificate
# renews: Let's Encrypt and cert-manager's own self check reach it on 8089
# through Envoy. The policies above let Envoy reach the app's port only,
# which left the solver unreachable (found 2026-10-26, when ynab's renewal
# hung).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: acme-http01-solver
spec:
podSelector:
matchLabels:
acme.cert-manager.io/http01-solver: "true"
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
ports:
- port: 8089
+7
View File
@@ -1,6 +1,8 @@
# Volume policy for the backup schedules. emptyDir volumes hold nothing # Volume policy for the backup schedules. emptyDir volumes hold nothing
# worth keeping, and backing them up fails when the pod is gone before its # worth keeping, and backing them up fails when the pod is gone before its
# volume is read, as happens with Velero's own repository maintenance jobs. # volume is read, as happens with Velero's own repository maintenance jobs.
# PVCs in Longhorn's no-backup group (caches, such as immich's ML models)
# are skipped too.
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@@ -14,3 +16,8 @@ data:
- emptyDir - emptyDir
action: action:
type: skip type: skip
- conditions:
pvcLabels:
recurring-job-group.longhorn.io/no-backup: enabled
action:
type: skip
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
- mountPath: /target - mountPath: /target
name: plugins name: plugins
configuration: configuration:
# Garage on the NAS (nas/garage). Until 2026-10-25 this was MinIO on # Garage on the NAS (nas/garage). Until 2026-10-26 this was MinIO on
# the same NAS; the bucket was copied over with rclone, so the # the same NAS; the bucket was copied over with rclone, so the
# existing backups and kopia repositories carried on. # existing backups and kopia repositories carried on.
backupStorageLocation: backupStorageLocation: