Compare commits

..
Author SHA1 Message Date
Ruben Hensen 30540ea579 node001: send the visitor's address to the home cluster
The stream server on 443 now starts each connection with a PROXY header.
nginx sends it to every upstream or none, so Stalwart (8443) and the
local nginx for Gatus and ntfy (8444) get theirs through a hop on
8445/8446 that takes it off again; nothing changes for them.

Deploy after the Gateway accepts the header, or the tunneled hosts stop
answering until it does.
2026-10-26 23:11:38 +01:00
Ruben Hensen 53d74768ad gateway: take the visitor's address from a PROXY header
node001 passes TLS through without opening it, so every request from
the tunneled hosts arrives from 46.224.26.65. Wordfence locked that
address out after 20 failed logins on 2026-10-26, and with it everyone
who logs in to scoutingmbg.nl. With a PROXY header from node001, Envoy
puts the visitor's address in X-Forwarded-For instead.

The header is optional: s2 and ntfy2 point straight at home, and
node001's proxy on port 80 sends none either.
2026-10-26 23:11:38 +01:00
Ruben Hensen 20bd3a4bca Merge pull request #39 from rubenhensen/immich-ml-v2
immich: the optimized ML models, cached on a volume
2026-10-25 17:48:40 +01:00
Ruben Hensen 6cba1a9363 immich: the optimized ML models, cached on a volume 2026-10-25 17:46:03 +01:00
Ruben Hensen acfce6645e velero: skip PVCs in Longhorn's no-backup group 2026-10-25 17:46:03 +01:00
Ruben Hensen 0de86dd65c Merge pull request #36 from rubenhensen/renovate-re2
Renovate: skip the non-app folders without a lookahead
2026-10-25 17:41:32 +01:00
Ruben Hensen d7ffdc05f6 docs: Renovate rejected the lookahead (#32) 2026-10-25 17:33:39 +01:00
Ruben Hensen c0f8034e14 renovate: skip the non-app folders through ignorePaths, not a lookahead 2026-10-25 17:33:39 +01:00
Ruben Hensen a15a4f7c75 Merge pull request #34 from rubenhensen/velero-garage-secret
Garage on the NAS, and Velero's key for it
2026-10-25 17:23:33 +01:00
Ruben Hensen b072099cc1 nas: Garage keeps its metadata in SQLite 2026-10-25 17:08:13 +01:00
Ruben Hensen e10ca337f1 velero: a key for Garage, from Vault 2026-10-25 16:28:43 +01:00
Ruben Hensen c72b08867f nas: Garage, which takes over from MinIO 2026-10-25 16:28:43 +01:00
Ruben Hensen d85362ce3b docs: the restore test passed against the real backups 2026-10-25 14:57:18 +01:00
Ruben Hensen 018f528fb3 docs: a WordPress cache without plugins 2026-10-25 14:56:10 +01:00
Ruben Hensen 714712b862 docs: WordPress is capped at 12 workers 2026-10-25 13:11:20 +01:00
Ruben Hensen 7dc13466cb mbgwp: cap Apache at 12 workers so a crawler can't run it out of memory 2026-10-25 13:07:30 +01:00
Ruben Hensen 9b77596bcb backup-testing: take endpoint, region and bucket from Velero's location 2026-10-25 12:59:25 +01:00
Ruben Hensen fbf2cb2d92 docs: the unseal key is kept safe 2026-10-25 12:57:18 +01:00
Ruben Hensen 122873ca5c docs: hensen.io can expire 2026-10-25 12:54:32 +01:00
Ruben Hensen 71449dd185 docs: record the external-secrets upgrade 2026-10-25 12:54:01 +01:00
Ruben Hensen 297d2dffa5 Merge pull request #17 from rubenhensen/renovate/external-secrets-2.x
chore(deps): update helm chart external-secrets to v2.14.0
2026-10-25 12:50:28 +01:00
12 changed files with 276 additions and 33 deletions

No files matched your search

+17
View File
@@ -101,6 +101,23 @@ spec:
timeoutSeconds: 5
failureThreshold: 30
# apache2.conf includes ports.conf after mods-enabled/*.conf, so the
# prefork settings here replace mpm_prefork.conf's MaxRequestWorkers
# 150. A worker takes ~45-60 MB (PSS), so 150 never fit in 1Gi: a
# crawler asking for 88 pages at once got the container OOMKilled on
# 2026-10-25. With 12 workers a burst waits in the listen backlog
# instead. Listen is the chart's default (containerPort).
apachePortsConfig: |
Listen 8000
<IfModule mpm_prefork_module>
StartServers 4
MinSpareServers 2
MaxSpareServers 6
ServerLimit 12
MaxRequestWorkers 12
MaxConnectionsPerChild 1000
</IfModule>
# Custom PHP configuration
customPhpConfig: |
upload_max_filesize = 64M
+13 -3
View File
@@ -33,8 +33,6 @@
set -euo pipefail
PROD_KUBECONFIG=${PROD_KUBECONFIG:-$HOME/.kube/homelab-config}
MINIO=${MINIO:-http://192.168.1.99:9000}
BUCKET=velero-backups
PORT=${PORT:-16443}
K3S=rancher/k3s:v1.34.9-k3s1
RCLONE=rclone/rclone:1.75.1
@@ -73,6 +71,18 @@ wait_for() {
done
}
# Endpoint, region and bucket as Velero itself uses them. MinIO answers a
# request signed for the wrong region (rclone defaults to us-east-1) with a
# bare 400 on HEAD.
if [[ -n ${FROM_VOLUME:-} ]]; then
BUCKET=${BUCKET:-velero-backups}
else
bsl=$(pk -n velero get backupstoragelocations default -o json)
MINIO=${MINIO:-$(jq -r '.spec.config.s3Url' <<<"$bsl")}
REGION=${REGION:-$(jq -r '.spec.config.region // "us-east-1"' <<<"$bsl")}
BUCKET=${BUCKET:-$(jq -r '.spec.objectStorage.bucket' <<<"$bsl")}
fi
backup=${1:-}
shift || true
namespaces=("$@")
@@ -94,7 +104,7 @@ echo "== copy from the NAS (read-only)"
(umask 077; pk -n velero get secret cloud-credentials -o jsonpath='{.data.cloud}' | base64 -d |
awk -F' *= *' '/aws_access_key_id/ {print "RCLONE_CONFIG_NAS_ACCESS_KEY_ID="$2}
/aws_secret_access_key/ {print "RCLONE_CONFIG_NAS_SECRET_ACCESS_KEY="$2}' > "$WORK/nas.env"
printf 'RCLONE_CONFIG_NAS_TYPE=s3\nRCLONE_CONFIG_NAS_PROVIDER=Minio\nRCLONE_CONFIG_NAS_ENDPOINT=%s\n' "$MINIO" >> "$WORK/nas.env")
printf 'RCLONE_CONFIG_NAS_TYPE=s3\nRCLONE_CONFIG_NAS_PROVIDER=Minio\nRCLONE_CONFIG_NAS_ENDPOINT=%s\nRCLONE_CONFIG_NAS_REGION=%s\n' "$MINIO" "$REGION" >> "$WORK/nas.env")
copy() { docker run --rm --env-file "$WORK/nas.env" -v "$VOL:/data" "$RCLONE" copy --stats-one-line "nas:$BUCKET/$1" "/data/$BUCKET/$1"; }
copy "backups/$backup"
for ns in "${namespaces[@]}"; do copy "kopia/$ns"; done
+58 -22
View File
@@ -50,8 +50,9 @@ group's ns2: ns1 is stuck on an old serial and still answers
mbg.hensen.io (docs/immich-scouts-online.md).
- [x] Expiry date: 2027-01-09 (whois, 2026-10-25; registrar Key-Systems)
- [ ] Have the group's DNS host fix ns1; keep hensen.io until it answers
rubenhensen.nl too
- [x] ns1 caught up on 2026-10-24 16:50 UTC; both ns1 and ns2 answer
`rubenhensen.nl` for foto (checked 2026-10-25), and nothing in the
cluster uses hensen.io anymore. hensen.io can expire
### 2. Off-site backups
@@ -99,8 +100,17 @@ friend.
ReadOnly location: data restored byte for byte (same sha256), deployment
available, everything removed afterwards. Two pitfalls found on the way
are in the script's header. Not yet run against the real backups:
- [ ] Run `backup-testing/restore-test.sh` (it reads the MinIO key from
the cluster), then schedule it
- [x] Ran `backup-testing/restore-test.sh` on 2026-10-25 against
daily-backup-20261025010027 (ntfy, freshrss, actual-budget): every pod
volume came back with its backed-up byte count (actual-budget 8.4 MB,
freshrss 293 MB, ntfy 180 KB) and all deployments became available. The
first run failed on a 400 from MinIO: rclone signed for us-east-1 while
MinIO runs region `minio`; the script now reads endpoint, region and
bucket from Velero's location. Copying freshrss's kopia repository took
14 of the ~17 minutes
- [ ] Run it on a schedule. It needs Docker and the MinIO key, so not in
the cluster itself: the Mac (launchd) or node001, ideally with a
read-only MinIO user
### 3. authentik 2026.2.1 → 2026.8.5
@@ -155,8 +165,8 @@ unseals it.
- The OIDC job called `vault auth tune -listing_visibility`, a flag that
doesn't exist, so the OIDC mount's listing visibility was never set.
Fixed; the job now uses the 2.2.1 CLI instead of 1.15
- [ ] There is one unseal key (1 share, threshold 1). Keep it in two
places outside this cluster; without it the data in Vault is lost
- [x] There is one unseal key (1 share, threshold 1), kept safe outside
this cluster (2026-10-25); without it the data in Vault is lost
- [x] Unsealing is in the SSD install steps
- [ ] After a week without problems: delete the three snapshots
@@ -279,16 +289,29 @@ Counted 2026-10-25 over the 73 running containers:
kube-system
- [ ] Memory limits, once there are daytime numbers: the night ones miss
immich's machine learning at work (it idles at 234Mi)
- [ ] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on
- [x] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on
2026-10-25, its third restart: a crawler asked for 88 pages and assets
within a few seconds (normally ~6 a minute), Apache started a PHP worker
for each, and the container went past its 1Gi limit (500s first, then
the kill; back within seconds). Unrelated to the NetworkPolicies; memory
sat at ~500Mi before. Fixes, in order: cap Apache's prefork workers
(`MaxRequestWorkers` ~16, so a burst queues instead of filling memory),
a page cache plugin. Rate limiting per visitor in Envoy doesn't work
yet: node001 passes TLS through, so every request arrives from
46.224.26.65; that needs the PROXY protocol from node001's nginx to Envoy
for each (~45-60 MB each, `MaxRequestWorkers` was the default 150), and
the container went past its 1Gi limit. Unrelated to the
NetworkPolicies. Now capped at 12 workers through the chart's
`apachePortsConfig` (ports.conf comes after mods-enabled/*.conf), checked
with `apache2ctl -t` in the same image first. Replaying the burst (90
page requests, 40 at a time): all 200, at most 13 Apache processes and
572Mi, no restart; the price is waiting (4.3s median, up to ~22s at the
peak) instead of a crash
- [ ] Optional, no plugins wanted on WordPress: a small nginx cache in the
cluster between Envoy and WordPress that keeps anonymous GETs for a few
minutes and passes anything with a WordPress cookie through. A page
renders in 0.4-0.7s when the site is quiet and OPcache is on, so the
waiting in a burst is queueing; the cap alone already prevents the crash
- [ ] Fewer plugins on the site (17 now; up to whoever runs it):
`litespeed-cache` only caches pages on a LiteSpeed server, so it does
little here, `wordpress-importer` is usually for a one-off import, and
`code-snippets` runs PHP from the dashboard
- [ ] Rate limiting per visitor in Envoy needs the client's address, which
node001 loses: it passes TLS through, so every request arrives from
46.224.26.65. That needs the PROXY protocol from node001's nginx to Envoy
(ClientTrafficPolicy `enableProxyProtocol`)
- [x] Probes for gitea (`/api/healthz`), freshrss (TCP, since
mod_auth_openidc answers 401 to non-browsers) and serpbear (`/login`)
@@ -337,10 +360,16 @@ PR before Argo CD applies it.
cover every app folder, a regex manager reads `imageName` in
`*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off
(those need pg_upgrade or a dump). Expect a batch of new PRs
(2026-10-25; `renovate-config-validator` passes). They only come once
fewer than 10 Renovate PRs are open (`config:recommended`'s
`prConcurrentLimit`); until then they wait under "Rate-Limited" on the
Dependency Dashboard (#2)
(2026-10-25). They only come once fewer than 10 Renovate PRs are open
(`config:recommended`'s `prConcurrentLimit`); until then they wait
under "Rate-Limited" on the Dependency Dashboard (#2)
- The other folders were left out with a `(?!…)` lookahead. Renovate
compiles these patterns with RE2, which has no lookahead, so it
rejected the config and stopped opening PRs (#32). Each manager now
takes every folder and skips the others through its own
`ignorePaths`. Renovate 44.181.6 with RE2 loaded fails the old config
with #32's error and passes the new one, and its matcher picks the
same 166 files as the old pattern did
- [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager
v1.21.3 (security release; all three pods on v1.21.3), #24
external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as
@@ -350,10 +379,11 @@ PR before Argo CD applies it.
day: #23 ntfy v2.34.0, #27 go-dqlite v3.0.4, #33 (see 7) and #26 Gitea
28.2.0 (migrations up to 357 ran clean, `/api/healthz` 200; no backup of
its own beforehand, the nightly ones are the way back)
- [ ] Still open: #17 external-secrets 2.13.0 → 2.14.0, no breaking
changes in the notes; every secret goes through it, so merge when
there's time to watch a force-sync. It only changes `targetRevision`, so
it still merges cleanly after #33 added values to the same file
- [x] #17 external-secrets 2.14.0, merged 2026-10-25 11:50 UTC after a
manual rebase (#33 had added values on the next line, which git counts
as a conflict). All three pods on v2.14.0, the store Ready, and 24 of 25
ExternalSecrets refreshed on a force-sync; the 25th is the generated
`gatus/backup-check-token`, which never refreshes by design
### 9. One node
@@ -369,6 +399,12 @@ step after that, if downtime starts to matter.
`backup-testing/test-velero-backup.sh.bak` (an older copy); the 15
completed kopia maintenance jobs of repositories that no longer exist
(nextcloud, owntracks, sogo, sogo-mail, ttrss)
- [ ] freshrss mounts `freshrss-claim2` and `freshrss-claim3` as whole
volumes over the file paths `data/config.custom.php` and
`data/config-user.custom.php`, so FreshRSS sees two empty directories
there and the volumes hold nothing (0 bytes in every backup). Drop both
mounts and the two PVCs, or make them ConfigMap files if custom config
is wanted
- [ ] `kubectl delete namespace secrets`, empty since its app went
- [ ] 41 Failed Velero backups: the dailies from 2026-09-26 to 10-21 and
the weeklies since 08-02. Velero doesn't restore from a Failed backup;
+23
View File
@@ -0,0 +1,23 @@
# node001 passes TLS for the tunneled hosts through without opening it, so
# without this every request arrives from 46.224.26.65. Wordfence locks that
# address out after 20 failed logins (2026-10-26), and with it everyone who
# logs in to scoutingmbg.nl. node001 puts a PROXY header with the visitor's
# address in front of each connection; Envoy takes the address from there
# and adds it to X-Forwarded-For.
#
# Optional, because s2 and ntfy2 point straight at home and their clients
# send no header, and neither does node001's proxy on port 80. The price:
# whoever connects to the home address directly can send a header of their
# own and claim any address.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: proxy-protocol
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: homelab
proxyProtocol:
optional: true
+14 -3
View File
@@ -127,6 +127,9 @@ spec:
TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet
MPLCONFIGDIR: /cache/matplotlib-config
# The optimized models of v3.3.0: faster, less memory,
# and the same results, so no jobs to run again.
MACHINE_LEARNING_MODEL_REVISION: v2
# Trust store without the compromised eMudhra
# intermediates; requests would otherwise use certifi.
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
@@ -139,11 +142,19 @@ spec:
globalMounts:
- path: /etc/ssl/certs
readOnly: true
# The models and their one-off preparation, kept across restarts.
# Downloaded again if lost, so no backups: the no-backup group
# keeps it out of Longhorn's default one (backup-daily), and
# Velero skips that label too (velero/skip-emptydir-policy.yaml).
cache:
enabled: true
# Optional: to avoid downloading the ML models every start, use
# a persistentVolumeClaim with size, accessMode and storageClass
type: emptyDir
type: persistentVolumeClaim
storageClass: longhorn
accessMode: ReadWriteOnce
size: 10Gi
labels:
recurring-job.longhorn.io/source: enabled
recurring-job-group.longhorn.io/no-backup: enabled
destination:
server: https://kubernetes.default.svc
+35
View File
@@ -0,0 +1,35 @@
# Garage on the NAS
S3 storage for Velero, on the Synology (192.168.1.99) in Docker, outside
the cluster. Replaced MinIO, which is no longer maintained. One node, data
in `/volume1/docker/garage/{meta,data}`. S3 API on port 3900, admin API on
3903, both LAN only. Set up 2026-10-25 with Garage v2.5.0.
The NAS is reached as `ssh -p 34000 rubenhensen@192.168.1.99`; docker is
in `/usr/local/bin`, which a non-interactive SSH session doesn't have on
its PATH.
## Setup
`garage.toml` here has placeholders for the two secrets; on the NAS they
were filled in once with `openssl rand -hex 32` (`rpc_secret`) and
`openssl rand -base64 32` (`admin_token`), and only exist there.
```
cd /volume1/docker/garage
docker compose up -d
g="docker exec garage /garage"
$g layout assign -z nas -c 800G <node id from $g status>
$g layout apply --version 1
$g bucket create velero-backups
$g key create velero
$g bucket allow --read --write --key velero velero-backups
```
Velero's key is in Vault as `kv/velero-s3` (`access_key_id`,
`secret_access_key`) and reaches Velero through
`velero/external-secret.yaml`. Without the cluster, `$g key info velero
--show-secret` on the NAS shows it again.
Renovate updates the image tag in `compose.yaml`, but nothing deploys it:
after merging, `docker compose pull && docker compose up -d` on the NAS.
+12
View File
@@ -0,0 +1,12 @@
services:
garage:
image: dxflrs/garage:v2.5.0
container_name: garage
restart: unless-stopped
ports:
- "3900:3900" # S3 API
- "3903:3903" # admin API
volumes:
- ./garage.toml:/etc/garage.toml:ro
- ./meta:/var/lib/garage/meta
- ./data:/var/lib/garage/data
+29
View File
@@ -0,0 +1,29 @@
# One node on the Synology. Velero's kopia data is encrypted, so
# compression would only cost CPU.
#
# SQLite rather than the default LMDB: LMDB can be corrupted by an unclean
# shutdown, and a single node has no other node to repair from. With
# metadata_fsync SQLite runs WAL with synchronous=NORMAL, which only syncs
# at checkpoints; LMDB with metadata_fsync synced every transaction, ~270
# writes/s on the NAS's two disks with Garage idle. Converted 2026-10-25
# with `garage convert-db`.
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
metadata_fsync = true
data_fsync = true
metadata_auto_snapshot_interval = "6h"
replication_factor = 1
compression_level = "none"
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "@RPC_SECRET@"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
[admin]
api_bind_addr = "[::]:3903"
admin_token = "@ADMIN_TOKEN@"
+20 -3
View File
@@ -173,12 +173,18 @@ in
# - mail.rubenhensen.nl (and anything else) → local stalwart on 8443
# - tunneled hosts → home cluster ingress on 443
# - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy)
#
# Each connection starts with a PROXY header that carries the visitor's
# address, so the home cluster sees who is asking instead of this box
# (gateway/proxy-protocol.yaml in k8scd). nginx sends the header to
# every upstream or none, so the local ones go through a hop on
# 8445/8446 that takes it off again.
streamConfig = ''
map $ssl_preread_server_name $tunnel_upstream {
${sniMapEntries}
s1.rubenhensen.nl 127.0.0.1:8444;
ntfy1.rubenhensen.nl 127.0.0.1:8444;
default 127.0.0.1:8443;
s1.rubenhensen.nl 127.0.0.1:8446;
ntfy1.rubenhensen.nl 127.0.0.1:8446;
default 127.0.0.1:8445;
}
server {
@@ -186,6 +192,17 @@ ${sniMapEntries}
listen [::]:443;
proxy_pass $tunnel_upstream;
ssl_preread on;
proxy_protocol on;
}
server {
listen 127.0.0.1:8445 proxy_protocol;
proxy_pass 127.0.0.1:8443;
}
server {
listen 127.0.0.1:8446 proxy_protocol;
proxy_pass 127.0.0.1:8444;
}
'';
+18 -2
View File
@@ -5,8 +5,16 @@
"customManagers:githubActionsVersions"
],
"argocd": {
"description": "Every app folder; Renovate's regexes are RE2, which has no lookahead, so the other folders are skipped through ignorePaths",
"managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/"
"/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
]
},
"helm-values": {
@@ -15,8 +23,16 @@
]
},
"kubernetes": {
"description": "Every app folder, as for argocd",
"managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/"
"/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
]
},
"customManagers": [
+30
View File
@@ -0,0 +1,30 @@
# Velero's key for its bucket in Garage on the NAS (nas/garage), in the
# AWS credentials file format the aws plugin and kopia read.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: velero-s3-credentials
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: velero-s3-credentials
creationPolicy: Owner
template:
engineVersion: v2
data:
cloud: |
[default]
aws_access_key_id={{ .access_key_id }}
aws_secret_access_key={{ .secret_access_key }}
data:
- secretKey: access_key_id
remoteRef:
key: kv/velero-s3
property: access_key_id
- secretKey: secret_access_key
remoteRef:
key: kv/velero-s3
property: secret_access_key
+7
View File
@@ -1,6 +1,8 @@
# Volume policy for the backup schedules. emptyDir volumes hold nothing
# worth keeping, and backing them up fails when the pod is gone before its
# volume is read, as happens with Velero's own repository maintenance jobs.
# PVCs in Longhorn's no-backup group (caches, such as immich's ML models)
# are skipped too.
apiVersion: v1
kind: ConfigMap
metadata:
@@ -14,3 +16,8 @@ data:
- emptyDir
action:
type: skip
- conditions:
pvcLabels:
recurring-job-group.longhorn.io/no-backup: enabled
action:
type: skip