Compare commits

...
12 Commits
Author SHA1 Message Date
Ruben Hensen 30540ea579 node001: send the visitor's address to the home cluster
The stream server on 443 now starts each connection with a PROXY header.
nginx sends it to every upstream or none, so Stalwart (8443) and the
local nginx for Gatus and ntfy (8444) get theirs through a hop on
8445/8446 that takes it off again; nothing changes for them.

Deploy after the Gateway accepts the header, or the tunneled hosts stop
answering until it does.
2026-10-26 23:11:38 +01:00
Ruben Hensen 53d74768ad gateway: take the visitor's address from a PROXY header
node001 passes TLS through without opening it, so every request from
the tunneled hosts arrives from 46.224.26.65. Wordfence locked that
address out after 20 failed logins on 2026-10-26, and with it everyone
who logs in to scoutingmbg.nl. With a PROXY header from node001, Envoy
puts the visitor's address in X-Forwarded-For instead.

The header is optional: s2 and ntfy2 point straight at home, and
node001's proxy on port 80 sends none either.
2026-10-26 23:11:38 +01:00
Ruben Hensen 20bd3a4bca Merge pull request #39 from rubenhensen/immich-ml-v2
immich: the optimized ML models, cached on a volume
2026-10-25 17:48:40 +01:00
Ruben Hensen 6cba1a9363 immich: the optimized ML models, cached on a volume 2026-10-25 17:46:03 +01:00
Ruben Hensen acfce6645e velero: skip PVCs in Longhorn's no-backup group 2026-10-25 17:46:03 +01:00
Ruben Hensen 0de86dd65c Merge pull request #36 from rubenhensen/renovate-re2
Renovate: skip the non-app folders without a lookahead
2026-10-25 17:41:32 +01:00
Ruben Hensen d7ffdc05f6 docs: Renovate rejected the lookahead (#32) 2026-10-25 17:33:39 +01:00
Ruben Hensen c0f8034e14 renovate: skip the non-app folders through ignorePaths, not a lookahead 2026-10-25 17:33:39 +01:00
Ruben Hensen a15a4f7c75 Merge pull request #34 from rubenhensen/velero-garage-secret
Garage on the NAS, and Velero's key for it
2026-10-25 17:23:33 +01:00
Ruben Hensen b072099cc1 nas: Garage keeps its metadata in SQLite 2026-10-25 17:08:13 +01:00
Ruben Hensen e10ca337f1 velero: a key for Garage, from Vault 2026-10-25 16:28:43 +01:00
Ruben Hensen c72b08867f nas: Garage, which takes over from MinIO 2026-10-25 16:28:43 +01:00
10 changed files with 198 additions and 12 deletions

No files matched your search

+10 -4
View File
@@ -360,10 +360,16 @@ PR before Argo CD applies it.
cover every app folder, a regex manager reads `imageName` in cover every app folder, a regex manager reads `imageName` in
`*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off `*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off
(those need pg_upgrade or a dump). Expect a batch of new PRs (those need pg_upgrade or a dump). Expect a batch of new PRs
(2026-10-25; `renovate-config-validator` passes). They only come once (2026-10-25). They only come once fewer than 10 Renovate PRs are open
fewer than 10 Renovate PRs are open (`config:recommended`'s (`config:recommended`'s `prConcurrentLimit`); until then they wait
`prConcurrentLimit`); until then they wait under "Rate-Limited" on the under "Rate-Limited" on the Dependency Dashboard (#2)
Dependency Dashboard (#2) - The other folders were left out with a `(?!…)` lookahead. Renovate
compiles these patterns with RE2, which has no lookahead, so it
rejected the config and stopped opening PRs (#32). Each manager now
takes every folder and skips the others through its own
`ignorePaths`. Renovate 44.181.6 with RE2 loaded fails the old config
with #32's error and passes the new one, and its matcher picks the
same 166 files as the old pattern did
- [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager - [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager
v1.21.3 (security release; all three pods on v1.21.3), #24 v1.21.3 (security release; all three pods on v1.21.3), #24
external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as
+23
View File
@@ -0,0 +1,23 @@
# node001 passes TLS for the tunneled hosts through without opening it, so
# without this every request arrives from 46.224.26.65. Wordfence locks that
# address out after 20 failed logins (2026-10-26), and with it everyone who
# logs in to scoutingmbg.nl. node001 puts a PROXY header with the visitor's
# address in front of each connection; Envoy takes the address from there
# and adds it to X-Forwarded-For.
#
# Optional, because s2 and ntfy2 point straight at home and their clients
# send no header, and neither does node001's proxy on port 80. The price:
# whoever connects to the home address directly can send a header of their
# own and claim any address.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: proxy-protocol
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: homelab
proxyProtocol:
optional: true
+14 -3
View File
@@ -127,6 +127,9 @@ spec:
TRANSFORMERS_CACHE: /cache TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet HF_XET_CACHE: /cache/huggingface-xet
MPLCONFIGDIR: /cache/matplotlib-config MPLCONFIGDIR: /cache/matplotlib-config
# The optimized models of v3.3.0: faster, less memory,
# and the same results, so no jobs to run again.
MACHINE_LEARNING_MODEL_REVISION: v2
# Trust store without the compromised eMudhra # Trust store without the compromised eMudhra
# intermediates; requests would otherwise use certifi. # intermediates; requests would otherwise use certifi.
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
@@ -139,11 +142,19 @@ spec:
globalMounts: globalMounts:
- path: /etc/ssl/certs - path: /etc/ssl/certs
readOnly: true readOnly: true
# The models and their one-off preparation, kept across restarts.
# Downloaded again if lost, so no backups: the no-backup group
# keeps it out of Longhorn's default one (backup-daily), and
# Velero skips that label too (velero/skip-emptydir-policy.yaml).
cache: cache:
enabled: true enabled: true
# Optional: to avoid downloading the ML models every start, use type: persistentVolumeClaim
# a persistentVolumeClaim with size, accessMode and storageClass storageClass: longhorn
type: emptyDir accessMode: ReadWriteOnce
size: 10Gi
labels:
recurring-job.longhorn.io/source: enabled
recurring-job-group.longhorn.io/no-backup: enabled
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
+35
View File
@@ -0,0 +1,35 @@
# Garage on the NAS
S3 storage for Velero, on the Synology (192.168.1.99) in Docker, outside
the cluster. Replaced MinIO, which is no longer maintained. One node, data
in `/volume1/docker/garage/{meta,data}`. S3 API on port 3900, admin API on
3903, both LAN only. Set up 2026-10-25 with Garage v2.5.0.
The NAS is reached as `ssh -p 34000 rubenhensen@192.168.1.99`; docker is
in `/usr/local/bin`, which a non-interactive SSH session doesn't have on
its PATH.
## Setup
`garage.toml` here has placeholders for the two secrets; on the NAS they
were filled in once with `openssl rand -hex 32` (`rpc_secret`) and
`openssl rand -base64 32` (`admin_token`), and only exist there.
```
cd /volume1/docker/garage
docker compose up -d
g="docker exec garage /garage"
$g layout assign -z nas -c 800G <node id from $g status>
$g layout apply --version 1
$g bucket create velero-backups
$g key create velero
$g bucket allow --read --write --key velero velero-backups
```
Velero's key is in Vault as `kv/velero-s3` (`access_key_id`,
`secret_access_key`) and reaches Velero through
`velero/external-secret.yaml`. Without the cluster, `$g key info velero
--show-secret` on the NAS shows it again.
Renovate updates the image tag in `compose.yaml`, but nothing deploys it:
after merging, `docker compose pull && docker compose up -d` on the NAS.
+12
View File
@@ -0,0 +1,12 @@
services:
garage:
image: dxflrs/garage:v2.5.0
container_name: garage
restart: unless-stopped
ports:
- "3900:3900" # S3 API
- "3903:3903" # admin API
volumes:
- ./garage.toml:/etc/garage.toml:ro
- ./meta:/var/lib/garage/meta
- ./data:/var/lib/garage/data
+29
View File
@@ -0,0 +1,29 @@
# One node on the Synology. Velero's kopia data is encrypted, so
# compression would only cost CPU.
#
# SQLite rather than the default LMDB: LMDB can be corrupted by an unclean
# shutdown, and a single node has no other node to repair from. With
# metadata_fsync SQLite runs WAL with synchronous=NORMAL, which only syncs
# at checkpoints; LMDB with metadata_fsync synced every transaction, ~270
# writes/s on the NAS's two disks with Garage idle. Converted 2026-10-25
# with `garage convert-db`.
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
metadata_fsync = true
data_fsync = true
metadata_auto_snapshot_interval = "6h"
replication_factor = 1
compression_level = "none"
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "@RPC_SECRET@"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
[admin]
api_bind_addr = "[::]:3903"
admin_token = "@ADMIN_TOKEN@"
+20 -3
View File
@@ -173,12 +173,18 @@ in
# - mail.rubenhensen.nl (and anything else) → local stalwart on 8443 # - mail.rubenhensen.nl (and anything else) → local stalwart on 8443
# - tunneled hosts → home cluster ingress on 443 # - tunneled hosts → home cluster ingress on 443
# - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy) # - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy)
#
# Each connection starts with a PROXY header that carries the visitor's
# address, so the home cluster sees who is asking instead of this box
# (gateway/proxy-protocol.yaml in k8scd). nginx sends the header to
# every upstream or none, so the local ones go through a hop on
# 8445/8446 that takes it off again.
streamConfig = '' streamConfig = ''
map $ssl_preread_server_name $tunnel_upstream { map $ssl_preread_server_name $tunnel_upstream {
${sniMapEntries} ${sniMapEntries}
s1.rubenhensen.nl 127.0.0.1:8444; s1.rubenhensen.nl 127.0.0.1:8446;
ntfy1.rubenhensen.nl 127.0.0.1:8444; ntfy1.rubenhensen.nl 127.0.0.1:8446;
default 127.0.0.1:8443; default 127.0.0.1:8445;
} }
server { server {
@@ -186,6 +192,17 @@ ${sniMapEntries}
listen [::]:443; listen [::]:443;
proxy_pass $tunnel_upstream; proxy_pass $tunnel_upstream;
ssl_preread on; ssl_preread on;
proxy_protocol on;
}
server {
listen 127.0.0.1:8445 proxy_protocol;
proxy_pass 127.0.0.1:8443;
}
server {
listen 127.0.0.1:8446 proxy_protocol;
proxy_pass 127.0.0.1:8444;
} }
''; '';
+18 -2
View File
@@ -5,8 +5,16 @@
"customManagers:githubActionsVersions" "customManagers:githubActionsVersions"
], ],
"argocd": { "argocd": {
"description": "Every app folder; Renovate's regexes are RE2, which has no lookahead, so the other folders are skipped through ignorePaths",
"managerFilePatterns": [ "managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/" "/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
] ]
}, },
"helm-values": { "helm-values": {
@@ -15,8 +23,16 @@
] ]
}, },
"kubernetes": { "kubernetes": {
"description": "Every app folder, as for argocd",
"managerFilePatterns": [ "managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/" "/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
] ]
}, },
"customManagers": [ "customManagers": [
+30
View File
@@ -0,0 +1,30 @@
# Velero's key for its bucket in Garage on the NAS (nas/garage), in the
# AWS credentials file format the aws plugin and kopia read.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: velero-s3-credentials
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: velero-s3-credentials
creationPolicy: Owner
template:
engineVersion: v2
data:
cloud: |
[default]
aws_access_key_id={{ .access_key_id }}
aws_secret_access_key={{ .secret_access_key }}
data:
- secretKey: access_key_id
remoteRef:
key: kv/velero-s3
property: access_key_id
- secretKey: secret_access_key
remoteRef:
key: kv/velero-s3
property: secret_access_key
+7
View File
@@ -1,6 +1,8 @@
# Volume policy for the backup schedules. emptyDir volumes hold nothing # Volume policy for the backup schedules. emptyDir volumes hold nothing
# worth keeping, and backing them up fails when the pod is gone before its # worth keeping, and backing them up fails when the pod is gone before its
# volume is read, as happens with Velero's own repository maintenance jobs. # volume is read, as happens with Velero's own repository maintenance jobs.
# PVCs in Longhorn's no-backup group (caches, such as immich's ML models)
# are skipped too.
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@@ -14,3 +16,8 @@ data:
- emptyDir - emptyDir
action: action:
type: skip type: skip
- conditions:
pvcLabels:
recurring-job-group.longhorn.io/no-backup: enabled
action:
type: skip