Compare commits

...
Author SHA1 Message Date
Ruben Hensen a8ccbef274 envoy-gateway-system: NetworkPolicy 2026-10-26 23:28:05 +01:00
Ruben Hensen 0dc69d34f0 Merge pull request #50 from rubenhensen/proxy-protocol
Pass the visitor's address from node001 to the home cluster
2026-10-26 23:12:14 +01:00
Ruben Hensen 30540ea579 node001: send the visitor's address to the home cluster
The stream server on 443 now starts each connection with a PROXY header.
nginx sends it to every upstream or none, so Stalwart (8443) and the
local nginx for Gatus and ntfy (8444) get theirs through a hop on
8445/8446 that takes it off again; nothing changes for them.

Deploy after the Gateway accepts the header, or the tunneled hosts stop
answering until it does.
2026-10-26 23:11:38 +01:00
Ruben Hensen 53d74768ad gateway: take the visitor's address from a PROXY header
node001 passes TLS through without opening it, so every request from
the tunneled hosts arrives from 46.224.26.65. Wordfence locked that
address out after 20 failed logins on 2026-10-26, and with it everyone
who logs in to scoutingmbg.nl. With a PROXY header from node001, Envoy
puts the visitor's address in X-Forwarded-For instead.

The header is optional: s2 and ntfy2 point straight at home, and
node001's proxy on port 80 sends none either.
2026-10-26 23:11:38 +01:00
Ruben Hensen b6462fb3a9 Merge pull request #49 from rubenhensen/roadmap-garage
docs: Velero on Garage
2026-10-26 23:04:15 +01:00
Ruben Hensen bed68c70a8 velero: the switch to Garage was on 10-26 2026-10-26 22:57:35 +01:00
Ruben Hensen 3991a8d726 docs: Velero on Garage, and the off-site copy from the bucket 2026-10-26 22:57:35 +01:00
5 changed files with 221 additions and 34 deletions

No files matched your search

+57 -30
View File
@@ -10,11 +10,11 @@ One node: opti1, an OptiPlex 5060 with an i5-8600T, 32 GB and one 1 TB
SATA SSD that is 72% full. microk8s v1.36.5 from 1.36/edge, snap held. SATA SSD that is 72% full. microk8s v1.36.5 from 1.36/edge, snap held.
35 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn 35 Argo CD apps (with metrics-server), all Synced/Healthy; every Longhorn
volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the volume has one replica. authentik 2026.8.5, Vault 2.2.1. Backups go to the
NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to MinIO at NAS (192.168.1.99): Longhorn to NFS at 00:00 (7 kept), Velero to Garage
01:00 (daily kept 30 days, weekly 90 days). Gatus on s1 (node001, Hetzner) (MinIO until 2026-10-26) at 01:00 (daily kept 30 days, weekly 90 days).
and s2 (home) watch each other and alert through ntfy; s2 also gets an Gatus on s1 (node001, Hetzner) and s2 (home) watch each other and alert
hourly report from inside the cluster on the backups, Vault's seal, the through ntfy; s2 also gets an hourly report from inside the cluster on the
Argo CD apps and the CNPG clusters. backups, Vault's seal, the Argo CD apps and the CNPG clusters.
Every app namespace has NetworkPolicies, and CI renders and validates all Every app namespace has NetworkPolicies, and CI renders and validates all
apps on every PR. node001 runs mail and tunnels the public hosts home. apps on every PR. node001 runs mail and tunnels the public hosts home.
@@ -63,24 +63,25 @@ belong to the group.
There is no off-site copy (2026-10-25). The plan: swap S3 buckets with a There is no off-site copy (2026-10-25). The plan: swap S3 buckets with a
friend. friend.
- [ ] Ours to his bucket: Hyper Backup on the Synology, destination "S3 - [ ] Ours to his bucket: `rclone sync` from Garage's `velero-backups`
Storage" with his endpoint as a custom server. Client-side encryption (~190 GB) to his bucket through an rclone `crypt` remote, in Docker on
on: the kopia repositories use Velero's default password the NAS like the copy from MinIO (see 5). Encrypted on our side: the
(`static-passw0rd`), so without it he could read everything, the kopia repositories use Velero's default password (`static-passw0rd`),
group's photos included. Keep the encryption key next to the Vault so without it he could read everything, the group's photos included.
unseal key. Source: the shared folder with MinIO's `velero-backups` Keep the crypt passwords next to the Vault unseal key. Daily after the
bucket (~150-200 GB). Daily after the Velero backup, Smart Recycle, Velero backup, ntfy on failure. Hyper Backup, the earlier plan, backs up
mail on failure shared folders; Garage keeps its objects as blocks plus a metadata
database, and a copy of that folder taken while Garage runs is not
consistent
- [ ] Needed from him: endpoint, which S3 server he runs, space (~200 GB - [ ] Needed from him: endpoint, which S3 server he runs, space (~200 GB
now, 500 GB to grow), and versioning or object lock on the bucket, so now, 500 GB to grow), and versioning or object lock on the bucket, so
whoever takes over the NAS can't delete the off-site copy as well whoever takes over the NAS can't delete the off-site copy as well
- [ ] His backups with us: Garage (not MinIO, see 5) in Container Manager - [ ] His backups with us: a bucket and key for him, with a quota, in the
on the Synology, in its own shared folder, with a bucket and key for him Garage on the NAS (`nas/garage`, see 5). Only the S3 API public, as
and a quota. Only the S3 API public, as `s3.rubenhensen.nl` through `s3.rubenhensen.nl` through node001 and Envoy with a cert-manager
node001 and Envoy with a cert-manager certificate; the admin API stays certificate; the admin API stays on the LAN. The cluster side (DNS,
on the LAN. The cluster side (DNS, node001 tunnel, Certificate, node001 tunnel, Certificate, HTTPRoute) goes in git. Needed: how much
HTTPRoute) goes in git. Needed: how much space he wants, free space on space he wants, free space on the NAS
the NAS
- [ ] Decide which copy of the immich library goes off-site (see 5) - [ ] Decide which copy of the immich library goes off-site (see 5)
- [ ] A restore test that runs on a schedule. `backup-testing/` predates - [ ] A restore test that runs on a schedule. `backup-testing/` predates
Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth. Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth.
@@ -108,9 +109,12 @@ friend.
MinIO runs region `minio`; the script now reads endpoint, region and MinIO runs region `minio`; the script now reads endpoint, region and
bucket from Velero's location. Copying freshrss's kopia repository took bucket from Velero's location. Copying freshrss's kopia repository took
14 of the ~17 minutes 14 of the ~17 minutes
- [ ] Run it on a schedule. It needs Docker and the MinIO key, so not in - [x] Ran it again on 2026-10-26 against the copy in Garage
the cluster itself: the Mac (launchd) or node001, ideally with a (daily-backup-20261026010028, same namespaces), before Velero switched
read-only MinIO user over (see 5): passed, in 5 minutes instead of 17
- [ ] Run it on a schedule. It needs Docker and an S3 key, so not in the
cluster itself: the Mac (launchd) or node001, ideally with a read-only
Garage key (`garage key create`, `garage bucket allow --read`)
### 3. authentik 2026.2.1 → 2026.8.5 ### 3. authentik 2026.2.1 → 2026.8.5
@@ -193,7 +197,7 @@ Findings from 2026-10-25:
- Velero copies the CNPG and MariaDB data directories file by file while - Velero copies the CNPG and MariaDB data directories file by file while
the databases run, which is not a consistent copy. The Longhorn backups the databases run, which is not a consistent copy. The Longhorn backups
start from a snapshot and are crash-consistent. CNPG can back up to start from a snapshot and are crash-consistent. CNPG can back up to
MinIO itself (barman cloud plugin), with point-in-time recovery. Garage itself (barman cloud plugin), with point-in-time recovery.
- The PartiallyFailed daily backup of 10-23 (emptyDirs of Velero's own - The PartiallyFailed daily backup of 10-23 (emptyDirs of Velero's own
maintenance job pods) was fixed that afternoon by maintenance job pods) was fixed that afternoon by
`velero/skip-emptydir-policy.yaml`. `velero/skip-emptydir-policy.yaml`.
@@ -201,10 +205,31 @@ Findings from 2026-10-25:
- [ ] Whether the immich library stays in Velero. It stays for now - [ ] Whether the immich library stays in Velero. It stays for now
(2026-10-25); decide again with the off-site copy (2) (2026-10-25); decide again with the off-site copy (2)
- [ ] CNPG backups through the barman cloud plugin - [ ] CNPG backups through the barman cloud plugin
- [ ] Velero's target, MinIO on the NAS, is no longer maintained: no - [x] Velero's target, MinIO on the NAS, is no longer maintained: no
community images since October 2025, maintenance mode since December community images since October 2025, maintenance mode since December
2025, and the repository is archived. Move Velero to the Garage from 2 2025, and the repository is archived. Since 2026-10-26 Velero backs up
and switch MinIO off to Garage on the same NAS (`nas/garage`, PR #35). The bucket was copied
with rclone (185 GiB in about 6 hours, then a catch-up sync just before
the switch), so the existing backups and kopia repositories carried on:
after the switch all 55 backups were still listed, and a manual backup
of ntfy found the existing repository and used the night's snapshot as
its parent. Two things made the NAS slow during the copy: Garage on
LMDB with `metadata_fsync`, ~270 disk writes/s while idle (now SQLite,
see `nas/garage/garage.toml`), and MinIO's scanner, which kept one disk
at 97% busy. The scanner is now at `slowest` (`mc admin config set local
scanner speed=slowest` in the MinIO container); it may have slowed
Velero's backups to MinIO as well
- [ ] Switch MinIO off once Velero has run on Garage for a week (from
2026-11-02): stop the container, then delete `/volume1/minio` (343 GiB)
and the hand-made Secret `velero/cloud-credentials` (MinIO's key). Only
MinIO still has the kopia repositories of namespaces that are gone
(~156 GB: nextcloud, ttrss, whisper-priv, whisper-webui, blog, ghost,
mbg-wordpress, sogo, sogo-mail, outrun-game, owntracks); decide first
whether any of them is worth keeping
- [ ] Garage's SQLite WAL (`meta/db.sqlite-wal`) was 1.7 GB on
2026-10-26 next to a 270 MB database, and going by the folders in
`meta/snapshots`, each 6-hourly metadata snapshot takes about 70
minutes. Not growing; check again after a week of normal use
### 6. Monitoring ### 6. Monitoring
@@ -323,8 +348,10 @@ Counted 2026-10-25 over the 73 running containers:
### 8. Renovate and CI ### 8. Renovate and CI
10 Renovate PRs are open, the oldest from February, and nothing checks a On 2026-10-25 10 Renovate PRs were open, the oldest from February, and
PR before Argo CD applies it. nothing checked a PR before Argo CD applied it. All of them have since been
merged or closed (below), none are open (checked 2026-10-25), and CI checks
every PR.
- [x] `.github/workflows/validate.yaml` runs `tools/ci/validate.py` on - [x] `.github/workflows/validate.yaml` runs `tools/ci/validate.py` on
every PR and push (2026-10-25): it renders all 36 apps the way Argo CD every PR and push (2026-10-25): it renders all 36 apps the way Argo CD
@@ -446,7 +473,7 @@ critical / high:
| tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag | | tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag |
| longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below | | longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below |
| actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it | | actualbudget/actual-server:26.6.0 | 8 | 83 | chart 1.8.9; Renovate now covers it |
| gitea/gitea:1.27.3 | 8 | 43 | PR #26 (28.2.0) | | gitea/gitea:1.27.3 | 8 | 43 | now 28.2.0 (PR #26, 2026-10-25); not scanned again |
| immich-server / machine-learning v3.3.5 | 7, 7 | 113, 78 | a newer immich release | | immich-server / machine-learning v3.3.5 | 7, 7 | 113, 78 | a newer immich release |
| towfiqi/serpbear:3.1.0 | 6 | 77 | a newer release | | towfiqi/serpbear:3.1.0 | 6 | 77 | a newer release |
| rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image | | rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image |
+120
View File
@@ -0,0 +1,120 @@
# Envoy Gateway: the controller and the proxy it runs (envoyproxy.yaml).
# Both reach each other (the proxy gets its config from the controller on
# 18000) and DNS, and take connections from each other and the node: the
# API server calls the controller's topology webhook (9443) from there and
# the kubelet probes come from there too (10.1.7.64 is the node's Calico
# VXLAN address, projectcalico.org/IPv4VXLANTunnelAddr). Nothing scrapes
# the metrics on 19001.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default
namespace: envoy-gateway-system
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {}
- from:
- ipBlock:
cidr: 192.168.1.204/32
- ipBlock:
cidr: 10.1.7.64/32
egress:
- to:
- podSelector: {}
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# The proxy holds the node's ports 80 and 443 (10080 and 10443 inside) and
# takes connections there from anywhere: the internet through node001 and
# the router, the LAN, and pods that call a public hostname. It reaches
# every app it routes to (pods in any namespace; their own policies decide
# which port), Home Assistant on 192.168.1.202:8123 (outside the cluster,
# home-assistant/), and the internet on 443 for Longhorn's OIDC login
# (authentik's token endpoint, at its public address).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: proxy
namespace: envoy-gateway-system
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: envoy
app.kubernetes.io/component: proxy
policyTypes:
- Ingress
- Egress
ingress:
- ports:
- port: 10080
- port: 10443
egress:
- to:
- namespaceSelector: {}
- to:
- ipBlock:
cidr: 192.168.1.202/32
ports:
- port: 8123
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 100.64.0.0/10
- 169.254.0.0/16
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
---
# Everything but the proxy (the controller, and the chart's certgen job on
# upgrades) reaches the API server (the node, port 16443). The controller
# also fetches the OIDC discovery document of Longhorn's SecurityPolicy
# from authentik's public address.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: control-plane
namespace: envoy-gateway-system
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- envoy
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 192.168.1.204/32
ports:
- port: 16443
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 100.64.0.0/10
- 169.254.0.0/16
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
+23
View File
@@ -0,0 +1,23 @@
# node001 passes TLS for the tunneled hosts through without opening it, so
# without this every request arrives from 46.224.26.65. Wordfence locks that
# address out after 20 failed logins (2026-10-26), and with it everyone who
# logs in to scoutingmbg.nl. node001 puts a PROXY header with the visitor's
# address in front of each connection; Envoy takes the address from there
# and adds it to X-Forwarded-For.
#
# Optional, because s2 and ntfy2 point straight at home and their clients
# send no header, and neither does node001's proxy on port 80. The price:
# whoever connects to the home address directly can send a header of their
# own and claim any address.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: proxy-protocol
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: homelab
proxyProtocol:
optional: true
+20 -3
View File
@@ -173,12 +173,18 @@ in
# - mail.rubenhensen.nl (and anything else) → local stalwart on 8443 # - mail.rubenhensen.nl (and anything else) → local stalwart on 8443
# - tunneled hosts → home cluster ingress on 443 # - tunneled hosts → home cluster ingress on 443
# - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy) # - s1 / ntfy1.rubenhensen.nl → local nginx on 8444 (Gatus, ntfy)
#
# Each connection starts with a PROXY header that carries the visitor's
# address, so the home cluster sees who is asking instead of this box
# (gateway/proxy-protocol.yaml in k8scd). nginx sends the header to
# every upstream or none, so the local ones go through a hop on
# 8445/8446 that takes it off again.
streamConfig = '' streamConfig = ''
map $ssl_preread_server_name $tunnel_upstream { map $ssl_preread_server_name $tunnel_upstream {
${sniMapEntries} ${sniMapEntries}
s1.rubenhensen.nl 127.0.0.1:8444; s1.rubenhensen.nl 127.0.0.1:8446;
ntfy1.rubenhensen.nl 127.0.0.1:8444; ntfy1.rubenhensen.nl 127.0.0.1:8446;
default 127.0.0.1:8443; default 127.0.0.1:8445;
} }
server { server {
@@ -186,6 +192,17 @@ ${sniMapEntries}
listen [::]:443; listen [::]:443;
proxy_pass $tunnel_upstream; proxy_pass $tunnel_upstream;
ssl_preread on; ssl_preread on;
proxy_protocol on;
}
server {
listen 127.0.0.1:8445 proxy_protocol;
proxy_pass 127.0.0.1:8443;
}
server {
listen 127.0.0.1:8446 proxy_protocol;
proxy_pass 127.0.0.1:8444;
} }
''; '';
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
- mountPath: /target - mountPath: /target
name: plugins name: plugins
configuration: configuration:
# Garage on the NAS (nas/garage). Until 2026-10-25 this was MinIO on # Garage on the NAS (nas/garage). Until 2026-10-26 this was MinIO on
# the same NAS; the bucket was copied over with rclone, so the # the same NAS; the bucket was copied over with rclone, so the
# existing backups and kopia repositories carried on. # existing backups and kopia repositories carried on.
backupStorageLocation: backupStorageLocation: