mirror of
https://github.com/rubenhensen/k8scd.git
synced 2026-11-15 22:29:09 +01:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a8ccbef274 |
No files matched your search
+15
-36
@@ -38,8 +38,7 @@ docs/longhorn-second-ssd.md:
|
|||||||
2026-10-25). The snap is held, so microk8s gets no updates at all until
|
2026-10-25). The snap is held, so microk8s gets no updates at all until
|
||||||
someone refreshes it
|
someone refreshes it
|
||||||
- [ ] Second SSD, docs/longhorn-second-ssd.md
|
- [ ] Second SSD, docs/longhorn-second-ssd.md
|
||||||
- [x] `MACHINE_LEARNING_MODEL_REVISION=v2` for immich ML (PR #39,
|
- [ ] Optional: `MACHINE_LEARNING_MODEL_REVISION=v2` for immich ML
|
||||||
2026-10-25; the models are cached on a volume)
|
|
||||||
|
|
||||||
## Next, in order
|
## Next, in order
|
||||||
|
|
||||||
@@ -83,9 +82,7 @@ friend.
|
|||||||
certificate; the admin API stays on the LAN. The cluster side (DNS,
|
certificate; the admin API stays on the LAN. The cluster side (DNS,
|
||||||
node001 tunnel, Certificate, HTTPRoute) goes in git. Needed: how much
|
node001 tunnel, Certificate, HTTPRoute) goes in git. Needed: how much
|
||||||
space he wants, free space on the NAS
|
space he wants, free space on the NAS
|
||||||
- [x] Which copy of the immich library goes off-site: Velero's, with the
|
- [ ] Decide which copy of the immich library goes off-site (see 5)
|
||||||
rest of the bucket (decided 2026-10-26). It is most of the ~190 GB, so
|
|
||||||
ask him for room for it
|
|
||||||
- [ ] A restore test that runs on a schedule. `backup-testing/` predates
|
- [ ] A restore test that runs on a schedule. `backup-testing/` predates
|
||||||
Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth.
|
Envoy Gateway and still rewrites `*.hensen.io` and bypasses sol-auth.
|
||||||
Planned on 2026-10-25, not run: a second Velero on the real bucket is
|
Planned on 2026-10-25, not run: a second Velero on the real bucket is
|
||||||
@@ -205,8 +202,8 @@ Findings from 2026-10-25:
|
|||||||
maintenance job pods) was fixed that afternoon by
|
maintenance job pods) was fixed that afternoon by
|
||||||
`velero/skip-emptydir-policy.yaml`.
|
`velero/skip-emptydir-policy.yaml`.
|
||||||
|
|
||||||
- [x] Whether the immich library stays in Velero: it stays, since
|
- [ ] Whether the immich library stays in Velero. It stays for now
|
||||||
Velero's bucket is what goes off-site (2; decided 2026-10-26)
|
(2026-10-25); decide again with the off-site copy (2)
|
||||||
- [ ] CNPG backups through the barman cloud plugin
|
- [ ] CNPG backups through the barman cloud plugin
|
||||||
- [x] Velero's target, MinIO on the NAS, is no longer maintained: no
|
- [x] Velero's target, MinIO on the NAS, is no longer maintained: no
|
||||||
community images since October 2025, maintenance mode since December
|
community images since October 2025, maintenance mode since December
|
||||||
@@ -301,22 +298,10 @@ Counted 2026-10-25 over the 73 running containers:
|
|||||||
OIDC logins of Vault and FreshRSS reach authentik, all 24
|
OIDC logins of Vault and FreshRSS reach authentik, all 24
|
||||||
ExternalSecrets refresh, raft intact; Gitea no longer reaches the NAS
|
ExternalSecrets refresh, raft intact; Gitea no longer reaches the NAS
|
||||||
or Vault, and a pod in `default` no longer reaches Vault or Gitea
|
or Vault, and a pod in `default` no longer reaches Vault or Gitea
|
||||||
- [ ] Certificates couldn't renew in those 11 namespaces: Envoy may only
|
|
||||||
reach each app's own port, and cert-manager's HTTP-01 solver listens on
|
|
||||||
8089. ynab's renewal hung from 2026-10-26 17:35 UTC; a policy for the
|
|
||||||
solver pods, applied by hand in actual-budget that evening, let it
|
|
||||||
through within 20 seconds (now valid until 2027-01-24). PR #51 adds it
|
|
||||||
to all 11. Without it the next renewals hang too: gitea on 11-15,
|
|
||||||
bolletjestrui on 11-18, the rest from 12-03
|
|
||||||
- [ ] The infrastructure namespaces (cert-manager, cnpg-system,
|
- [ ] The infrastructure namespaces (cert-manager, cnpg-system,
|
||||||
external-secrets, envoy-gateway-system, velero, mariadb-system, dns,
|
external-secrets, envoy-gateway-system, velero, mariadb-system, dns,
|
||||||
default) still have none; `default` can reach the LAN and everything
|
default) still have none; `default` can reach the LAN and everything
|
||||||
without a policy. Prepared on 2026-10-26, not yet merged: PR #53 for
|
without a policy
|
||||||
all but Envoy (one commit per namespace; `default` denies everything),
|
|
||||||
PR #54 for envoy-gateway-system on its own, since a mistake there takes
|
|
||||||
every public host down. Merge them when someone is home to check: the
|
|
||||||
cert-manager, CNPG, ESO and MariaDB webhooks fail closed. Velero's
|
|
||||||
policy only lets it reach Garage, not MinIO's port 9000
|
|
||||||
- [x] Requests: PR #33, merged 2026-10-25 11:23 UTC, for everything that
|
- [x] Requests: PR #33, merged 2026-10-25 11:23 UTC, for everything that
|
||||||
had none except the Vault server, from `kubectl top` every 5 minutes
|
had none except the Vault server, from `kubectl top` every 5 minutes
|
||||||
from 03:13 to 08:40 UTC (highest memory plus 10%, CPU at p95, 10m
|
from 03:13 to 08:40 UTC (highest memory plus 10%, CPU at p95, 10m
|
||||||
@@ -349,14 +334,10 @@ Counted 2026-10-25 over the 73 running containers:
|
|||||||
`litespeed-cache` only caches pages on a LiteSpeed server, so it does
|
`litespeed-cache` only caches pages on a LiteSpeed server, so it does
|
||||||
little here, `wordpress-importer` is usually for a one-off import, and
|
little here, `wordpress-importer` is usually for a one-off import, and
|
||||||
`code-snippets` runs PHP from the dashboard
|
`code-snippets` runs PHP from the dashboard
|
||||||
- [ ] Rate limiting per visitor in Envoy, on hold. Since PR #50
|
- [ ] Rate limiting per visitor in Envoy needs the client's address, which
|
||||||
(2026-10-26) node001 passes the visitor's address on with the PROXY
|
node001 loses: it passes TLS through, so every request arrives from
|
||||||
protocol, and the apps see it: authentik logs it, Apache logs it for
|
46.224.26.65. That needs the PROXY protocol from node001's nginx to Envoy
|
||||||
WordPress, so Wordfence's lockout after 20 failed logins now hits one
|
(ClientTrafficPolicy `enableProxyProtocol`)
|
||||||
visitor instead of everyone. Envoy Gateway's local rate limit can't
|
|
||||||
count per address (no `Distinct`); that takes its global rate limit,
|
|
||||||
with a Redis and a ratelimit service next to it. Not worth two more
|
|
||||||
components on the one node while the apps' own limits work
|
|
||||||
- [x] Probes for gitea (`/api/healthz`), freshrss (TCP, since
|
- [x] Probes for gitea (`/api/healthz`), freshrss (TCP, since
|
||||||
mod_auth_openidc answers 401 to non-browsers) and serpbear (`/login`)
|
mod_auth_openidc answers 401 to non-browsers) and serpbear (`/login`)
|
||||||
(2026-10-25). Liveness only after 5 failures 30s apart, to ride out the
|
(2026-10-25). Liveness only after 5 failures 30s apart, to ride out the
|
||||||
@@ -394,10 +375,7 @@ every PR.
|
|||||||
fail the run; tested with a planted Secret and an untagged image), plus
|
fail the run; tested with a planted Secret and an untagged image), plus
|
||||||
PR annotations for containers without requests or probes. freshrss's
|
PR annotations for containers without requests or probes. freshrss's
|
||||||
`alpine/git` init container had no tag; now `v2.52.0`
|
`alpine/git` init container had no tag; now `v2.52.0`
|
||||||
- [ ] Automerge patch versions of low-risk apps now that CI runs: PR #52
|
- [ ] Automerge patch versions of low-risk apps now that CI runs
|
||||||
(ntfy, gatus, freshrss, serpbear, actual-budget and CI's own tools;
|
|
||||||
three days after the release, only on weekday evenings and at the
|
|
||||||
weekend, when the cluster can be reached)
|
|
||||||
- [x] Renovate listed external-snapshotter twice (#15 and #24): its
|
- [x] Renovate listed external-snapshotter twice (#15 and #24): its
|
||||||
`argocd` manager and a custom `github-tags` regex both matched. The
|
`argocd` manager and a custom `github-tags` regex both matched. The
|
||||||
regex is gone; Renovate should close #15 by itself
|
regex is gone; Renovate should close #15 by itself
|
||||||
@@ -454,10 +432,11 @@ step after that, if downtime starts to matter.
|
|||||||
there and the volumes hold nothing (0 bytes in every backup). Drop both
|
there and the volumes hold nothing (0 bytes in every backup). Drop both
|
||||||
mounts and the two PVCs, or make them ConfigMap files if custom config
|
mounts and the two PVCs, or make them ConfigMap files if custom config
|
||||||
is wanted
|
is wanted
|
||||||
- [x] The namespace `secrets` was already gone on 2026-10-26
|
- [ ] `kubectl delete namespace secrets`, empty since its app went
|
||||||
- [x] The Failed Velero backups (the dailies from 2026-09-26 to 10-21 and
|
- [ ] 41 Failed Velero backups: the dailies from 2026-09-26 to 10-21 and
|
||||||
the weeklies since 08-02; 40 by then, one had expired), deleted through
|
the weeklies since 08-02. Velero doesn't restore from a Failed backup;
|
||||||
Velero on 2026-10-26. Left: 13 Completed, 3 PartiallyFailed
|
left alone they expire by 2027-01-16.
|
||||||
|
`kubectl -n velero exec deploy/velero -c velero -- /velero backup delete $(kubectl -n velero get backups.velero.io -o json | jq -r '.items[] | select(.status.phase=="Failed") | .metadata.name' | xargs) --confirm`
|
||||||
- [ ] The mariadb-operator Backup `mbgwp-pre-migration-backup`, a dump
|
- [ ] The mariadb-operator Backup `mbgwp-pre-migration-backup`, a dump
|
||||||
from 2026-02-22 on its own 10 GiB volume (which has a Longhorn backup
|
from 2026-02-22 on its own 10 GiB volume (which has a Longhorn backup
|
||||||
from 2026-10-22). The PVC has no owner, so it goes separately:
|
from 2026-10-22). The PVC has no owner, so it goes separately:
|
||||||
|
|||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# Envoy Gateway: the controller and the proxy it runs (envoyproxy.yaml).
|
||||||
|
# Both reach each other (the proxy gets its config from the controller on
|
||||||
|
# 18000) and DNS, and take connections from each other and the node: the
|
||||||
|
# API server calls the controller's topology webhook (9443) from there and
|
||||||
|
# the kubelet probes come from there too (10.1.7.64 is the node's Calico
|
||||||
|
# VXLAN address, projectcalico.org/IPv4VXLANTunnelAddr). Nothing scrapes
|
||||||
|
# the metrics on 19001.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: default
|
||||||
|
namespace: envoy-gateway-system
|
||||||
|
spec:
|
||||||
|
podSelector: {}
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- podSelector: {}
|
||||||
|
- from:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 192.168.1.204/32
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 10.1.7.64/32
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- podSelector: {}
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
k8s-app: kube-dns
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
---
|
||||||
|
# The proxy holds the node's ports 80 and 443 (10080 and 10443 inside) and
|
||||||
|
# takes connections there from anywhere: the internet through node001 and
|
||||||
|
# the router, the LAN, and pods that call a public hostname. It reaches
|
||||||
|
# every app it routes to (pods in any namespace; their own policies decide
|
||||||
|
# which port), Home Assistant on 192.168.1.202:8123 (outside the cluster,
|
||||||
|
# home-assistant/), and the internet on 443 for Longhorn's OIDC login
|
||||||
|
# (authentik's token endpoint, at its public address).
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: proxy
|
||||||
|
namespace: envoy-gateway-system
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: envoy
|
||||||
|
app.kubernetes.io/component: proxy
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress:
|
||||||
|
- ports:
|
||||||
|
- port: 10080
|
||||||
|
- port: 10443
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- namespaceSelector: {}
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 192.168.1.202/32
|
||||||
|
ports:
|
||||||
|
- port: 8123
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 0.0.0.0/0
|
||||||
|
except:
|
||||||
|
- 10.0.0.0/8
|
||||||
|
- 100.64.0.0/10
|
||||||
|
- 169.254.0.0/16
|
||||||
|
- 172.16.0.0/12
|
||||||
|
- 192.168.0.0/16
|
||||||
|
ports:
|
||||||
|
- port: 443
|
||||||
|
---
|
||||||
|
# Everything but the proxy (the controller, and the chart's certgen job on
|
||||||
|
# upgrades) reaches the API server (the node, port 16443). The controller
|
||||||
|
# also fetches the OIDC discovery document of Longhorn's SecurityPolicy
|
||||||
|
# from authentik's public address.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: control-plane
|
||||||
|
namespace: envoy-gateway-system
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: app.kubernetes.io/name
|
||||||
|
operator: NotIn
|
||||||
|
values:
|
||||||
|
- envoy
|
||||||
|
policyTypes:
|
||||||
|
- Egress
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 192.168.1.204/32
|
||||||
|
ports:
|
||||||
|
- port: 16443
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 0.0.0.0/0
|
||||||
|
except:
|
||||||
|
- 10.0.0.0/8
|
||||||
|
- 100.64.0.0/10
|
||||||
|
- 169.254.0.0/16
|
||||||
|
- 172.16.0.0/12
|
||||||
|
- 192.168.0.0/16
|
||||||
|
ports:
|
||||||
|
- port: 443
|
||||||
Reference in new issue
Block a user