mirror of
https://github.com/rubenhensen/k8scd.git
synced 2026-11-15 14:29:09 +01:00
Ruben Hensen
e23779044f
Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at the home IP like s2. s2 now alerts on mail and on s1 through it. The home services are left to s1, which sees them from outside. ntfy denies everything except the provisioned user. The user, its password hash and the Gatus token come from Vault kv/ntfy.
106 lines
4.8 KiB
Markdown
106 lines
4.8 KiB
Markdown
# Gatus
|
|
|
|
Two Gatus instances check the same endpoints from two places:
|
|
|
|
| Status page | Runs on | Config |
|
|
|---|---|---|
|
|
| https://s1.rubenhensen.nl | Hetzner (node001) | `nix-infra-machine/nodes/node001.nix` |
|
|
| https://s2.rubenhensen.nl | Home cluster | `gatus/config.yaml` (this folder) |
|
|
|
|
Each one also checks the other's `/health`. The s2 DNS record points straight
|
|
at the home IP instead of going through the Hetzner tunnel, so s2 stays
|
|
reachable when Hetzner is down.
|
|
|
|
The endpoint list exists twice, so a new service has to be added in both
|
|
places. s1 builds its list from `tunneledHosts`, so new tunneled hosts show up
|
|
there automatically. s2 needs them added by hand.
|
|
|
|
## Alerting
|
|
|
|
Each Gatus sends alerts to an ntfy instance on its own node:
|
|
`ntfy1.rubenhensen.nl` (Hetzner, node001.nix) and `ntfy2.rubenhensen.nl`
|
|
(home, `ntfy/` in this repo). Subscribe the phone app to topic `alerts` on
|
|
both servers, with the same login on each.
|
|
|
|
Only the watcher that sees a service from outside alerts on it:
|
|
|
|
- s1 alerts on everything that runs at home (all tunneled apps, Scouting MBG)
|
|
and on s2.
|
|
- s2 alerts on mail and on s1.
|
|
|
|
An outage on either node still reaches you, through the other node's ntfy,
|
|
and you get one notification per outage instead of two. An alert fires after
|
|
3 failed checks in a row (about 3 minutes), and you get another notification
|
|
when the service recovers.
|
|
|
|
Both ntfy servers deny everything except the provisioned user. The
|
|
credentials are the same on both:
|
|
|
|
- Home: Vault `kv/ntfy` with `auth_users`, `auth_tokens` and `gatus_token`,
|
|
pulled in by External Secrets.
|
|
- Hetzner: the nix-infra secrets `ntfy-auth-users`, `ntfy-auth-tokens` and
|
|
`ntfy-gatus-token`.
|
|
|
|
## Reading the two pages
|
|
|
|
- s1 sees the homelab from outside, so trust it for homelab services. s2 sees
|
|
Hetzner from outside, so trust it for mail and anything else on Hetzner.
|
|
- Every `*.rubenhensen.nl` app is tunneled Hetzner → home, so a Hetzner
|
|
outage shows all of them as down on both pages. That is accurate: nobody
|
|
can reach them then.
|
|
- If s1 and s2 disagree about an endpoint, the problem is the network path
|
|
between one watcher and that service, not the service itself.
|
|
- Neither page has history for the time its own node was down.
|
|
|
|
## Later: one URL with self-hosted DNS failover
|
|
|
|
This is the plan for replacing s1/s2 with a single `status.rubenhensen.nl`
|
|
that always points at a watcher that's up, with no third party involved.
|
|
|
|
Hand `status.rubenhensen.nl` over to two nameservers we run ourselves, one per
|
|
node, both PowerDNS Authoritative. PowerDNS LUA records health-check
|
|
addresses and only return the healthy ones:
|
|
|
|
```
|
|
@ 30 IN LUA A "ifurlup('https://status.rubenhensen.nl/health', {{'46.224.26.65'}, {'62.41.86.27'}})"
|
|
```
|
|
|
|
The double braces make two groups. PowerDNS returns the first group that has
|
|
a healthy address, so Hetzner is primary and home is the fallback. Each
|
|
nameserver runs the check from its own location.
|
|
|
|
| Situation | Hetzner NS | Home NS | Visitors end up on |
|
|
|---|---|---|---|
|
|
| Both up | answers Hetzner | answers Hetzner | Hetzner |
|
|
| Hetzner down | dead | Hetzner check fails, answers home | home |
|
|
| Home down / home internet down | answers Hetzner | dead | Hetzner |
|
|
| Hetzner up, its Gatus broken | own check fails, answers home | answers home | home |
|
|
|
|
A node that goes down takes its nameserver down with it, so DNS resolvers
|
|
switch to the other nameserver. The two nodes never have to agree with each
|
|
other: the answer depends on what the internet can reach. The 30s TTL caps
|
|
how long clients keep a stale answer.
|
|
|
|
Steps:
|
|
|
|
1. Parent zone (`dns/domains/rubenhensen.nl.yaml`, synced to TransIP): add
|
|
`status NS ns-hetzner.rubenhensen.nl.` and `status NS ns-home.rubenhensen.nl.`,
|
|
plus A records for `ns-hetzner` (46.224.26.65) and `ns-home` (62.41.86.27).
|
|
First check that TransIP accepts NS records for a subdomain.
|
|
2. Hetzner: run PowerDNS from node001.nix with `enable-lua-records=yes` and
|
|
the bind backend. Use the same zone file as home. Open 53/udp and 53/tcp.
|
|
3. Home: run PowerDNS in the cluster with the same zone file, on 53/udp and
|
|
53/tcp of the home IP (router port forward, and check how the cluster
|
|
exposes UDP).
|
|
4. Both Gatus instances also serve `status.rubenhensen.nl`. On Hetzner, add the
|
|
name to the SNI map, the same way as s1.
|
|
5. TLS: Hetzner gets its cert through HTTP-01 as usual. On Hetzner's :80 vhost
|
|
for `status`, serve local ACME challenges and proxy the rest to the home
|
|
cluster (`try_files $uri @home`). That way cert-manager's HTTP-01 also
|
|
works while DNS points at Hetzner. When Hetzner is down, DNS points home
|
|
and cert-manager reaches the home cluster directly.
|
|
6. Requirements: a static home IP (`homeIP` in node001.nix already assumes
|
|
one), and port 53 open on both nodes.
|
|
|
|
After that, s1/s2 can be removed, or kept as direct links to each instance.
|