k8scd/gatus

Gatus

Two Gatus instances check the same endpoints from two places:

Status page	Runs on	Config
https://s1.rubenhensen.nl	Hetzner (node001)	nix-infra-machine/nodes/node001.nix
https://s2.rubenhensen.nl	Home cluster	gatus/config.yaml (this folder)

Each one also checks the other's /health. The s2 DNS record points straight at the home IP instead of going through the Hetzner tunnel, so s2 stays reachable when Hetzner is down.

The endpoint list exists twice, so a new service has to be added in both places. s1 builds its list from tunneledHosts, so new tunneled hosts show up there automatically. s2 needs them added by hand.

Alerting

Each Gatus sends alerts to an ntfy instance on its own node: ntfy1.rubenhensen.nl (Hetzner, node001.nix) and ntfy2.rubenhensen.nl (home, ntfy/ in this repo). Subscribe the phone app to topic alerts on both servers, with the same login on each.

Only the watcher that sees a service from outside alerts on it:

• s1 alerts on everything that runs at home (all tunneled apps, Scouting MBG) and on s2.
• s2 alerts on mail and on s1.

An outage on either node still reaches you, through the other node's ntfy, and you get one notification per outage instead of two. An alert fires after 3 failed checks in a row (about 3 minutes), and you get another notification when the service recovers.

Both ntfy servers deny everything except the provisioned user. The credentials are the same on both:

• Home: Vault kv/ntfy with auth_users, auth_tokens and gatus_token, pulled in by External Secrets.
• Hetzner: the nix-infra secrets ntfy-auth-users, ntfy-auth-tokens and ntfy-gatus-token.

Reading the two pages

• s1 sees the homelab from outside, so trust it for homelab services. s2 sees Hetzner from outside, so trust it for mail and anything else on Hetzner.
• Every *.rubenhensen.nl app is tunneled Hetzner → home, so a Hetzner outage shows all of them as down on both pages. That is accurate: nobody can reach them then.
• If s1 and s2 disagree about an endpoint, the problem is the network path between one watcher and that service, not the service itself.
• Neither page has history for the time its own node was down.

Later: one URL with self-hosted DNS failover

This is the plan for replacing s1/s2 with a single status.rubenhensen.nl that always points at a watcher that's up, with no third party involved.

Hand status.rubenhensen.nl over to two nameservers we run ourselves, one per node, both PowerDNS Authoritative. PowerDNS LUA records health-check addresses and only return the healthy ones:

@  30  IN  LUA  A  "ifurlup('https://status.rubenhensen.nl/health', {{'46.224.26.65'}, {'62.41.86.27'}})"

The double braces make two groups. PowerDNS returns the first group that has a healthy address, so Hetzner is primary and home is the fallback. Each nameserver runs the check from its own location.

Situation	Hetzner NS	Home NS	Visitors end up on
Both up	answers Hetzner	answers Hetzner	Hetzner
Hetzner down	dead	Hetzner check fails, answers home	home
Home down / home internet down	answers Hetzner	dead	Hetzner
Hetzner up, its Gatus broken	own check fails, answers home	answers home	home

A node that goes down takes its nameserver down with it, so DNS resolvers switch to the other nameserver. The two nodes never have to agree with each other: the answer depends on what the internet can reach. The 30s TTL caps how long clients keep a stale answer.

Steps:

1. Parent zone (dns/domains/rubenhensen.nl.yaml, synced to TransIP): add status NS ns-hetzner.rubenhensen.nl. and status NS ns-home.rubenhensen.nl., plus A records for ns-hetzner (46.224.26.65) and ns-home (62.41.86.27). First check that TransIP accepts NS records for a subdomain.
2. Hetzner: run PowerDNS from node001.nix with enable-lua-records=yes and the bind backend. Use the same zone file as home. Open 53/udp and 53/tcp.
3. Home: run PowerDNS in the cluster with the same zone file, on 53/udp and 53/tcp of the home IP (router port forward, and check how the cluster exposes UDP).
4. Both Gatus instances also serve status.rubenhensen.nl. On Hetzner, add the name to the SNI map, the same way as s1.
5. TLS: Hetzner gets its cert through HTTP-01 as usual. On Hetzner's :80 vhost for status, serve local ACME challenges and proxy the rest to the home cluster (try_files $uri @home). That way cert-manager's HTTP-01 also works while DNS points at Hetzner. When Hetzner is down, DNS points home and cert-manager reaches the home cluster directly.
6. Requirements: a static home IP (homeIP in node001.nix already assumes one), and port 53 open on both nodes.

After that, s1/s2 can be removed, or kept as direct links to each instance.