gatus: alert through self-hosted ntfy

Add ntfy at ntfy2.rubenhensen.nl on the home cluster, pointed straight at
the home IP like s2. s2 now alerts on mail and on s1 through it. The
home services are left to s1, which sees them from outside.

ntfy denies everything except the provisioned user. The user, its
password hash and the Gatus token come from Vault kv/ntfy.
This commit is contained in:
Ruben Hensen committed 2026-10-20 15:34:42 +02:00
1 parent 53a079dc22
commit e23779044f
14 files changed
+246 -1

No files matched your search

+26
View File
@@ -15,6 +15,32 @@ The endpoint list exists twice, so a new service has to be added in both
places. s1 builds its list from `tunneledHosts`, so new tunneled hosts show up
there automatically. s2 needs them added by hand.
## Alerting
Each Gatus sends alerts to an ntfy instance on its own node:
`ntfy1.rubenhensen.nl` (Hetzner, node001.nix) and `ntfy2.rubenhensen.nl`
(home, `ntfy/` in this repo). Subscribe the phone app to topic `alerts` on
both servers, with the same login on each.
Only the watcher that sees a service from outside alerts on it:
- s1 alerts on everything that runs at home (all tunneled apps, Scouting MBG)
and on s2.
- s2 alerts on mail and on s1.
An outage on either node still reaches you, through the other node's ntfy,
and you get one notification per outage instead of two. An alert fires after
3 failed checks in a row (about 3 minutes), and you get another notification
when the service recovers.
Both ntfy servers deny everything except the provisioned user. The
credentials are the same on both:
- Home: Vault `kv/ntfy` with `auth_users`, `auth_tokens` and `gatus_token`,
pulled in by External Secrets.
- Hetzner: the nix-infra secrets `ntfy-auth-users`, `ntfy-auth-tokens` and
`ntfy-gatus-token`.
## Reading the two pages
- s1 sees the homelab from outside, so trust it for homelab services. s2 sees