From f2e4c5aff6fbf906a2e9e4570a42e0f822d629cf Mon Sep 17 00:00:00 2001 From: Ruben Hensen Date: Sun, 25 Oct 2026 03:49:43 +0100 Subject: [PATCH] docs: record the backup check --- docs/roadmap.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index 3b6835b..bbc278d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -177,6 +177,15 @@ fails). The dqlite stalls were tracked down by hand from the API server's metrics, and Velero's scheduled backups failed every night from August until 2026-10-21 without anyone noticing. +- [x] A stopgap for the worst blind spots (2026-10-25, + `gatus/backup-check.yaml`): an hourly CronJob reads the API and pushes + four external endpoints to s2 (group "cluster"): the newest daily and + weekly Velero backup Completed and recent, every attached Longhorn + volume healthy with a backup younger than 26h, and no Vault pod sealed. + Gatus alerts through ntfy after two failed reports, or when no report + arrives for 3 hours. The token comes from an ESO Password generator. + Tested with made-up data for each failure (a Failed or old backup, a + degraded or unbacked-up volume, a sealed pod) - [ ] After the second SSD (a time series database writes all the time and would add to today's stalls): metrics-server, then Prometheus or VictoriaMetrics with Grafana, alerting to ntfy. Loki later