From 14e61ee5eed78cbceeb9d111743dfbb6c72bf7c5 Mon Sep 17 00:00:00 2001
From: Ruben Hensen <ruben.hensen@protonmail.com>
Date: Sat, 24 Oct 2026 12:19:37 +0200
Subject: [PATCH] gateway: retry requests on dropped backend connections

When a backend closes a kept-alive connection just as Envoy sends a
request over it, Envoy returns a 503 (response flag UC) and does not
retry by default. ingress-nginx retried these, so the race never showed.

The Apache in front of immich (sol-auth) recycles its processes after 30
connections, which made this happen during page loads: three chunks got
a 503 and immich failed with "error loading dynamically imported module".
authentik and ntfy saw it too, less often.

Retry up to twice on connection failures and resets, for every route on
the Gateway.
---
 gateway/retry-policy.yaml | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 gateway/retry-policy.yaml

diff --git a/gateway/retry-policy.yaml b/gateway/retry-policy.yaml
new file mode 100644
index 0000000..0552e32
--- /dev/null
+++ b/gateway/retry-policy.yaml
@@ -0,0 +1,25 @@
+# Retry a request when the backend drops a kept-alive connection just as
+# Envoy sends a request over it. Envoy answers that with a 503 (flag UC) and
+# does not retry by default; ingress-nginx did. It happens most behind the
+# Apache of sol-auth (immich), which recycles its processes after 30
+# connections, and broke immich's page loads.
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: BackendTrafficPolicy
+metadata:
+  name: retry-on-reset
+  namespace: envoy-gateway-system
+spec:
+  targetRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: homelab
+  retry:
+    numRetries: 2
+    retryOn:
+      triggers:
+      - connect-failure
+      - reset
+    perRetry:
+      backOff:
+        baseInterval: 25ms
+        maxInterval: 250ms