When a backend closes a kept-alive connection just as Envoy sends a
request over it, Envoy returns a 503 (response flag UC) and does not
retry by default. ingress-nginx retried these, so the race never showed.
The Apache in front of immich (sol-auth) recycles its processes after 30
connections, which made this happen during page loads: three chunks got
a 503 and immich failed with "error loading dynamically imported module".
authentik and ntfy saw it too, less often.
Retry up to twice on connection failures and resets, for every route on
the Gateway.
Longhorn has no users of its own; nginx guarded it with basic auth using apr1 hashes, which Envoy Gateway does not support. An Envoy Gateway SecurityPolicy now sends visitors through an authentik OIDC application that only admins may use. The client secret lives in Vault at kv/longhorn-oidc and reaches both authentik and Envoy through External Secrets.
Envoy moves from the test ports to 80 and 443, and Let's Encrypt HTTP-01 challenges go through an HTTPRoute on the Gateway. Push only after sudo microk8s disable ingress, since both want the same host ports.
With server.insecure, TLS ends at nginx or Envoy and argocd-server serves plain HTTP, which is how Argo CD documents the Gateway API setup. The nginx Ingress now talks HTTP to the http port; the ssl-passthrough annotation did nothing without the controller flag. argocd-folder diffs server-side for the new HTTPRoute.
Each app gets an HTTPS listener on the Gateway with its existing certificate, an HTTPRoute, and a ReferenceGrant for that certificate. The routes drop Envoy's 15s request timeout, as nginx has no overall limit either. Argo CD and Longhorn follow separately; the Ingresses stay until the switch-over.
First app on the Gateway API: an HTTPS listener for s2.rubenhensen.nl using the existing certificate in the gatus namespace, an HTTPRoute, and a ReferenceGrant for the certificate. Plain HTTP on the Gateway now redirects to HTTPS. The Ingress stays until the switch-over.
Envoy runs as a DaemonSet on host ports 10080 and 10443 for now, so it can be tested next to ingress-nginx, which keeps 80 and 443. The Gateway has a plain HTTP listener and no routes yet.
Ruben Hensen