Use restore-test.sh. It restores a few namespaces from a real backup into a throwaway k3s in Docker, from a read-only copy of the bucket, and checks that every volume comes back with the bytes it was backed up with:

backup-testing/restore-test.sh                          # newest daily backup; ntfy, freshrss, actual-budget
backup-testing/restore-test.sh <backup> <namespace>...  # something else

How it works and why it copies instead of pointing Velero at the NAS is at the top of the script. The rest of this README describes the older k3d scripts, which predate Envoy Gateway and still expect *.hensen.io and sol-auth.

This guide explains how to test your Velero backups using a temporary K3d cluster.

Features

Creates isolated K3d test cluster
Installs nginx-ingress controller for ingress support
Installs cert-manager with self-signed certificates (no real DNS/certs needed)
Automatically rewrites ingress hosts from *.hensen.io to *.local
Bypasses authentication proxies (sol-auth) by rewriting ingress backends
Generates /etc/hosts entries for local DNS resolution
Restores backups with storage class mapping (longhorn -> local-path)

Prerequisites

Ensure you have the following tools installed on your local machine:

Docker
kubectl
Helm
K3d
Velero CLI
jq (for JSON parsing)

Setup

First, clone this repository to your local machine:

git clone https://github.com/yourusername/backup-testing.git
cd backup-testing

Make the scripts executable:

chmod +x test-velero-backup.sh validate-restore.sh

Running a Backup Test

Execute the test script with appropriate parameters:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name <BACKUP_NAME> \
  --original-namespace <NAMESPACE> \
  --test-namespace <NAMESPACE> \
  --generate-hosts

This will:

Create a temporary K3d cluster
Install nginx-ingress controller
Install cert-manager with self-signed ClusterIssuer
Install Velero configured to access your MinIO backup location
Restore the specified backup (excluding sol-auth and certificates)
Remove any restored sol-auth resources
Rewrite ingress backends from sol-auth to direct backend services
Rewrite ingress hosts from *.hensen.io to *.local
Generate /etc/hosts entries (with --generate-hosts flag)

Command Line Options

Option	Description	Default
`--backup-name NAME`	Name of the Velero backup to restore	(lists backups if empty)
`--s3-access-key KEY`	S3/MinIO access key	(required)
`--s3-secret-key KEY`	S3/MinIO secret key	(required)
`--s3-bucket BUCKET`	S3 bucket containing backups	`velero-backups`
`--s3-region REGION`	S3 region	`minio`
`--s3-url URL`	S3 endpoint URL	`http://192.168.1.99:9000`
`--original-namespace NS`	Namespace in the original backup	(required)
`--test-namespace NS`	Namespace for restoration	`default`
`--original-domain DOMAIN`	Domain to replace in ingresses	`hensen.io`
`--local-domain DOMAIN`	Local domain suffix	`local`
`--skip-ingress-setup`	Skip nginx-ingress and cert-manager	`false`
`--generate-hosts`	Generate /etc/hosts entries	`false`
`--debug`	Enable debug output	`false`

Local DNS Resolution

After the restore completes, the script can generate /etc/hosts entries. Add these to your /etc/hosts file to access services locally:

# Example output with --generate-hosts:
127.0.0.1    foto.local
127.0.0.1    immich.local

Then access your restored services at https://foto.local or https://immich.local.

Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.

Testing Specific Applications

Immich

For testing Immich backups:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name daily-backup-YYYYMMDD \
  --original-namespace immich \
  --test-namespace immich \
  --generate-hosts

Then add the generated hosts entries and access via https://immich.local

Alternatively, use port-forwarding:

kubectl port-forward service/helm-immich-server -n immich 8080:2283

Reachable on localhost:8080

Custom Domain Mapping

By default, the script rewrites *.hensen.io to *.local. You can customize this:

./test-velero-backup.sh \
  --s3-access-key YOUR_ACCESS_KEY \
  --s3-secret-key YOUR_SECRET_KEY \
  --backup-name your-backup \
  --original-namespace myapp \
  --original-domain example.com \
  --local-domain test.local

This will rewrite app.example.com to app.test.local.

Authentication Proxy Bypass

The production environment uses sol-auth (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:

Sol-auth resources are removed after restore (deployment, service, configmap)
Ingress backends are rewritten from sol-auth-svc:8002 to the actual backend service (e.g., helm-immich-server:2283)

This allows you to access the application directly without needing to authenticate through the Scouting login portal.

Production flow:

Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283

Test flow:

Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)

Storage Class Mapping

The script automatically applies change-storage-class.yaml which maps:

longhorn -> local-path

This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.

Troubleshooting

Check Velero logs

kubectl logs -n velero deploy/velero

Examine restore details

velero restore describe <RESTORE_NAME>
velero restore logs <RESTORE_NAME>

Check cluster resources

kubectl get pods -A
kubectl get pvc -A
kubectl get ingress -A

Check ingress controller

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller

Check cert-manager

kubectl get certificates -A
kubectl get clusterissuer

Debug mode

Run with --debug for verbose output:

./test-velero-backup.sh --debug ...

Architecture

+------------------+     +------------------+     +------------------+
|   K3d Cluster    |     |     MinIO        |     |   Your Browser   |
|                  |     |   (NAS/S3)       |     |                  |
|  +-----------+   |     |                  |     |                  |
|  |  Velero   |<--+-----+ velero-backups   |     |                  |
|  +-----------+   |     |     bucket       |     |                  |
|        |         |     +------------------+     |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  | Restored  |   |                              |                  |
|  |   App     |   |                              |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |     *.local domains          |                  |
|  |  Ingress  |<---------------------------------+  https://app.local
|  |  nginx    |   |     (via /etc/hosts)         |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  |cert-manager|  |     Self-signed certs        |                  |
|  |(selfsigned)|  |                              |                  |
|  +-----------+   |                              |                  |
+------------------+                              +------------------+