k8scd/backup-testing/README.md

Velero Backup Testing Guide

Use restore-test.sh. It restores a few namespaces from a real backup into a throwaway k3s in Docker, from a read-only copy of the bucket, and checks that every volume comes back with the bytes it was backed up with:

backup-testing/restore-test.sh                          # newest daily backup; ntfy, freshrss, actual-budget
backup-testing/restore-test.sh <backup> <namespace>...  # something else

How it works and why it copies instead of pointing Velero at the NAS is at the top of the script. The rest of this README describes the older k3d scripts, which predate Envoy Gateway and still expect *.hensen.io and sol-auth.

This guide explains how to test your Velero backups using a temporary K3d cluster.

Features

• Creates isolated K3d test cluster
• Installs nginx-ingress controller for ingress support
• Installs cert-manager with self-signed certificates (no real DNS/certs needed)
• Automatically rewrites ingress hosts from *.hensen.io to *.local
• Bypasses authentication proxies (sol-auth) by rewriting ingress backends
• Generates /etc/hosts entries for local DNS resolution
• Restores backups with storage class mapping (longhorn -> local-path)

Prerequisites

Ensure you have the following tools installed on your local machine:

• Docker
• kubectl
• Helm
• K3d
• Velero CLI
• jq (for JSON parsing)

Setup

1. First, clone this repository to your local machine:

git clone https://github.com/yourusername/backup-testing.git
cd backup-testing

2. Make the scripts executable:

chmod +x test-velero-backup.sh validate-restore.sh

Running a Backup Test

Execute the test script with appropriate parameters:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name <BACKUP_NAME> \
  --original-namespace <NAMESPACE> \
  --test-namespace <NAMESPACE> \
  --generate-hosts

This will:

1. Create a temporary K3d cluster
2. Install nginx-ingress controller
3. Install cert-manager with self-signed ClusterIssuer
4. Install Velero configured to access your MinIO backup location
5. Restore the specified backup (excluding sol-auth and certificates)
6. Remove any restored sol-auth resources
7. Rewrite ingress backends from sol-auth to direct backend services
8. Rewrite ingress hosts from *.hensen.io to *.local
9. Generate /etc/hosts entries (with --generate-hosts flag)

Command Line Options

Option	Description	Default
--backup-name NAME	Name of the Velero backup to restore	(lists backups if empty)
--s3-access-key KEY	S3/MinIO access key	(required)
--s3-secret-key KEY	S3/MinIO secret key	(required)
--s3-bucket BUCKET	S3 bucket containing backups	velero-backups
--s3-region REGION	S3 region	minio
--s3-url URL	S3 endpoint URL	http://192.168.1.99:9000
--original-namespace NS	Namespace in the original backup	(required)
--test-namespace NS	Namespace for restoration	default
--original-domain DOMAIN	Domain to replace in ingresses	hensen.io
--local-domain DOMAIN	Local domain suffix	local
--skip-ingress-setup	Skip nginx-ingress and cert-manager	false
--generate-hosts	Generate /etc/hosts entries	false
--debug	Enable debug output	false

Local DNS Resolution

After the restore completes, the script can generate /etc/hosts entries. Add these to your /etc/hosts file to access services locally:

# Example output with --generate-hosts:
127.0.0.1    foto.local
127.0.0.1    immich.local

Then access your restored services at https://foto.local or https://immich.local.

Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.

Testing Specific Applications

Immich

For testing Immich backups:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name daily-backup-YYYYMMDD \
  --original-namespace immich \
  --test-namespace immich \
  --generate-hosts

Then add the generated hosts entries and access via https://immich.local

Alternatively, use port-forwarding:

kubectl port-forward service/helm-immich-server -n immich 8080:2283

Reachable on localhost:8080

Custom Domain Mapping

By default, the script rewrites *.hensen.io to *.local. You can customize this:

./test-velero-backup.sh \
  --s3-access-key YOUR_ACCESS_KEY \
  --s3-secret-key YOUR_SECRET_KEY \
  --backup-name your-backup \
  --original-namespace myapp \
  --original-domain example.com \
  --local-domain test.local

This will rewrite app.example.com to app.test.local.

Authentication Proxy Bypass

The production environment uses sol-auth (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:

1. Sol-auth resources are removed after restore (deployment, service, configmap)
2. Ingress backends are rewritten from sol-auth-svc:8002 to the actual backend service (e.g., helm-immich-server:2283)

This allows you to access the application directly without needing to authenticate through the Scouting login portal.

Production flow:

Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283

Test flow:

Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)

Storage Class Mapping

The script automatically applies change-storage-class.yaml which maps:

• longhorn -> local-path

This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.

Troubleshooting

Check Velero logs

kubectl logs -n velero deploy/velero

Examine restore details

velero restore describe <RESTORE_NAME>
velero restore logs <RESTORE_NAME>

Check cluster resources

kubectl get pods -A
kubectl get pvc -A
kubectl get ingress -A

Check ingress controller

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller

Check cert-manager

kubectl get certificates -A
kubectl get clusterissuer

Debug mode

Run with --debug for verbose output:

./test-velero-backup.sh --debug ...

Architecture

+------------------+     +------------------+     +------------------+
|   K3d Cluster    |     |     MinIO        |     |   Your Browser   |
|                  |     |   (NAS/S3)       |     |                  |
|  +-----------+   |     |                  |     |                  |
|  |  Velero   |<--+-----+ velero-backups   |     |                  |
|  +-----------+   |     |     bucket       |     |                  |
|        |         |     +------------------+     |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  | Restored  |   |                              |                  |
|  |   App     |   |                              |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |     *.local domains          |                  |
|  |  Ingress  |<---------------------------------+  https://app.local
|  |  nginx    |   |     (via /etc/hosts)         |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  |cert-manager|  |     Self-signed certs        |                  |
|  |(selfsigned)|  |                              |                  |
|  +-----------+   |                              |                  |
+------------------+                              +------------------+