k8scd/backup-testing

Velero Backup Testing Guide

This guide explains how to test your Velero backups using a temporary K3d cluster.

Features

• Creates isolated K3d test cluster
• Installs nginx-ingress controller for ingress support
• Installs cert-manager with self-signed certificates (no real DNS/certs needed)
• Automatically rewrites ingress hosts from *.hensen.io to *.local
• Bypasses authentication proxies (sol-auth) by rewriting ingress backends
• Generates /etc/hosts entries for local DNS resolution
• Restores backups with storage class mapping (longhorn -> local-path)

Prerequisites

Ensure you have the following tools installed on your local machine:

• Docker
• kubectl
• Helm
• K3d
• Velero CLI
• jq (for JSON parsing)

Setup

1. First, clone this repository to your local machine:

git clone https://github.com/yourusername/backup-testing.git
cd backup-testing

2. Make the scripts executable:

chmod +x test-velero-backup.sh validate-restore.sh

Running a Backup Test

Execute the test script with appropriate parameters:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name <BACKUP_NAME> \
  --original-namespace <NAMESPACE> \
  --test-namespace <NAMESPACE> \
  --generate-hosts

This will:

1. Create a temporary K3d cluster
2. Install nginx-ingress controller
3. Install cert-manager with self-signed ClusterIssuer
4. Install Velero configured to access your MinIO backup location
5. Restore the specified backup (excluding sol-auth and certificates)
6. Remove any restored sol-auth resources
7. Rewrite ingress backends from sol-auth to direct backend services
8. Rewrite ingress hosts from *.hensen.io to *.local
9. Generate /etc/hosts entries (with --generate-hosts flag)

Command Line Options

Option	Description	Default
--backup-name NAME	Name of the Velero backup to restore	(lists backups if empty)
--s3-access-key KEY	S3/MinIO access key	(required)
--s3-secret-key KEY	S3/MinIO secret key	(required)
--s3-bucket BUCKET	S3 bucket containing backups	velero-backups
--s3-region REGION	S3 region	minio
--s3-url URL	S3 endpoint URL	http://192.168.1.99:9000
--original-namespace NS	Namespace in the original backup	(required)
--test-namespace NS	Namespace for restoration	default
--original-domain DOMAIN	Domain to replace in ingresses	hensen.io
--local-domain DOMAIN	Local domain suffix	local
--skip-ingress-setup	Skip nginx-ingress and cert-manager	false
--generate-hosts	Generate /etc/hosts entries	false
--debug	Enable debug output	false

Local DNS Resolution

After the restore completes, the script can generate /etc/hosts entries. Add these to your /etc/hosts file to access services locally:

# Example output with --generate-hosts:
127.0.0.1    foto.local
127.0.0.1    immich.local

Then access your restored services at https://foto.local or https://immich.local.

Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.

Testing Specific Applications

Immich

For testing Immich backups:

./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name daily-backup-YYYYMMDD \
  --original-namespace immich \
  --test-namespace immich \
  --generate-hosts

Then add the generated hosts entries and access via https://immich.local

Alternatively, use port-forwarding:

kubectl port-forward service/helm-immich-server -n immich 8080:2283

Reachable on localhost:8080

Custom Domain Mapping

By default, the script rewrites *.hensen.io to *.local. You can customize this:

./test-velero-backup.sh \
  --s3-access-key YOUR_ACCESS_KEY \
  --s3-secret-key YOUR_SECRET_KEY \
  --backup-name your-backup \
  --original-namespace myapp \
  --original-domain example.com \
  --local-domain test.local

This will rewrite app.example.com to app.test.local.

Authentication Proxy Bypass

The production environment uses sol-auth (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:

1. Sol-auth resources are removed after restore (deployment, service, configmap)
2. Ingress backends are rewritten from sol-auth-svc:8002 to the actual backend service (e.g., helm-immich-server:2283)

This allows you to access the application directly without needing to authenticate through the Scouting login portal.

Production flow:

Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283

Test flow:

Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)

Storage Class Mapping

The script automatically applies change-storage-class.yaml which maps:

• longhorn -> local-path

This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.

Troubleshooting

Check Velero logs

kubectl logs -n velero deploy/velero

Examine restore details

velero restore describe <RESTORE_NAME>
velero restore logs <RESTORE_NAME>

Check cluster resources

kubectl get pods -A
kubectl get pvc -A
kubectl get ingress -A

Check ingress controller

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller

Check cert-manager

kubectl get certificates -A
kubectl get clusterissuer

Debug mode

Run with --debug for verbose output:

./test-velero-backup.sh --debug ...

Architecture

+------------------+     +------------------+     +------------------+
|   K3d Cluster    |     |     MinIO        |     |   Your Browser   |
|                  |     |   (NAS/S3)       |     |                  |
|  +-----------+   |     |                  |     |                  |
|  |  Velero   |<--+-----+ velero-backups   |     |                  |
|  +-----------+   |     |     bucket       |     |                  |
|        |         |     +------------------+     |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  | Restored  |   |                              |                  |
|  |   App     |   |                              |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |     *.local domains          |                  |
|  |  Ingress  |<---------------------------------+  https://app.local
|  |  nginx    |   |     (via /etc/hosts)         |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  |cert-manager|  |     Self-signed certs        |                  |
|  |(selfsigned)|  |                              |                  |
|  +-----------+   |                              |                  |
+------------------+                              +------------------+