# Velero Backup Testing Guide

**Use `restore-test.sh`.** It restores a few namespaces from a real backup
into a throwaway k3s in Docker, from a read-only copy of the bucket, and
checks that every volume comes back with the bytes it was backed up with:

```bash
backup-testing/restore-test.sh                          # newest daily backup; ntfy, freshrss, actual-budget
backup-testing/restore-test.sh <backup> <namespace>...  # something else
```

How it works and why it copies instead of pointing Velero at the NAS is at
the top of the script. The rest of this README describes the older k3d
scripts, which predate Envoy Gateway and still expect `*.hensen.io` and
sol-auth.

This guide explains how to test your Velero backups using a temporary K3d cluster.

## Features

- Creates isolated K3d test cluster
- Installs nginx-ingress controller for ingress support
- Installs cert-manager with self-signed certificates (no real DNS/certs needed)
- Automatically rewrites ingress hosts from `*.hensen.io` to `*.local`
- Bypasses authentication proxies (sol-auth) by rewriting ingress backends
- Generates `/etc/hosts` entries for local DNS resolution
- Restores backups with storage class mapping (longhorn -> local-path)

## Prerequisites

Ensure you have the following tools installed on your local machine:

- Docker
- kubectl
- Helm
- K3d
- Velero CLI
- jq (for JSON parsing)

## Setup

1. First, clone this repository to your local machine:

```bash
git clone https://github.com/yourusername/backup-testing.git
cd backup-testing
```

2. Make the scripts executable:

```bash
chmod +x test-velero-backup.sh validate-restore.sh
```

## Running a Backup Test

Execute the test script with appropriate parameters:

```bash
./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name <BACKUP_NAME> \
  --original-namespace <NAMESPACE> \
  --test-namespace <NAMESPACE> \
  --generate-hosts
```

This will:
1. Create a temporary K3d cluster
2. Install nginx-ingress controller
3. Install cert-manager with self-signed ClusterIssuer
4. Install Velero configured to access your MinIO backup location
5. Restore the specified backup (excluding sol-auth and certificates)
6. Remove any restored sol-auth resources
7. Rewrite ingress backends from sol-auth to direct backend services
8. Rewrite ingress hosts from `*.hensen.io` to `*.local`
9. Generate `/etc/hosts` entries (with `--generate-hosts` flag)

## Command Line Options

| Option | Description | Default |
|--------|-------------|---------|
| `--backup-name NAME` | Name of the Velero backup to restore | (lists backups if empty) |
| `--s3-access-key KEY` | S3/MinIO access key | (required) |
| `--s3-secret-key KEY` | S3/MinIO secret key | (required) |
| `--s3-bucket BUCKET` | S3 bucket containing backups | `velero-backups` |
| `--s3-region REGION` | S3 region | `minio` |
| `--s3-url URL` | S3 endpoint URL | `http://192.168.1.99:9000` |
| `--original-namespace NS` | Namespace in the original backup | (required) |
| `--test-namespace NS` | Namespace for restoration | `default` |
| `--original-domain DOMAIN` | Domain to replace in ingresses | `hensen.io` |
| `--local-domain DOMAIN` | Local domain suffix | `local` |
| `--skip-ingress-setup` | Skip nginx-ingress and cert-manager | `false` |
| `--generate-hosts` | Generate /etc/hosts entries | `false` |
| `--debug` | Enable debug output | `false` |

## Local DNS Resolution

After the restore completes, the script can generate `/etc/hosts` entries. Add these to your `/etc/hosts` file to access services locally:

```bash
# Example output with --generate-hosts:
127.0.0.1    foto.local
127.0.0.1    immich.local
```

Then access your restored services at `https://foto.local` or `https://immich.local`.

Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.

## Testing Specific Applications

### Immich

For testing Immich backups:

```bash
./test-velero-backup.sh \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY> \
  --backup-name daily-backup-YYYYMMDD \
  --original-namespace immich \
  --test-namespace immich \
  --generate-hosts
```

Then add the generated hosts entries and access via `https://immich.local`

Alternatively, use port-forwarding:
```bash
kubectl port-forward service/helm-immich-server -n immich 8080:2283
```
Reachable on `localhost:8080`



## Custom Domain Mapping

By default, the script rewrites `*.hensen.io` to `*.local`. You can customize this:

```bash
./test-velero-backup.sh \
  --s3-access-key YOUR_ACCESS_KEY \
  --s3-secret-key YOUR_SECRET_KEY \
  --backup-name your-backup \
  --original-namespace myapp \
  --original-domain example.com \
  --local-domain test.local
```

This will rewrite `app.example.com` to `app.test.local`.

## Authentication Proxy Bypass

The production environment uses `sol-auth` (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:

1. **Sol-auth resources are removed** after restore (deployment, service, configmap)
2. **Ingress backends are rewritten** from `sol-auth-svc:8002` to the actual backend service (e.g., `helm-immich-server:2283`)

This allows you to access the application directly without needing to authenticate through the Scouting login portal.

**Production flow:**
```
Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283
```

**Test flow:**
```
Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)
```

## Storage Class Mapping

The script automatically applies `change-storage-class.yaml` which maps:
- `longhorn` -> `local-path`

This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.

## Troubleshooting

### Check Velero logs
```bash
kubectl logs -n velero deploy/velero
```

### Examine restore details
```bash
velero restore describe <RESTORE_NAME>
velero restore logs <RESTORE_NAME>
```

### Check cluster resources
```bash
kubectl get pods -A
kubectl get pvc -A
kubectl get ingress -A
```

### Check ingress controller
```bash
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
```

### Check cert-manager
```bash
kubectl get certificates -A
kubectl get clusterissuer
```

### Debug mode
Run with `--debug` for verbose output:
```bash
./test-velero-backup.sh --debug ...
```

## Architecture

```
+------------------+     +------------------+     +------------------+
|   K3d Cluster    |     |     MinIO        |     |   Your Browser   |
|                  |     |   (NAS/S3)       |     |                  |
|  +-----------+   |     |                  |     |                  |
|  |  Velero   |<--+-----+ velero-backups   |     |                  |
|  +-----------+   |     |     bucket       |     |                  |
|        |         |     +------------------+     |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  | Restored  |   |                              |                  |
|  |   App     |   |                              |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |     *.local domains          |                  |
|  |  Ingress  |<---------------------------------+  https://app.local
|  |  nginx    |   |     (via /etc/hosts)         |                  |
|  +-----------+   |                              |                  |
|        |         |                              |                  |
|        v         |                              |                  |
|  +-----------+   |                              |                  |
|  |cert-manager|  |     Self-signed certs        |                  |
|  |(selfsigned)|  |                              |                  |
|  +-----------+   |                              |                  |
+------------------+                              +------------------+
```