rubenhensen/k8scd
1
0
Fork 0
mirror of https://github.com/rubenhensen/k8scd.git synced 2026-09-17 02:12:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update restore script

Browse Source
This commit is contained in:
Ruben Hensen
2026-01-20 22:46:46 +01:00
parent 6f75c24b8c
commit 2f3374c511
2 changed files with 582 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-testing/README.md
+160 -47
View File
@@ -2,6 +2,16 @@
This guide explains how to test your Velero backups using a temporary K3d cluster.
## Features
- Creates isolated K3d test cluster
- Installs nginx-ingress controller for ingress support
- Installs cert-manager with self-signed certificates (no real DNS/certs needed)
- Automatically rewrites ingress hosts from `*.hensen.io` to `*.local`
- Bypasses authentication proxies (sol-auth) by rewriting ingress backends
- Generates `/etc/hosts` entries for local DNS resolution
- Restores backups with storage class mapping (longhorn -> local-path)
## Prerequisites
Ensure you have the following tools installed on your local machine:
@@ -11,6 +21,7 @@ Ensure you have the following tools installed on your local machine:
- Helm
- K3d
- Velero CLI
- jq (for JSON parsing)
## Setup
@@ -33,20 +44,56 @@ Execute the test script with appropriate parameters:
```bash
./test-velero-backup.sh \
--s3-access-key <ACCESS_KEY> \
--s3-secret-key <SECRET_KEY> \
--backup-name <BACKUP_NAME> \
--original-namespace <NAMESPACE> \
--test-namespace <NAMESPACE>
--s3-access-key <ACCESS_KEY> \
--s3-secret-key <SECRET_KEY> \
--backup-name <BACKUP_NAME> \
--original-namespace <NAMESPACE> \
--test-namespace <NAMESPACE> \
--generate-hosts
```
This will:
+ Create a temporary K3d cluster
+ Install Velero configured to access your MinIO backup location
+ List available backups
+ Allow you to test restoring a specific backup
1. Create a temporary K3d cluster
2. Install nginx-ingress controller
3. Install cert-manager with self-signed ClusterIssuer
4. Install Velero configured to access your MinIO backup location
5. Restore the specified backup (excluding sol-auth and certificates)
6. Remove any restored sol-auth resources
7. Rewrite ingress backends from sol-auth to direct backend services
8. Rewrite ingress hosts from `*.hensen.io` to `*.local`
9. Generate `/etc/hosts` entries (with `--generate-hosts` flag)
To restore a specific backup, run:
## Command Line Options
| Option | Description | Default |
|--------|-------------|---------|
| `--backup-name NAME` | Name of the Velero backup to restore | (lists backups if empty) |
| `--s3-access-key KEY` | S3/MinIO access key | (required) |
| `--s3-secret-key KEY` | S3/MinIO secret key | (required) |
| `--s3-bucket BUCKET` | S3 bucket containing backups | `velero-backups` |
| `--s3-region REGION` | S3 region | `minio` |
| `--s3-url URL` | S3 endpoint URL | `http://192.168.1.99:9000` |
| `--original-namespace NS` | Namespace in the original backup | (required) |
| `--test-namespace NS` | Namespace for restoration | `default` |
| `--original-domain DOMAIN` | Domain to replace in ingresses | `hensen.io` |
| `--local-domain DOMAIN` | Local domain suffix | `local` |
| `--skip-ingress-setup` | Skip nginx-ingress and cert-manager | `false` |
| `--generate-hosts` | Generate /etc/hosts entries | `false` |
| `--debug` | Enable debug output | `false` |
## Local DNS Resolution
After the restore completes, the script can generate `/etc/hosts` entries. Add these to your `/etc/hosts` file to access services locally:
```bash
# Example output with --generate-hosts:
127.0.0.1 foto.local
127.0.0.1 immich.local
```
Then access your restored services at `https://foto.local` or `https://immich.local`.
Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.
## Testing Specific Applications
@@ -58,61 +105,127 @@ For testing Immich backups:
./test-velero-backup.sh \
--s3-access-key <ACCESS_KEY> \
--s3-secret-key <SECRET_KEY> \
--backup-name immich \
--backup-name daily-backup-YYYYMMDD \
--original-namespace immich \
--test-namespace immich
--test-namespace immich \
--generate-hosts
```
Then add the generated hosts entries and access via `https://immich.local`
Alternatively, use port-forwarding:
```bash
kubectl port-forward service/helm-immich-server -n immich 8080:2283
```
Reachable on `localhost:8080`
## Testing with Volume Snapshots (not tested yet)
## Custom Domain Mapping
To test CSI volume snapshot backups:
By default, the script rewrites `*.hensen.io` to `*.local`. You can customize this:
```bash
./test-velero-backup.sh \
--s3-access-key YOUR_ACCESS_KEY \
--s3-secret-key YOUR_SECRET_KEY \
--original-namespace immich \
--include-volume-snapshots
```
## Customizing Validation (not tested yet)
You can extend the `validate-restore.sh` script to perform application-specific validation checks:
1. Edit the script to add application-specific checks
2. Run your test with the modified validation:
```bash
./test-velero-backup.sh \
--s3-access-key YOUR_ACCESS_KEY \
--s3-secret-key YOUR_SECRET_KEY \
--original-namespace immich \
--backup-name your-backup \
--custom-validation-script ./your-custom-validation.sh
--original-namespace myapp \
--original-domain example.com \
--local-domain test.local
```
## Troubleshooting (not tested yet)
This will rewrite `app.example.com` to `app.test.local`.
If you encounter issues:
## Authentication Proxy Bypass
1. Check the logs of the Velero pod:
```
kubectl logs -n velero deploy/velero
```
The production environment uses `sol-auth` (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:
2. Examine restore details:
```
velero restore describe RESTORE_NAME
velero restore logs RESTORE_NAME
```
1. **Sol-auth resources are removed** after restore (deployment, service, configmap)
2. **Ingress backends are rewritten** from `sol-auth-svc:8002` to the actual backend service (e.g., `helm-immich-server:2283`)
3. Check cluster resources:
```
kubectl get pods -A
kubectl get pvc -A
```
This allows you to access the application directly without needing to authenticate through the Scouting login portal.
**Production flow:**
```
Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283
```
**Test flow:**
```
Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)
```
## Storage Class Mapping
The script automatically applies `change-storage-class.yaml` which maps:
- `longhorn` -> `local-path`
This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.
## Troubleshooting
### Check Velero logs
```bash
kubectl logs -n velero deploy/velero
```
### Examine restore details
```bash
velero restore describe <RESTORE_NAME>
velero restore logs <RESTORE_NAME>
```
### Check cluster resources
```bash
kubectl get pods -A
kubectl get pvc -A
kubectl get ingress -A
```
### Check ingress controller
```bash
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
```
### Check cert-manager
```bash
kubectl get certificates -A
kubectl get clusterissuer
```
### Debug mode
Run with `--debug` for verbose output:
```bash
./test-velero-backup.sh --debug ...
```
## Architecture
```
+------------------+ +------------------+ +------------------+
| K3d Cluster | | MinIO | | Your Browser |
| | | (NAS/S3) | | |
| +-----------+ | | | | |
| | Velero |<--+-----+ velero-backups | | |
| +-----------+ | | bucket | | |
| | | +------------------+ | |
| v | | |
| +-----------+ | | |
| | Restored | | | |
| | App | | | |
| +-----------+ | | |
| | | | |
| v | | |
| +-----------+ | *.local domains | |
| | Ingress |<---------------------------------+ https://app.local
| | nginx | | (via /etc/hosts) | |
| +-----------+ | | |
| | | | |
| v | | |
| +-----------+ | | |
| |cert-manager| | Self-signed certs | |
| |(selfsigned)| | | |
| +-----------+ | | |
+------------------+ +------------------+
```