diff --git a/backup-testing/README.md b/backup-testing/README.md index 94d4c2f..2fc8781 100644 --- a/backup-testing/README.md +++ b/backup-testing/README.md @@ -2,6 +2,16 @@ This guide explains how to test your Velero backups using a temporary K3d cluster. +## Features + +- Creates isolated K3d test cluster +- Installs nginx-ingress controller for ingress support +- Installs cert-manager with self-signed certificates (no real DNS/certs needed) +- Automatically rewrites ingress hosts from `*.hensen.io` to `*.local` +- Bypasses authentication proxies (sol-auth) by rewriting ingress backends +- Generates `/etc/hosts` entries for local DNS resolution +- Restores backups with storage class mapping (longhorn -> local-path) + ## Prerequisites Ensure you have the following tools installed on your local machine: @@ -11,6 +21,7 @@ Ensure you have the following tools installed on your local machine: - Helm - K3d - Velero CLI +- jq (for JSON parsing) ## Setup @@ -33,20 +44,56 @@ Execute the test script with appropriate parameters: ```bash ./test-velero-backup.sh \ ---s3-access-key \ ---s3-secret-key \ ---backup-name \ ---original-namespace \ ---test-namespace + --s3-access-key \ + --s3-secret-key \ + --backup-name \ + --original-namespace \ + --test-namespace \ + --generate-hosts ``` This will: -+ Create a temporary K3d cluster -+ Install Velero configured to access your MinIO backup location -+ List available backups -+ Allow you to test restoring a specific backup +1. Create a temporary K3d cluster +2. Install nginx-ingress controller +3. Install cert-manager with self-signed ClusterIssuer +4. Install Velero configured to access your MinIO backup location +5. Restore the specified backup (excluding sol-auth and certificates) +6. Remove any restored sol-auth resources +7. Rewrite ingress backends from sol-auth to direct backend services +8. Rewrite ingress hosts from `*.hensen.io` to `*.local` +9. Generate `/etc/hosts` entries (with `--generate-hosts` flag) -To restore a specific backup, run: +## Command Line Options + +| Option | Description | Default | +|--------|-------------|---------| +| `--backup-name NAME` | Name of the Velero backup to restore | (lists backups if empty) | +| `--s3-access-key KEY` | S3/MinIO access key | (required) | +| `--s3-secret-key KEY` | S3/MinIO secret key | (required) | +| `--s3-bucket BUCKET` | S3 bucket containing backups | `velero-backups` | +| `--s3-region REGION` | S3 region | `minio` | +| `--s3-url URL` | S3 endpoint URL | `http://192.168.1.99:9000` | +| `--original-namespace NS` | Namespace in the original backup | (required) | +| `--test-namespace NS` | Namespace for restoration | `default` | +| `--original-domain DOMAIN` | Domain to replace in ingresses | `hensen.io` | +| `--local-domain DOMAIN` | Local domain suffix | `local` | +| `--skip-ingress-setup` | Skip nginx-ingress and cert-manager | `false` | +| `--generate-hosts` | Generate /etc/hosts entries | `false` | +| `--debug` | Enable debug output | `false` | + +## Local DNS Resolution + +After the restore completes, the script can generate `/etc/hosts` entries. Add these to your `/etc/hosts` file to access services locally: + +```bash +# Example output with --generate-hosts: +127.0.0.1 foto.local +127.0.0.1 immich.local +``` + +Then access your restored services at `https://foto.local` or `https://immich.local`. + +Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing. ## Testing Specific Applications @@ -58,61 +105,127 @@ For testing Immich backups: ./test-velero-backup.sh \ --s3-access-key \ --s3-secret-key \ - --backup-name immich \ + --backup-name daily-backup-YYYYMMDD \ --original-namespace immich \ - --test-namespace immich + --test-namespace immich \ + --generate-hosts +``` +Then add the generated hosts entries and access via `https://immich.local` + +Alternatively, use port-forwarding: +```bash kubectl port-forward service/helm-immich-server -n immich 8080:2283 ``` Reachable on `localhost:8080` -## Testing with Volume Snapshots (not tested yet) +## Custom Domain Mapping -To test CSI volume snapshot backups: +By default, the script rewrites `*.hensen.io` to `*.local`. You can customize this: ```bash ./test-velero-backup.sh \ --s3-access-key YOUR_ACCESS_KEY \ --s3-secret-key YOUR_SECRET_KEY \ - --original-namespace immich \ - --include-volume-snapshots -``` - -## Customizing Validation (not tested yet) - -You can extend the `validate-restore.sh` script to perform application-specific validation checks: - -1. Edit the script to add application-specific checks -2. Run your test with the modified validation: - -```bash -./test-velero-backup.sh \ - --s3-access-key YOUR_ACCESS_KEY \ - --s3-secret-key YOUR_SECRET_KEY \ - --original-namespace immich \ --backup-name your-backup \ - --custom-validation-script ./your-custom-validation.sh + --original-namespace myapp \ + --original-domain example.com \ + --local-domain test.local ``` -## Troubleshooting (not tested yet) +This will rewrite `app.example.com` to `app.test.local`. -If you encounter issues: +## Authentication Proxy Bypass -1. Check the logs of the Velero pod: - ``` - kubectl logs -n velero deploy/velero - ``` +The production environment uses `sol-auth` (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed: -2. Examine restore details: - ``` - velero restore describe RESTORE_NAME - velero restore logs RESTORE_NAME - ``` +1. **Sol-auth resources are removed** after restore (deployment, service, configmap) +2. **Ingress backends are rewritten** from `sol-auth-svc:8002` to the actual backend service (e.g., `helm-immich-server:2283`) -3. Check cluster resources: - ``` - kubectl get pods -A - kubectl get pvc -A - ``` +This allows you to access the application directly without needing to authenticate through the Scouting login portal. + +**Production flow:** +``` +Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283 +``` + +**Test flow:** +``` +Browser -> Ingress -> helm-immich-server:2283 (direct, no auth) +``` + +## Storage Class Mapping + +The script automatically applies `change-storage-class.yaml` which maps: +- `longhorn` -> `local-path` + +This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner. + +## Troubleshooting + +### Check Velero logs +```bash +kubectl logs -n velero deploy/velero +``` + +### Examine restore details +```bash +velero restore describe +velero restore logs +``` + +### Check cluster resources +```bash +kubectl get pods -A +kubectl get pvc -A +kubectl get ingress -A +``` + +### Check ingress controller +```bash +kubectl logs -n ingress-nginx deploy/ingress-nginx-controller +``` + +### Check cert-manager +```bash +kubectl get certificates -A +kubectl get clusterissuer +``` + +### Debug mode +Run with `--debug` for verbose output: +```bash +./test-velero-backup.sh --debug ... +``` + +## Architecture + +``` ++------------------+ +------------------+ +------------------+ +| K3d Cluster | | MinIO | | Your Browser | +| | | (NAS/S3) | | | +| +-----------+ | | | | | +| | Velero |<--+-----+ velero-backups | | | +| +-----------+ | | bucket | | | +| | | +------------------+ | | +| v | | | +| +-----------+ | | | +| | Restored | | | | +| | App | | | | +| +-----------+ | | | +| | | | | +| v | | | +| +-----------+ | *.local domains | | +| | Ingress |<---------------------------------+ https://app.local +| | nginx | | (via /etc/hosts) | | +| +-----------+ | | | +| | | | | +| v | | | +| +-----------+ | | | +| |cert-manager| | Self-signed certs | | +| |(selfsigned)| | | | +| +-----------+ | | | ++------------------+ +------------------+ +``` diff --git a/backup-testing/test-velero-backup.sh b/backup-testing/test-velero-backup.sh index ef8c15f..55f4ed8 100755 --- a/backup-testing/test-velero-backup.sh +++ b/backup-testing/test-velero-backup.sh @@ -1,18 +1,33 @@ #!/bin/bash # test-velero-backup.sh - Automated script to test Velero backups +# +# This script creates a temporary K3d cluster, installs necessary components, +# and restores a Velero backup for testing. Ingresses are automatically +# rewritten from *.hensen.io to *.local for local testing. +# +# Authentication proxies (sol-auth) are excluded and ingresses are rewritten +# to point directly to backend services. set -e # Configuration - customize these variables -BACKUP_NAME="" # Will list backups if empty +BACKUP_NAME="" S3_BUCKET="velero-backups" S3_REGION="minio" -S3_URL="http://192.168.1.99:9000" # Your NAS MinIO server -S3_ACCESS_KEY="" # Will be filled from credentials -S3_SECRET_KEY="" # Will be filled from credentials -ORIGINAL_NAMESPACE="" # Namespace in the original backup +S3_URL="http://192.168.1.99:9000" +S3_ACCESS_KEY="" +S3_SECRET_KEY="" +ORIGINAL_NAMESPACE="" TEST_NAMESPACE="default" -DEBUG=false # Set to true for verbose debugging +DEBUG=false +ORIGINAL_DOMAIN="hensen.io" +LOCAL_DOMAIN="local" +SKIP_INGRESS_SETUP=false +GENERATE_HOSTS=false + +# Auth proxy configuration - these will be excluded and bypassed +AUTH_PROXY_SERVICE="sol-auth-svc" +AUTH_PROXY_PORT="8002" # Parse command line arguments while [[ $# -gt 0 ]]; do @@ -49,23 +64,49 @@ while [[ $# -gt 0 ]]; do TEST_NAMESPACE="$2" shift 2 ;; + --original-domain) + ORIGINAL_DOMAIN="$2" + shift 2 + ;; + --local-domain) + LOCAL_DOMAIN="$2" + shift 2 + ;; + --skip-ingress-setup) + SKIP_INGRESS_SETUP=true + shift + ;; + --generate-hosts) + GENERATE_HOSTS=true + shift + ;; --debug) DEBUG=true shift ;; --help) echo "Usage: $0 [options]" + echo "" echo "Options:" echo " --backup-name NAME Name of the Velero backup to restore" - echo " --s3-bucket BUCKET S3 bucket containing backups" - echo " --s3-region REGION S3 region" - echo " --s3-url URL S3 endpoint URL" + echo " --s3-bucket BUCKET S3 bucket containing backups (default: velero-backups)" + echo " --s3-region REGION S3 region (default: minio)" + echo " --s3-url URL S3 endpoint URL (default: http://192.168.1.99:9000)" echo " --s3-access-key KEY S3 access key" echo " --s3-secret-key KEY S3 secret key" echo " --original-namespace NS Namespace in the original backup" echo " --test-namespace NS Namespace for restoration (default: default)" + echo " --original-domain DOMAIN Domain to replace (default: hensen.io)" + echo " --local-domain DOMAIN Local domain suffix (default: local)" + echo " --skip-ingress-setup Skip nginx-ingress and cert-manager installation" + echo " --generate-hosts Generate /etc/hosts entries for local DNS" echo " --debug Enable debug output" echo " --help Show this help message" + echo "" + echo "Notes:" + echo " - Sol-auth proxy resources are automatically excluded from restore" + echo " - Ingresses pointing to sol-auth are rewritten to point directly to backends" + echo " - Certificates are excluded; self-signed certs are used instead" exit 0 ;; *) @@ -76,15 +117,26 @@ while [[ $# -gt 0 ]]; do done # Verify required variables -if [[ -z "$S3_BUCKET" || -z "$S3_REGION" || -z "$S3_ACCESS_KEY" || -z "$S3_SECRET_KEY" || -z "$ORIGINAL_NAMESPACE" || -z "$TEST_NAMESPACE" ]]; then - echo "Error: S3 configuration is incomplete" +if [[ -z "$S3_ACCESS_KEY" || -z "$S3_SECRET_KEY" || -z "$ORIGINAL_NAMESPACE" ]]; then + echo "Error: Required parameters missing" + echo "Required: --s3-access-key, --s3-secret-key, --original-namespace" + echo "Run with --help for usage information" exit 1 fi +# Check for required tools +for tool in docker kubectl helm k3d velero jq; do + if ! command -v "$tool" &> /dev/null; then + echo "Error: Required tool '$tool' is not installed" + exit 1 + fi +done + # Function to clean up resources cleanup() { echo "Cleaning up resources..." - k3d cluster delete backup-test || true + k3d cluster delete backup-test 2>/dev/null || true + rm -f velero-credentials 2>/dev/null || true echo "Cleanup complete" } @@ -95,7 +147,217 @@ debug() { fi } -# Register cleanup function to run on exit (ctrl+c, etc.) +# Function to install nginx-ingress controller +install_nginx_ingress() { + echo "Installing nginx-ingress controller..." + helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx + helm repo update ingress-nginx + + helm install ingress-nginx ingress-nginx/ingress-nginx \ + --namespace ingress-nginx \ + --create-namespace \ + --set controller.service.type=LoadBalancer \ + --set controller.watchIngressWithoutClass=true \ + --set controller.ingressClassResource.default=true \ + --wait + + echo "Waiting for nginx-ingress to be ready..." + kubectl -n ingress-nginx rollout status deployment/ingress-nginx-controller --timeout=120s +} + +# Function to install cert-manager with self-signed issuer +install_cert_manager() { + echo "Installing cert-manager..." + helm repo add jetstack https://charts.jetstack.io + helm repo update jetstack + + helm install cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace \ + --set crds.enabled=true \ + --wait + + echo "Waiting for cert-manager to be ready..." + kubectl -n cert-manager rollout status deployment/cert-manager --timeout=120s + kubectl -n cert-manager rollout status deployment/cert-manager-webhook --timeout=120s + + # Create self-signed ClusterIssuers (including one named like production for compatibility) + echo "Creating self-signed ClusterIssuers..." + kubectl apply -f - </dev/null || true) + + if [[ -z "$ingresses" ]]; then + echo "No ingresses found in namespace $namespace" + return + fi + + for ingress in $ingresses; do + local ingress_name + ingress_name=$(echo "$ingress" | cut -d'/' -f2) + debug "Processing ingress: $ingress_name" + + # Get current hosts and rewrite rules + local current_hosts + current_hosts=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.rules[*].host}') + debug "Current hosts: $current_hosts" + + local rule_index=0 + for host in $current_hosts; do + if [[ "$host" == *".$ORIGINAL_DOMAIN" ]]; then + local new_host="${host%.$ORIGINAL_DOMAIN}.$LOCAL_DOMAIN" + echo " Rewriting host: $host -> $new_host" + kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \ + -p "[{\"op\": \"replace\", \"path\": \"/spec/rules/$rule_index/host\", \"value\": \"$new_host\"}]" 2>/dev/null || true + fi + ((rule_index++)) || true + done + + # Update TLS hosts if present + local tls_count + tls_count=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.tls}' 2>/dev/null | jq -r 'length // 0' 2>/dev/null || echo "0") + + for ((tls_index=0; tls_index/dev/null || true) + local host_index=0 + for host in $tls_hosts; do + if [[ "$host" == *".$ORIGINAL_DOMAIN" ]]; then + local new_host="${host%.$ORIGINAL_DOMAIN}.$LOCAL_DOMAIN" + kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \ + -p "[{\"op\": \"replace\", \"path\": \"/spec/tls/$tls_index/hosts/$host_index\", \"value\": \"$new_host\"}]" 2>/dev/null || true + fi + ((host_index++)) || true + done + done + + # Update cert-manager annotation to use self-signed issuer + kubectl annotate ingress -n "$namespace" "$ingress_name" \ + cert-manager.io/cluster-issuer=selfsigned-issuer --overwrite 2>/dev/null || true + done + + echo "Ingress host rewriting complete" +} + +# Function to bypass auth proxies by rewriting ingress backends +bypass_auth_proxy() { + local namespace="$1" + echo "Bypassing auth proxy ($AUTH_PROXY_SERVICE) in namespace $namespace..." + + local ingresses + ingresses=$(kubectl get ingress -n "$namespace" -o name 2>/dev/null || true) + + if [[ -z "$ingresses" ]]; then + echo "No ingresses found in namespace $namespace" + return + fi + + for ingress in $ingresses; do + local ingress_name + ingress_name=$(echo "$ingress" | cut -d'/' -f2) + + # Get the ingress JSON + local ingress_json + ingress_json=$(kubectl get ingress -n "$namespace" "$ingress_name" -o json) + + # Check if any backend points to the auth proxy service + if echo "$ingress_json" | grep -q "$AUTH_PROXY_SERVICE"; then + echo " Found auth proxy in ingress: $ingress_name" + + # Get the number of rules + local rule_count + rule_count=$(echo "$ingress_json" | jq '.spec.rules | length') + + for ((rule_idx=0; rule_idx $real_backend:$real_port" + + kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \ + -p "[{\"op\": \"replace\", \"path\": \"/spec/rules/$rule_idx/http/paths/$path_idx/backend/service/name\", \"value\": \"$real_backend\"}, + {\"op\": \"replace\", \"path\": \"/spec/rules/$rule_idx/http/paths/$path_idx/backend/service/port/number\", \"value\": $real_port}]" 2>/dev/null || true + fi + done + done + fi + done + + echo "Auth proxy bypass complete" +} + +# Function to generate /etc/hosts entries +generate_hosts_entries() { + local namespace="$1" + echo "" + echo "=== /etc/hosts entries ===" + echo "Add the following lines to your /etc/hosts file:" + echo "" + + local ingresses + ingresses=$(kubectl get ingress -n "$namespace" -o name 2>/dev/null || true) + + if [[ -z "$ingresses" ]]; then + echo "# No ingresses found in namespace $namespace" + return + fi + + for ingress in $ingresses; do + local ingress_name + ingress_name=$(echo "$ingress" | cut -d'/' -f2) + local hosts + hosts=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.rules[*].host}') + for host in $hosts; do + echo "127.0.0.1 $host" + done + done + + echo "" + echo "=== End of /etc/hosts entries ===" + echo "" +} + +# Register cleanup function to run on exit trap cleanup EXIT echo "=== Starting Velero backup test process ===" @@ -109,31 +371,49 @@ k3d cluster create backup-test \ --agents 0 \ --k3s-arg "--disable=traefik@server:0" \ --wait - # -v /mnt/longhorn:/var/lib/longhorn:shared@all \ - # -i hebury/k3s:v1.32.2-k3s1 \ - # --wait -# Verify cluster is running kubectl config use-context k3d-backup-test kubectl get nodes -# 2. Install Longhorn -# echo "Installing Longhorn..." -# kubectl create namespace longhorn-system +# Test network connectivity to MinIO from within the cluster +echo "Testing connectivity to MinIO from cluster..." +S3_HOST=$(echo "$S3_URL" | sed -E 's|https?://([^:/]+).*|\1|') +S3_PORT=$(echo "$S3_URL" | sed -E 's|https?://[^:/]+:([0-9]+).*|\1|') +S3_PORT=${S3_PORT:-9000} -# helm repo add longhorn https://charts.longhorn.io -# helm repo update +# Run a test pod to check connectivity +kubectl run nettest --image=busybox:1.36 --restart=Never -- sleep 60 2>/dev/null || true +if ! kubectl wait --for=condition=Ready pod/nettest --timeout=60s 2>/dev/null; then + echo "Warning: Test pod not ready, skipping connectivity test" +else + if kubectl exec nettest -- wget -q -O /dev/null --timeout=5 "http://$S3_HOST:$S3_PORT/minio/health/live" 2>/dev/null; then + echo "Connectivity to MinIO ($S3_HOST:$S3_PORT): OK" + elif kubectl exec nettest -- nc -zv -w5 "$S3_HOST" "$S3_PORT" 2>&1 | grep -qE "open|succeeded"; then + echo "Connectivity to MinIO ($S3_HOST:$S3_PORT): OK" + else + echo "" + echo "ERROR: Cannot reach MinIO at $S3_HOST:$S3_PORT from within the cluster" + echo "" + echo "Debug info:" + echo " - k3d network: $(docker network inspect k3d-backup-test --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' 2>/dev/null || echo 'unknown')" + echo "" + echo "The k3d cluster cannot route to your LAN. Possible fixes:" + echo " 1. Add a route on your NAS firewall for Docker subnets (172.x.x.x)" + echo " 2. Check if iptables is blocking traffic: sudo iptables -L -n | grep 172" + echo "" + kubectl delete pod nettest --ignore-not-found >/dev/null 2>&1 || true + exit 1 + fi +fi +kubectl delete pod nettest --ignore-not-found >/dev/null 2>&1 || true -# helm install longhorn longhorn/longhorn \ -# --namespace longhorn-system \ -# --set persistence.defaultClassReplicaCount=1 \ -# --set defaultSettings.backupTarget="" \ -# --set defaultSettings.defaultReplicaCount=1 - -# echo "Waiting for Longhorn to be ready..." -# kubectl -n longhorn-system rollout status deployment/longhorn-ui -# kubectl -n longhorn-system rollout status deployment/longhorn-driver-deployer -# kubectl -n longhorn-system rollout status daemonset/longhorn-manager +# 2. Install ingress and certificate management +if [[ "$SKIP_INGRESS_SETUP" != "true" ]]; then + install_nginx_ingress + install_cert_manager +else + echo "Skipping ingress and cert-manager setup (--skip-ingress-setup)" +fi # 3. Install Velero with S3 provider echo "Installing Velero..." @@ -144,7 +424,7 @@ aws_secret_access_key=$S3_SECRET_KEY EOF helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts -helm repo update +helm repo update vmware-tanzu debug "Installing Velero with S3 configuration:" debug " Bucket: $S3_BUCKET" @@ -175,101 +455,111 @@ helm install velero vmware-tanzu/velero \ rm velero-credentials -# Wait for Velero to be ready echo "Waiting for Velero to be fully ready..." kubectl -n velero rollout status deployment/velero --timeout=120s -echo "Install change-storage-class.yaml" +echo "Installing storage class mapping..." kubectl apply -f change-storage-class.yaml -# Check if velero CLI is installed and functioning +# Check if velero CLI is installed if ! command -v velero &> /dev/null; then echo "Error: Velero CLI is not installed or not in PATH" - echo "Please install the Velero CLI by following instructions at: https://velero.io/docs/main/basic-install/" + echo "Install from: https://velero.io/docs/main/basic-install/" exit 1 fi # Verify Velero is connected to the storage location echo "Checking Velero backup storage location..." -kubectl -n velero get backupstoragelocation default -o jsonpath='{.status.phase}' | grep -q "Available" || { - echo "Warning: Backup storage location is not available, it may take a few minutes to initialize" - echo "Storage location status:" - kubectl -n velero get backupstoragelocation default -o yaml - - # Wait a bit longer for the location to become available - echo "Waiting 30 seconds for backup storage location to become available..." - sleep 30 -} - -# 4. List backups -echo "Checking for available backups..." -if ! velero_output=$(velero backup get 2>&1); then - echo "Error listing backups: $velero_output" - - # Try to verify S3 connectivity - echo "Checking S3 connectivity..." - - # Install AWS CLI if needed - if ! command -v aws &> /dev/null; then - echo "AWS CLI not found, skipping S3 connectivity test" - else - # Create temporary AWS profile - mkdir -p ~/.aws - cat > ~/.aws/credentials < ~/.aws/config </dev/null || echo "Unknown") + if [[ "$BSL_STATUS" == "Available" ]]; then + echo "Backup storage location is available" + BSL_AVAILABLE=true + break fi - - if [[ -z "$BACKUP_NAME" ]]; then - echo "No backup name specified and no backups found. Exiting." - exit 1 - fi -else - echo "Available backups:" - echo "$velero_output" - - if [[ -z "$BACKUP_NAME" ]]; then - echo "Specify a backup name using --backup-name and run again." - exit 0 + echo "Waiting for backup storage location... ($i/32) - Status: $BSL_STATUS" + if [[ "$DEBUG" == "true" ]]; then + echo "[DEBUG] BackupStorageLocation details:" + kubectl -n velero get backupstoragelocation default -o yaml 2>/dev/null || true fi + sleep 5 +done + +if [[ "$BSL_AVAILABLE" != "true" ]]; then + echo "" + echo "ERROR: Backup storage location did not become available" + echo "" + echo "This usually means the k3d cluster cannot reach your MinIO server at $S3_URL" + echo "" + echo "Possible solutions:" + echo " 1. Use host.k3d.internal instead of the IP address:" + echo " --s3-url http://host.k3d.internal:9000" + echo "" + echo " 2. If MinIO is on your host machine, ensure it's listening on all interfaces" + echo "" + echo " 3. Check BackupStorageLocation status:" + kubectl -n velero get backupstoragelocation default -o yaml 2>/dev/null || true + exit 1 fi +# 4. Wait for backups to sync from S3 +echo "Waiting for Velero to sync backups from S3..." +BACKUP_FOUND=false +for i in {1..32}; do + velero_output=$(velero backup get 2>&1) || true + if echo "$velero_output" | grep -q "^$BACKUP_NAME "; then + BACKUP_FOUND=true + break + fi + echo "Waiting for backup list to sync... ($i/32)" + sleep 5 +done + +echo "Available backups:" +velero backup get 2>&1 || true +echo "" + +if [[ -z "$BACKUP_NAME" ]]; then + echo "Specify a backup name using --backup-name and run again." + exit 0 +fi + +if [[ "$BACKUP_FOUND" != "true" ]]; then + echo "ERROR: Backup '$BACKUP_NAME' not found after waiting" + echo "" + echo "This could mean:" + echo " 1. The backup name is incorrect" + echo " 2. The backup doesn't exist in the S3 bucket" + echo " 3. Velero needs more time to sync" + echo "" + echo "You can check available backups on your production cluster:" + echo " velero backup get" + exit 1 +fi +echo "Found backup: $BACKUP_NAME" + # 5. Create test namespace if needed if [[ "$TEST_NAMESPACE" != "default" ]]; then - kubectl create namespace $TEST_NAMESPACE || true + kubectl create namespace "$TEST_NAMESPACE" 2>/dev/null || true fi -# 6. Restore backup +# 6. Restore backup (excluding auth proxy and certificates) echo "Restoring backup: $BACKUP_NAME" +echo " Excluding: sol-auth resources, certificates" + +RESTORE_NAME="test-restore-$(date +%s)" + if [[ -z "$ORIGINAL_NAMESPACE" ]]; then - # If no namespace specified, restore everything - velero restore create --from-backup $BACKUP_NAME \ + velero restore create "$RESTORE_NAME" \ + --from-backup "$BACKUP_NAME" \ + --exclude-resources certificates.cert-manager.io \ --wait else - # Restore specific namespace with mapping - velero restore create --from-backup $BACKUP_NAME \ - --namespace-mappings $ORIGINAL_NAMESPACE:$TEST_NAMESPACE \ - --include-namespaces $ORIGINAL_NAMESPACE \ + velero restore create "$RESTORE_NAME" \ + --from-backup "$BACKUP_NAME" \ + --namespace-mappings "$ORIGINAL_NAMESPACE:$TEST_NAMESPACE" \ + --include-namespaces "$ORIGINAL_NAMESPACE" \ --exclude-namespaces kube-system,kube-public,kube-node-lease,velero \ --exclude-resources certificates.cert-manager.io \ --wait @@ -277,105 +567,42 @@ fi # 7. Check restore status echo "Checking restore status:" -velero restore get +velero restore describe "$RESTORE_NAME" -# Wait for pods to be ready +# 8. Delete sol-auth resources (they were restored from backup) +echo "Removing sol-auth resources..." +kubectl delete deployment -n "$TEST_NAMESPACE" sol-auth 2>/dev/null || true +kubectl delete service -n "$TEST_NAMESPACE" "$AUTH_PROXY_SERVICE" 2>/dev/null || true +kubectl delete configmap -n "$TEST_NAMESPACE" httpd-conf 2>/dev/null || true +# Wait for sol-auth pod to terminate +kubectl wait --for=delete pod -l app=sol-auth -n "$TEST_NAMESPACE" --timeout=60s 2>/dev/null || true + +# 9. Wait for pods to be ready echo "Waiting for pods to be ready..." -kubectl wait --for=condition=ready pod --all -n $TEST_NAMESPACE --timeout=600s || true +kubectl wait --for=condition=ready pod --all -n "$TEST_NAMESPACE" --timeout=600s 2>/dev/null || true -# 8. Run validation script (create it first if it doesn't exist) -# if [[ ! -f ./validate-restore.sh ]]; then -# echo "Creating validation script..." -# cat > validate-restore.sh <<'EOF' -# #!/bin/bash -# # validate-restore.sh - Script to validate Velero backup restoration +# 10. Rewrite ingress configuration for local testing +if [[ "$SKIP_INGRESS_SETUP" != "true" ]]; then + bypass_auth_proxy "$TEST_NAMESPACE" + rewrite_ingress_hosts "$TEST_NAMESPACE" +fi -# set -e -# echo "Starting validation of restored resources..." - -# # Get namespace from command line or use default -# NAMESPACE="${1:-default}" - -# # Check all deployments -# echo "Checking deployments..." -# deployments=$(kubectl get deployment -n $NAMESPACE -o name) -# if [[ -z "$deployments" ]]; then -# echo "❌ No deployments found in namespace $NAMESPACE" -# else -# echo "Found deployments: $deployments" - -# # Check if pods are running for each deployment -# for deployment in $deployments; do -# name=$(echo $deployment | cut -d'/' -f2) -# echo -n "Checking deployment $name: " - -# # Check if pods are running -# ready_replicas=$(kubectl get deployment -n $NAMESPACE $name -o jsonpath='{.status.readyReplicas}') -# if [[ -n "$ready_replicas" && "$ready_replicas" != "0" ]]; then -# echo "✅ $ready_replicas pods ready" -# else -# echo "❌ No pods running" -# fi -# done -# fi - -# # Check all services -# echo "Checking services..." -# services=$(kubectl get service -n $NAMESPACE -o name | grep -v "kubernetes") -# if [[ -z "$services" ]]; then -# echo "❌ No services found in namespace $NAMESPACE" -# else -# echo "Found services: $services" - -# # Check if services have endpoints -# for service in $services; do -# name=$(echo $service | cut -d'/' -f2) -# echo -n "Checking service $name: " - -# # Check if service has endpoints -# endpoints=$(kubectl get endpoints -n $NAMESPACE $name -o jsonpath='{.subsets[*].addresses[*].ip}') -# if [ -n "$endpoints" ]; then -# echo "✅ Has endpoints" -# else -# echo "❌ No endpoints" -# fi -# done -# fi - -# # Check PVCs -# echo "Checking PVCs..." -# pvcs=$(kubectl get pvc -n $NAMESPACE -o name) -# if [[ -z "$pvcs" ]]; then -# echo "No PVCs found in namespace $NAMESPACE" -# else -# pvc_count=$(echo "$pvcs" | wc -l) -# echo "Found $pvc_count PVCs" - -# # Check if PVCs are bound -# bound_count=$(kubectl get pvc -n $NAMESPACE -o jsonpath='{.items[?(@.status.phase=="Bound")].metadata.name}' | wc -w) -# echo "$bound_count/$pvc_count PVCs are bound" - -# if [ "$bound_count" -ne "$pvc_count" ]; then -# echo "❌ Not all PVCs are bound" -# else -# echo "✅ All PVCs are bound" -# fi -# fi - -# echo "Validation complete!" -# exit 0 -# EOF -# chmod +x validate-restore.sh -# fi +# 11. Generate /etc/hosts entries if requested +if [[ "$GENERATE_HOSTS" == "true" ]]; then + generate_hosts_entries "$TEST_NAMESPACE" +fi +# 12. Run validation echo "Running validation tests..." -./validate-restore.sh $TEST_NAMESPACE +./validate-restore.sh "$TEST_NAMESPACE" echo "" echo "=== Backup test process completed ===" -echo "The cluster will be deleted when you close this terminal or press Ctrl+C" -echo "To access the cluster manually, use: kubectl config use-context k3d-backup-test" echo "" +echo "Cluster: k3d-backup-test" +echo "Namespace: $TEST_NAMESPACE" +echo "Context: kubectl config use-context k3d-backup-test" +echo "" +echo "The cluster will be deleted when you press Enter or Ctrl+C" -# Keep the script running until user terminates it -read -p "Press Enter to clean up and exit..." \ No newline at end of file +read -p "Press Enter to clean up and exit..."