rubenhensen/k8scd
1
0
Fork 0
mirror of https://github.com/rubenhensen/k8scd.git synced 2026-09-17 02:12:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update restore script

Browse Source
This commit is contained in:
Ruben Hensen
2026-01-20 22:46:46 +01:00
parent 6f75c24b8c
commit 2f3374c511
2 changed files with 582 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-testing/README.md
+149 -36
View File
@@ -2,6 +2,16 @@
This guide explains how to test your Velero backups using a temporary K3d cluster. This guide explains how to test your Velero backups using a temporary K3d cluster.
## Features
- Creates isolated K3d test cluster
- Installs nginx-ingress controller for ingress support
- Installs cert-manager with self-signed certificates (no real DNS/certs needed)
- Automatically rewrites ingress hosts from `*.hensen.io` to `*.local`
- Bypasses authentication proxies (sol-auth) by rewriting ingress backends
- Generates `/etc/hosts` entries for local DNS resolution
- Restores backups with storage class mapping (longhorn -> local-path)
## Prerequisites ## Prerequisites
Ensure you have the following tools installed on your local machine: Ensure you have the following tools installed on your local machine:
@@ -11,6 +21,7 @@ Ensure you have the following tools installed on your local machine:
- Helm - Helm
- K3d - K3d
- Velero CLI - Velero CLI
- jq (for JSON parsing)
## Setup ## Setup
@@ -37,16 +48,52 @@ Execute the test script with appropriate parameters:
--s3-secret-key <SECRET_KEY> \ --s3-secret-key <SECRET_KEY> \
--backup-name <BACKUP_NAME> \ --backup-name <BACKUP_NAME> \
--original-namespace <NAMESPACE> \ --original-namespace <NAMESPACE> \
--test-namespace <NAMESPACE> --test-namespace <NAMESPACE> \
--generate-hosts
``` ```
This will: This will:
+ Create a temporary K3d cluster 1. Create a temporary K3d cluster
+ Install Velero configured to access your MinIO backup location 2. Install nginx-ingress controller
+ List available backups 3. Install cert-manager with self-signed ClusterIssuer
+ Allow you to test restoring a specific backup 4. Install Velero configured to access your MinIO backup location
5. Restore the specified backup (excluding sol-auth and certificates)
6. Remove any restored sol-auth resources
7. Rewrite ingress backends from sol-auth to direct backend services
8. Rewrite ingress hosts from `*.hensen.io` to `*.local`
9. Generate `/etc/hosts` entries (with `--generate-hosts` flag)
To restore a specific backup, run: ## Command Line Options
| Option | Description | Default |
|--------|-------------|---------|
| `--backup-name NAME` | Name of the Velero backup to restore | (lists backups if empty) |
| `--s3-access-key KEY` | S3/MinIO access key | (required) |
| `--s3-secret-key KEY` | S3/MinIO secret key | (required) |
| `--s3-bucket BUCKET` | S3 bucket containing backups | `velero-backups` |
| `--s3-region REGION` | S3 region | `minio` |
| `--s3-url URL` | S3 endpoint URL | `http://192.168.1.99:9000` |
| `--original-namespace NS` | Namespace in the original backup | (required) |
| `--test-namespace NS` | Namespace for restoration | `default` |
| `--original-domain DOMAIN` | Domain to replace in ingresses | `hensen.io` |
| `--local-domain DOMAIN` | Local domain suffix | `local` |
| `--skip-ingress-setup` | Skip nginx-ingress and cert-manager | `false` |
| `--generate-hosts` | Generate /etc/hosts entries | `false` |
| `--debug` | Enable debug output | `false` |
## Local DNS Resolution
After the restore completes, the script can generate `/etc/hosts` entries. Add these to your `/etc/hosts` file to access services locally:
```bash
# Example output with --generate-hosts:
127.0.0.1 foto.local
127.0.0.1 immich.local
```
Then access your restored services at `https://foto.local` or `https://immich.local`.
Note: Your browser will show certificate warnings since we use self-signed certificates. This is expected for local testing.
## Testing Specific Applications ## Testing Specific Applications
@@ -58,61 +105,127 @@ For testing Immich backups:
./test-velero-backup.sh \ ./test-velero-backup.sh \
--s3-access-key <ACCESS_KEY> \ --s3-access-key <ACCESS_KEY> \
--s3-secret-key <SECRET_KEY> \ --s3-secret-key <SECRET_KEY> \
--backup-name immich \ --backup-name daily-backup-YYYYMMDD \
--original-namespace immich \ --original-namespace immich \
--test-namespace immich --test-namespace immich \
--generate-hosts
```
Then add the generated hosts entries and access via `https://immich.local`
Alternatively, use port-forwarding:
```bash
kubectl port-forward service/helm-immich-server -n immich 8080:2283 kubectl port-forward service/helm-immich-server -n immich 8080:2283
``` ```
Reachable on `localhost:8080` Reachable on `localhost:8080`
## Testing with Volume Snapshots (not tested yet) ## Custom Domain Mapping
To test CSI volume snapshot backups: By default, the script rewrites `*.hensen.io` to `*.local`. You can customize this:
```bash ```bash
./test-velero-backup.sh \ ./test-velero-backup.sh \
--s3-access-key YOUR_ACCESS_KEY \ --s3-access-key YOUR_ACCESS_KEY \
--s3-secret-key YOUR_SECRET_KEY \ --s3-secret-key YOUR_SECRET_KEY \
--original-namespace immich \
--include-volume-snapshots
```
## Customizing Validation (not tested yet)
You can extend the `validate-restore.sh` script to perform application-specific validation checks:
1. Edit the script to add application-specific checks
2. Run your test with the modified validation:
```bash
./test-velero-backup.sh \
--s3-access-key YOUR_ACCESS_KEY \
--s3-secret-key YOUR_SECRET_KEY \
--original-namespace immich \
--backup-name your-backup \ --backup-name your-backup \
--custom-validation-script ./your-custom-validation.sh --original-namespace myapp \
--original-domain example.com \
--local-domain test.local
``` ```
## Troubleshooting (not tested yet) This will rewrite `app.example.com` to `app.test.local`.
If you encounter issues: ## Authentication Proxy Bypass
1. Check the logs of the Velero pod: The production environment uses `sol-auth` (an Apache httpd-based OpenID Connect proxy) to protect certain applications like Immich. For local testing, this authentication is bypassed:
1. **Sol-auth resources are removed** after restore (deployment, service, configmap)
2. **Ingress backends are rewritten** from `sol-auth-svc:8002` to the actual backend service (e.g., `helm-immich-server:2283`)
This allows you to access the application directly without needing to authenticate through the Scouting login portal.
**Production flow:**
``` ```
Browser -> Ingress -> sol-auth-svc:8002 -> (OpenID auth) -> helm-immich-server:2283
```
**Test flow:**
```
Browser -> Ingress -> helm-immich-server:2283 (direct, no auth)
```
## Storage Class Mapping
The script automatically applies `change-storage-class.yaml` which maps:
- `longhorn` -> `local-path`
This allows backups from Longhorn storage to be restored using K3d's default local-path provisioner.
## Troubleshooting
### Check Velero logs
```bash
kubectl logs -n velero deploy/velero kubectl logs -n velero deploy/velero
``` ```
2. Examine restore details: ### Examine restore details
``` ```bash
velero restore describe RESTORE_NAME velero restore describe <RESTORE_NAME>
velero restore logs RESTORE_NAME velero restore logs <RESTORE_NAME>
``` ```
3. Check cluster resources: ### Check cluster resources
``` ```bash
kubectl get pods -A kubectl get pods -A
kubectl get pvc -A kubectl get pvc -A
kubectl get ingress -A
```
### Check ingress controller
```bash
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
```
### Check cert-manager
```bash
kubectl get certificates -A
kubectl get clusterissuer
```
### Debug mode
Run with `--debug` for verbose output:
```bash
./test-velero-backup.sh --debug ...
```
## Architecture
```
+------------------+ +------------------+ +------------------+
| K3d Cluster | | MinIO | | Your Browser |
| | | (NAS/S3) | | |
| +-----------+ | | | | |
| | Velero |<--+-----+ velero-backups | | |
| +-----------+ | | bucket | | |
| | | +------------------+ | |
| v | | |
| +-----------+ | | |
| | Restored | | | |
| | App | | | |
| +-----------+ | | |
| | | | |
| v | | |
| +-----------+ | *.local domains | |
| | Ingress |<---------------------------------+ https://app.local
| | nginx | | (via /etc/hosts) | |
| +-----------+ | | |
| | | | |
| v | | |
| +-----------+ | | |
| |cert-manager| | Self-signed certs | |
| |(selfsigned)| | | |
| +-----------+ | | |
+------------------+ +------------------+
``` ```
backup-testing/test-velero-backup.sh
+412 -185
View File
@@ -1,18 +1,33 @@
#!/bin/bash #!/bin/bash
# test-velero-backup.sh - Automated script to test Velero backups # test-velero-backup.sh - Automated script to test Velero backups
#
# This script creates a temporary K3d cluster, installs necessary components,
# and restores a Velero backup for testing. Ingresses are automatically
# rewritten from *.hensen.io to *.local for local testing.
#
# Authentication proxies (sol-auth) are excluded and ingresses are rewritten
# to point directly to backend services.
set -e set -e
# Configuration - customize these variables # Configuration - customize these variables
BACKUP_NAME="" # Will list backups if empty BACKUP_NAME=""
S3_BUCKET="velero-backups" S3_BUCKET="velero-backups"
S3_REGION="minio" S3_REGION="minio"
S3_URL="http://192.168.1.99:9000" # Your NAS MinIO server S3_URL="http://192.168.1.99:9000"
S3_ACCESS_KEY="" # Will be filled from credentials S3_ACCESS_KEY=""
S3_SECRET_KEY="" # Will be filled from credentials S3_SECRET_KEY=""
ORIGINAL_NAMESPACE="" # Namespace in the original backup ORIGINAL_NAMESPACE=""
TEST_NAMESPACE="default" TEST_NAMESPACE="default"
DEBUG=false # Set to true for verbose debugging DEBUG=false
ORIGINAL_DOMAIN="hensen.io"
LOCAL_DOMAIN="local"
SKIP_INGRESS_SETUP=false
GENERATE_HOSTS=false
# Auth proxy configuration - these will be excluded and bypassed
AUTH_PROXY_SERVICE="sol-auth-svc"
AUTH_PROXY_PORT="8002"
# Parse command line arguments # Parse command line arguments
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
@@ -49,23 +64,49 @@ while [[ $# -gt 0 ]]; do
TEST_NAMESPACE="$2" TEST_NAMESPACE="$2"
shift 2 shift 2
;; ;;
--original-domain)
ORIGINAL_DOMAIN="$2"
shift 2
;;
--local-domain)
LOCAL_DOMAIN="$2"
shift 2
;;
--skip-ingress-setup)
SKIP_INGRESS_SETUP=true
shift
;;
--generate-hosts)
GENERATE_HOSTS=true
shift
;;
--debug) --debug)
DEBUG=true DEBUG=true
shift shift
;; ;;
--help) --help)
echo "Usage: $0 [options]" echo "Usage: $0 [options]"
echo ""
echo "Options:" echo "Options:"
echo " --backup-name NAME Name of the Velero backup to restore" echo " --backup-name NAME Name of the Velero backup to restore"
echo " --s3-bucket BUCKET S3 bucket containing backups" echo " --s3-bucket BUCKET S3 bucket containing backups (default: velero-backups)"
echo " --s3-region REGION S3 region" echo " --s3-region REGION S3 region (default: minio)"
echo " --s3-url URL S3 endpoint URL" echo " --s3-url URL S3 endpoint URL (default: http://192.168.1.99:9000)"
echo " --s3-access-key KEY S3 access key" echo " --s3-access-key KEY S3 access key"
echo " --s3-secret-key KEY S3 secret key" echo " --s3-secret-key KEY S3 secret key"
echo " --original-namespace NS Namespace in the original backup" echo " --original-namespace NS Namespace in the original backup"
echo " --test-namespace NS Namespace for restoration (default: default)" echo " --test-namespace NS Namespace for restoration (default: default)"
echo " --original-domain DOMAIN Domain to replace (default: hensen.io)"
echo " --local-domain DOMAIN Local domain suffix (default: local)"
echo " --skip-ingress-setup Skip nginx-ingress and cert-manager installation"
echo " --generate-hosts Generate /etc/hosts entries for local DNS"
echo " --debug Enable debug output" echo " --debug Enable debug output"
echo " --help Show this help message" echo " --help Show this help message"
echo ""
echo "Notes:"
echo " - Sol-auth proxy resources are automatically excluded from restore"
echo " - Ingresses pointing to sol-auth are rewritten to point directly to backends"
echo " - Certificates are excluded; self-signed certs are used instead"
exit 0 exit 0
;; ;;
*) *)
@@ -76,15 +117,26 @@ while [[ $# -gt 0 ]]; do
done done
# Verify required variables # Verify required variables
if [[ -z "$S3_BUCKET" || -z "$S3_REGION" || -z "$S3_ACCESS_KEY" || -z "$S3_SECRET_KEY" || -z "$ORIGINAL_NAMESPACE" || -z "$TEST_NAMESPACE" ]]; then if [[ -z "$S3_ACCESS_KEY" || -z "$S3_SECRET_KEY" || -z "$ORIGINAL_NAMESPACE" ]]; then
echo "Error: S3 configuration is incomplete" echo "Error: Required parameters missing"
echo "Required: --s3-access-key, --s3-secret-key, --original-namespace"
echo "Run with --help for usage information"
exit 1 exit 1
fi fi
# Check for required tools
for tool in docker kubectl helm k3d velero jq; do
if ! command -v "$tool" &> /dev/null; then
echo "Error: Required tool '$tool' is not installed"
exit 1
fi
done
# Function to clean up resources # Function to clean up resources
cleanup() { cleanup() {
echo "Cleaning up resources..." echo "Cleaning up resources..."
k3d cluster delete backup-test || true k3d cluster delete backup-test 2>/dev/null || true
rm -f velero-credentials 2>/dev/null || true
echo "Cleanup complete" echo "Cleanup complete"
} }
@@ -95,7 +147,217 @@ debug() {
fi fi
} }
# Register cleanup function to run on exit (ctrl+c, etc.) # Function to install nginx-ingress controller
install_nginx_ingress() {
echo "Installing nginx-ingress controller..."
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.type=LoadBalancer \
--set controller.watchIngressWithoutClass=true \
--set controller.ingressClassResource.default=true \
--wait
echo "Waiting for nginx-ingress to be ready..."
kubectl -n ingress-nginx rollout status deployment/ingress-nginx-controller --timeout=120s
}
# Function to install cert-manager with self-signed issuer
install_cert_manager() {
echo "Installing cert-manager..."
helm repo add jetstack https://charts.jetstack.io
helm repo update jetstack
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true \
--wait
echo "Waiting for cert-manager to be ready..."
kubectl -n cert-manager rollout status deployment/cert-manager --timeout=120s
kubectl -n cert-manager rollout status deployment/cert-manager-webhook --timeout=120s
# Create self-signed ClusterIssuers (including one named like production for compatibility)
echo "Creating self-signed ClusterIssuers..."
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-issuer
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: prod-cluster-issuer
spec:
selfSigned: {}
EOF
echo "cert-manager installed with self-signed issuers"
}
# Function to rewrite ingress hosts from original domain to local domain
rewrite_ingress_hosts() {
local namespace="$1"
echo "Rewriting ingress hosts from *.$ORIGINAL_DOMAIN to *.$LOCAL_DOMAIN in namespace $namespace..."
local ingresses
ingresses=$(kubectl get ingress -n "$namespace" -o name 2>/dev/null || true)
if [[ -z "$ingresses" ]]; then
echo "No ingresses found in namespace $namespace"
return
fi
for ingress in $ingresses; do
local ingress_name
ingress_name=$(echo "$ingress" | cut -d'/' -f2)
debug "Processing ingress: $ingress_name"
# Get current hosts and rewrite rules
local current_hosts
current_hosts=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.rules[*].host}')
debug "Current hosts: $current_hosts"
local rule_index=0
for host in $current_hosts; do
if [[ "$host" == *".$ORIGINAL_DOMAIN" ]]; then
local new_host="${host%.$ORIGINAL_DOMAIN}.$LOCAL_DOMAIN"
echo " Rewriting host: $host -> $new_host"
kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \
-p "[{\"op\": \"replace\", \"path\": \"/spec/rules/$rule_index/host\", \"value\": \"$new_host\"}]" 2>/dev/null || true
fi
((rule_index++)) || true
done
# Update TLS hosts if present
local tls_count
tls_count=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.tls}' 2>/dev/null | jq -r 'length // 0' 2>/dev/null || echo "0")
for ((tls_index=0; tls_index<tls_count; tls_index++)); do
local tls_hosts
tls_hosts=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath="{.spec.tls[$tls_index].hosts[*]}" 2>/dev/null || true)
local host_index=0
for host in $tls_hosts; do
if [[ "$host" == *".$ORIGINAL_DOMAIN" ]]; then
local new_host="${host%.$ORIGINAL_DOMAIN}.$LOCAL_DOMAIN"
kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \
-p "[{\"op\": \"replace\", \"path\": \"/spec/tls/$tls_index/hosts/$host_index\", \"value\": \"$new_host\"}]" 2>/dev/null || true
fi
((host_index++)) || true
done
done
# Update cert-manager annotation to use self-signed issuer
kubectl annotate ingress -n "$namespace" "$ingress_name" \
cert-manager.io/cluster-issuer=selfsigned-issuer --overwrite 2>/dev/null || true
done
echo "Ingress host rewriting complete"
}
# Function to bypass auth proxies by rewriting ingress backends
bypass_auth_proxy() {
local namespace="$1"
echo "Bypassing auth proxy ($AUTH_PROXY_SERVICE) in namespace $namespace..."
local ingresses
ingresses=$(kubectl get ingress -n "$namespace" -o name 2>/dev/null || true)
if [[ -z "$ingresses" ]]; then
echo "No ingresses found in namespace $namespace"
return
fi
for ingress in $ingresses; do
local ingress_name
ingress_name=$(echo "$ingress" | cut -d'/' -f2)
# Get the ingress JSON
local ingress_json
ingress_json=$(kubectl get ingress -n "$namespace" "$ingress_name" -o json)
# Check if any backend points to the auth proxy service
if echo "$ingress_json" | grep -q "$AUTH_PROXY_SERVICE"; then
echo " Found auth proxy in ingress: $ingress_name"
# Get the number of rules
local rule_count
rule_count=$(echo "$ingress_json" | jq '.spec.rules | length')
for ((rule_idx=0; rule_idx<rule_count; rule_idx++)); do
local path_count
path_count=$(echo "$ingress_json" | jq ".spec.rules[$rule_idx].http.paths | length")
for ((path_idx=0; path_idx<path_count; path_idx++)); do
local backend_service
backend_service=$(echo "$ingress_json" | jq -r ".spec.rules[$rule_idx].http.paths[$path_idx].backend.service.name")
if [[ "$backend_service" == "$AUTH_PROXY_SERVICE" ]]; then
# Find the actual backend service by looking at other services in the namespace
# For Immich, this is helm-immich-server on port 2283
local real_backend="helm-immich-server"
local real_port=2283
echo " Rewriting: $AUTH_PROXY_SERVICE:$AUTH_PROXY_PORT -> $real_backend:$real_port"
kubectl patch ingress -n "$namespace" "$ingress_name" --type=json \
-p "[{\"op\": \"replace\", \"path\": \"/spec/rules/$rule_idx/http/paths/$path_idx/backend/service/name\", \"value\": \"$real_backend\"},
{\"op\": \"replace\", \"path\": \"/spec/rules/$rule_idx/http/paths/$path_idx/backend/service/port/number\", \"value\": $real_port}]" 2>/dev/null || true
fi
done
done
fi
done
echo "Auth proxy bypass complete"
}
# Function to generate /etc/hosts entries
generate_hosts_entries() {
local namespace="$1"
echo ""
echo "=== /etc/hosts entries ==="
echo "Add the following lines to your /etc/hosts file:"
echo ""
local ingresses
ingresses=$(kubectl get ingress -n "$namespace" -o name 2>/dev/null || true)
if [[ -z "$ingresses" ]]; then
echo "# No ingresses found in namespace $namespace"
return
fi
for ingress in $ingresses; do
local ingress_name
ingress_name=$(echo "$ingress" | cut -d'/' -f2)
local hosts
hosts=$(kubectl get ingress -n "$namespace" "$ingress_name" -o jsonpath='{.spec.rules[*].host}')
for host in $hosts; do
echo "127.0.0.1 $host"
done
done
echo ""
echo "=== End of /etc/hosts entries ==="
echo ""
}
# Register cleanup function to run on exit
trap cleanup EXIT trap cleanup EXIT
echo "=== Starting Velero backup test process ===" echo "=== Starting Velero backup test process ==="
@@ -109,31 +371,49 @@ k3d cluster create backup-test \
--agents 0 \ --agents 0 \
--k3s-arg "--disable=traefik@server:0" \ --k3s-arg "--disable=traefik@server:0" \
--wait --wait
# -v /mnt/longhorn:/var/lib/longhorn:shared@all \
# -i hebury/k3s:v1.32.2-k3s1 \
# --wait
# Verify cluster is running
kubectl config use-context k3d-backup-test kubectl config use-context k3d-backup-test
kubectl get nodes kubectl get nodes
# 2. Install Longhorn # Test network connectivity to MinIO from within the cluster
# echo "Installing Longhorn..." echo "Testing connectivity to MinIO from cluster..."
# kubectl create namespace longhorn-system S3_HOST=$(echo "$S3_URL" | sed -E 's|https?://([^:/]+).*|\1|')
S3_PORT=$(echo "$S3_URL" | sed -E 's|https?://[^:/]+:([0-9]+).*|\1|')
S3_PORT=${S3_PORT:-9000}
# helm repo add longhorn https://charts.longhorn.io # Run a test pod to check connectivity
# helm repo update kubectl run nettest --image=busybox:1.36 --restart=Never -- sleep 60 2>/dev/null || true
if ! kubectl wait --for=condition=Ready pod/nettest --timeout=60s 2>/dev/null; then
echo "Warning: Test pod not ready, skipping connectivity test"
else
if kubectl exec nettest -- wget -q -O /dev/null --timeout=5 "http://$S3_HOST:$S3_PORT/minio/health/live" 2>/dev/null; then
echo "Connectivity to MinIO ($S3_HOST:$S3_PORT): OK"
elif kubectl exec nettest -- nc -zv -w5 "$S3_HOST" "$S3_PORT" 2>&1 | grep -qE "open|succeeded"; then
echo "Connectivity to MinIO ($S3_HOST:$S3_PORT): OK"
else
echo ""
echo "ERROR: Cannot reach MinIO at $S3_HOST:$S3_PORT from within the cluster"
echo ""
echo "Debug info:"
echo " - k3d network: $(docker network inspect k3d-backup-test --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' 2>/dev/null || echo 'unknown')"
echo ""
echo "The k3d cluster cannot route to your LAN. Possible fixes:"
echo " 1. Add a route on your NAS firewall for Docker subnets (172.x.x.x)"
echo " 2. Check if iptables is blocking traffic: sudo iptables -L -n | grep 172"
echo ""
kubectl delete pod nettest --ignore-not-found >/dev/null 2>&1 || true
exit 1
fi
fi
kubectl delete pod nettest --ignore-not-found >/dev/null 2>&1 || true
# helm install longhorn longhorn/longhorn \ # 2. Install ingress and certificate management
# --namespace longhorn-system \ if [[ "$SKIP_INGRESS_SETUP" != "true" ]]; then
# --set persistence.defaultClassReplicaCount=1 \ install_nginx_ingress
# --set defaultSettings.backupTarget="" \ install_cert_manager
# --set defaultSettings.defaultReplicaCount=1 else
echo "Skipping ingress and cert-manager setup (--skip-ingress-setup)"
# echo "Waiting for Longhorn to be ready..." fi
# kubectl -n longhorn-system rollout status deployment/longhorn-ui
# kubectl -n longhorn-system rollout status deployment/longhorn-driver-deployer
# kubectl -n longhorn-system rollout status daemonset/longhorn-manager
# 3. Install Velero with S3 provider # 3. Install Velero with S3 provider
echo "Installing Velero..." echo "Installing Velero..."
@@ -144,7 +424,7 @@ aws_secret_access_key=$S3_SECRET_KEY
EOF EOF
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update helm repo update vmware-tanzu
debug "Installing Velero with S3 configuration:" debug "Installing Velero with S3 configuration:"
debug " Bucket: $S3_BUCKET" debug " Bucket: $S3_BUCKET"
@@ -175,101 +455,111 @@ helm install velero vmware-tanzu/velero \
rm velero-credentials rm velero-credentials
# Wait for Velero to be ready
echo "Waiting for Velero to be fully ready..." echo "Waiting for Velero to be fully ready..."
kubectl -n velero rollout status deployment/velero --timeout=120s kubectl -n velero rollout status deployment/velero --timeout=120s
echo "Install change-storage-class.yaml" echo "Installing storage class mapping..."
kubectl apply -f change-storage-class.yaml kubectl apply -f change-storage-class.yaml
# Check if velero CLI is installed and functioning # Check if velero CLI is installed
if ! command -v velero &> /dev/null; then if ! command -v velero &> /dev/null; then
echo "Error: Velero CLI is not installed or not in PATH" echo "Error: Velero CLI is not installed or not in PATH"
echo "Please install the Velero CLI by following instructions at: https://velero.io/docs/main/basic-install/" echo "Install from: https://velero.io/docs/main/basic-install/"
exit 1 exit 1
fi fi
# Verify Velero is connected to the storage location # Verify Velero is connected to the storage location
echo "Checking Velero backup storage location..." echo "Checking Velero backup storage location..."
kubectl -n velero get backupstoragelocation default -o jsonpath='{.status.phase}' | grep -q "Available" || { BSL_AVAILABLE=false
echo "Warning: Backup storage location is not available, it may take a few minutes to initialize" for i in {1..32}; do
echo "Storage location status:" BSL_STATUS=$(kubectl -n velero get backupstoragelocation default -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown")
kubectl -n velero get backupstoragelocation default -o yaml if [[ "$BSL_STATUS" == "Available" ]]; then
echo "Backup storage location is available"
# Wait a bit longer for the location to become available BSL_AVAILABLE=true
echo "Waiting 30 seconds for backup storage location to become available..." break
sleep 30
}
# 4. List backups
echo "Checking for available backups..."
if ! velero_output=$(velero backup get 2>&1); then
echo "Error listing backups: $velero_output"
# Try to verify S3 connectivity
echo "Checking S3 connectivity..."
# Install AWS CLI if needed
if ! command -v aws &> /dev/null; then
echo "AWS CLI not found, skipping S3 connectivity test"
else
# Create temporary AWS profile
mkdir -p ~/.aws
cat > ~/.aws/credentials <<EOF
[velero-test]
aws_access_key_id=$S3_ACCESS_KEY
aws_secret_access_key=$S3_SECRET_KEY
EOF
cat > ~/.aws/config <<EOF
[profile velero-test]
region=$S3_REGION
output=json
EOF
# Try to list bucket contents
echo "Testing S3 connection to bucket $S3_BUCKET..."
if aws --profile velero-test --endpoint-url $S3_URL s3 ls s3://$S3_BUCKET/; then
echo "S3 connection successful, but Velero couldn't list backups."
echo "This might indicate that there are no backups in the bucket or the backups are in a different format than expected."
else
echo "Error connecting to S3. Please check your credentials and connectivity to $S3_URL"
fi fi
echo "Waiting for backup storage location... ($i/32) - Status: $BSL_STATUS"
# Cleanup AWS files if [[ "$DEBUG" == "true" ]]; then
rm ~/.aws/credentials ~/.aws/config echo "[DEBUG] BackupStorageLocation details:"
kubectl -n velero get backupstoragelocation default -o yaml 2>/dev/null || true
fi fi
sleep 5
done
if [[ -z "$BACKUP_NAME" ]]; then if [[ "$BSL_AVAILABLE" != "true" ]]; then
echo "No backup name specified and no backups found. Exiting." echo ""
echo "ERROR: Backup storage location did not become available"
echo ""
echo "This usually means the k3d cluster cannot reach your MinIO server at $S3_URL"
echo ""
echo "Possible solutions:"
echo " 1. Use host.k3d.internal instead of the IP address:"
echo " --s3-url http://host.k3d.internal:9000"
echo ""
echo " 2. If MinIO is on your host machine, ensure it's listening on all interfaces"
echo ""
echo " 3. Check BackupStorageLocation status:"
kubectl -n velero get backupstoragelocation default -o yaml 2>/dev/null || true
exit 1 exit 1
fi fi
else
# 4. Wait for backups to sync from S3
echo "Waiting for Velero to sync backups from S3..."
BACKUP_FOUND=false
for i in {1..32}; do
velero_output=$(velero backup get 2>&1) || true
if echo "$velero_output" | grep -q "^$BACKUP_NAME "; then
BACKUP_FOUND=true
break
fi
echo "Waiting for backup list to sync... ($i/32)"
sleep 5
done
echo "Available backups:" echo "Available backups:"
echo "$velero_output" velero backup get 2>&1 || true
echo ""
if [[ -z "$BACKUP_NAME" ]]; then if [[ -z "$BACKUP_NAME" ]]; then
echo "Specify a backup name using --backup-name and run again." echo "Specify a backup name using --backup-name and run again."
exit 0 exit 0
fi fi
if [[ "$BACKUP_FOUND" != "true" ]]; then
echo "ERROR: Backup '$BACKUP_NAME' not found after waiting"
echo ""
echo "This could mean:"
echo " 1. The backup name is incorrect"
echo " 2. The backup doesn't exist in the S3 bucket"
echo " 3. Velero needs more time to sync"
echo ""
echo "You can check available backups on your production cluster:"
echo " velero backup get"
exit 1
fi fi
echo "Found backup: $BACKUP_NAME"
# 5. Create test namespace if needed # 5. Create test namespace if needed
if [[ "$TEST_NAMESPACE" != "default" ]]; then if [[ "$TEST_NAMESPACE" != "default" ]]; then
kubectl create namespace $TEST_NAMESPACE || true kubectl create namespace "$TEST_NAMESPACE" 2>/dev/null || true
fi fi
# 6. Restore backup # 6. Restore backup (excluding auth proxy and certificates)
echo "Restoring backup: $BACKUP_NAME" echo "Restoring backup: $BACKUP_NAME"
echo " Excluding: sol-auth resources, certificates"
RESTORE_NAME="test-restore-$(date +%s)"
if [[ -z "$ORIGINAL_NAMESPACE" ]]; then if [[ -z "$ORIGINAL_NAMESPACE" ]]; then
# If no namespace specified, restore everything velero restore create "$RESTORE_NAME" \
velero restore create --from-backup $BACKUP_NAME \ --from-backup "$BACKUP_NAME" \
--exclude-resources certificates.cert-manager.io \
--wait --wait
else else
# Restore specific namespace with mapping velero restore create "$RESTORE_NAME" \
velero restore create --from-backup $BACKUP_NAME \ --from-backup "$BACKUP_NAME" \
--namespace-mappings $ORIGINAL_NAMESPACE:$TEST_NAMESPACE \ --namespace-mappings "$ORIGINAL_NAMESPACE:$TEST_NAMESPACE" \
--include-namespaces $ORIGINAL_NAMESPACE \ --include-namespaces "$ORIGINAL_NAMESPACE" \
--exclude-namespaces kube-system,kube-public,kube-node-lease,velero \ --exclude-namespaces kube-system,kube-public,kube-node-lease,velero \
--exclude-resources certificates.cert-manager.io \ --exclude-resources certificates.cert-manager.io \
--wait --wait
@@ -277,105 +567,42 @@ fi
# 7. Check restore status # 7. Check restore status
echo "Checking restore status:" echo "Checking restore status:"
velero restore get velero restore describe "$RESTORE_NAME"
# Wait for pods to be ready # 8. Delete sol-auth resources (they were restored from backup)
echo "Removing sol-auth resources..."
kubectl delete deployment -n "$TEST_NAMESPACE" sol-auth 2>/dev/null || true
kubectl delete service -n "$TEST_NAMESPACE" "$AUTH_PROXY_SERVICE" 2>/dev/null || true
kubectl delete configmap -n "$TEST_NAMESPACE" httpd-conf 2>/dev/null || true
# Wait for sol-auth pod to terminate
kubectl wait --for=delete pod -l app=sol-auth -n "$TEST_NAMESPACE" --timeout=60s 2>/dev/null || true
# 9. Wait for pods to be ready
echo "Waiting for pods to be ready..." echo "Waiting for pods to be ready..."
kubectl wait --for=condition=ready pod --all -n $TEST_NAMESPACE --timeout=600s || true kubectl wait --for=condition=ready pod --all -n "$TEST_NAMESPACE" --timeout=600s 2>/dev/null || true
# 8. Run validation script (create it first if it doesn't exist) # 10. Rewrite ingress configuration for local testing
# if [[ ! -f ./validate-restore.sh ]]; then if [[ "$SKIP_INGRESS_SETUP" != "true" ]]; then
# echo "Creating validation script..." bypass_auth_proxy "$TEST_NAMESPACE"
# cat > validate-restore.sh <<'EOF' rewrite_ingress_hosts "$TEST_NAMESPACE"
# #!/bin/bash fi
# # validate-restore.sh - Script to validate Velero backup restoration
# set -e # 11. Generate /etc/hosts entries if requested
# echo "Starting validation of restored resources..." if [[ "$GENERATE_HOSTS" == "true" ]]; then
generate_hosts_entries "$TEST_NAMESPACE"
# # Get namespace from command line or use default fi
# NAMESPACE="${1:-default}"
# # Check all deployments
# echo "Checking deployments..."
# deployments=$(kubectl get deployment -n $NAMESPACE -o name)
# if [[ -z "$deployments" ]]; then
# echo "❌ No deployments found in namespace $NAMESPACE"
# else
# echo "Found deployments: $deployments"
# # Check if pods are running for each deployment
# for deployment in $deployments; do
# name=$(echo $deployment | cut -d'/' -f2)
# echo -n "Checking deployment $name: "
# # Check if pods are running
# ready_replicas=$(kubectl get deployment -n $NAMESPACE $name -o jsonpath='{.status.readyReplicas}')
# if [[ -n "$ready_replicas" && "$ready_replicas" != "0" ]]; then
# echo "✅ $ready_replicas pods ready"
# else
# echo "❌ No pods running"
# fi
# done
# fi
# # Check all services
# echo "Checking services..."
# services=$(kubectl get service -n $NAMESPACE -o name | grep -v "kubernetes")
# if [[ -z "$services" ]]; then
# echo "❌ No services found in namespace $NAMESPACE"
# else
# echo "Found services: $services"
# # Check if services have endpoints
# for service in $services; do
# name=$(echo $service | cut -d'/' -f2)
# echo -n "Checking service $name: "
# # Check if service has endpoints
# endpoints=$(kubectl get endpoints -n $NAMESPACE $name -o jsonpath='{.subsets[*].addresses[*].ip}')
# if [ -n "$endpoints" ]; then
# echo "✅ Has endpoints"
# else
# echo "❌ No endpoints"
# fi
# done
# fi
# # Check PVCs
# echo "Checking PVCs..."
# pvcs=$(kubectl get pvc -n $NAMESPACE -o name)
# if [[ -z "$pvcs" ]]; then
# echo "No PVCs found in namespace $NAMESPACE"
# else
# pvc_count=$(echo "$pvcs" | wc -l)
# echo "Found $pvc_count PVCs"
# # Check if PVCs are bound
# bound_count=$(kubectl get pvc -n $NAMESPACE -o jsonpath='{.items[?(@.status.phase=="Bound")].metadata.name}' | wc -w)
# echo "$bound_count/$pvc_count PVCs are bound"
# if [ "$bound_count" -ne "$pvc_count" ]; then
# echo "❌ Not all PVCs are bound"
# else
# echo "✅ All PVCs are bound"
# fi
# fi
# echo "Validation complete!"
# exit 0
# EOF
# chmod +x validate-restore.sh
# fi
# 12. Run validation
echo "Running validation tests..." echo "Running validation tests..."
./validate-restore.sh $TEST_NAMESPACE ./validate-restore.sh "$TEST_NAMESPACE"
echo "" echo ""
echo "=== Backup test process completed ===" echo "=== Backup test process completed ==="
echo "The cluster will be deleted when you close this terminal or press Ctrl+C"
echo "To access the cluster manually, use: kubectl config use-context k3d-backup-test"
echo "" echo ""
echo "Cluster: k3d-backup-test"
echo "Namespace: $TEST_NAMESPACE"
echo "Context: kubectl config use-context k3d-backup-test"
echo ""
echo "The cluster will be deleted when you press Enter or Ctrl+C"
# Keep the script running until user terminates it
read -p "Press Enter to clean up and exit..." read -p "Press Enter to clean up and exit..."