Compare commits

..
Author SHA1 Message Date
Ruben Hensen 73eb276713 mbgwp: run mariadb-upgrade when MariaDB starts on 11.8
11.4 to 11.8 is a major MariaDB upgrade. MARIADB_AUTO_UPGRADE has the
image's entrypoint migrate the datadir on start, as operator 26.10.1's
updateStrategy.mariadbAutoUpgradeEnabled would; set as env because CI's
schema for MariaDB doesn't know that field yet.
2026-10-27 10:37:57 +01:00
renovate[bot] 57b24b5b5c chore(deps): update mariadb docker tag to v11.8.9 2026-10-26 21:54:14 +00:00
Ruben Hensen 648f99c505 Merge pull request #35 from rubenhensen/velero-garage
Velero backs up to Garage instead of MinIO
2026-10-26 22:52:42 +01:00
Ruben Hensen 20bd3a4bca Merge pull request #39 from rubenhensen/immich-ml-v2
immich: the optimized ML models, cached on a volume
2026-10-25 17:48:40 +01:00
Ruben Hensen 6cba1a9363 immich: the optimized ML models, cached on a volume 2026-10-25 17:46:03 +01:00
Ruben Hensen acfce6645e velero: skip PVCs in Longhorn's no-backup group 2026-10-25 17:46:03 +01:00
Ruben Hensen 0de86dd65c Merge pull request #36 from rubenhensen/renovate-re2
Renovate: skip the non-app folders without a lookahead
2026-10-25 17:41:32 +01:00
Ruben Hensen d7ffdc05f6 docs: Renovate rejected the lookahead (#32) 2026-10-25 17:33:39 +01:00
Ruben Hensen c0f8034e14 renovate: skip the non-app folders through ignorePaths, not a lookahead 2026-10-25 17:33:39 +01:00
Ruben Hensen a15a4f7c75 Merge pull request #34 from rubenhensen/velero-garage-secret
Garage on the NAS, and Velero's key for it
2026-10-25 17:23:33 +01:00
Ruben Hensen bdafcaf1e0 backup-testing: read Velero's key for Garage 2026-10-25 17:10:32 +01:00
Ruben Hensen 03c3196242 velero: back up to Garage instead of MinIO 2026-10-25 17:10:32 +01:00
Ruben Hensen b072099cc1 nas: Garage keeps its metadata in SQLite 2026-10-25 17:08:13 +01:00
Ruben Hensen e10ca337f1 velero: a key for Garage, from Vault 2026-10-25 16:28:43 +01:00
Ruben Hensen c72b08867f nas: Garage, which takes over from MinIO 2026-10-25 16:28:43 +01:00
12 changed files with 181 additions and 37 deletions

No files matched your search

+8 -8
View File
@@ -3,7 +3,7 @@
# k3s in Docker, to prove the backups can be restored. Nothing on the NAS
# changes:
# - one backup (backups/<name>/) and the kopia repositories of the chosen
# namespaces (kopia/<namespace>/) are copied read-only from MinIO with
# namespaces (kopia/<namespace>/) are copied read-only from the NAS with
# rclone into a Docker volume. A second Velero on the real bucket could
# run kopia maintenance on the same repositories as the cluster's own
# - `rclone serve s3` serves that copy to a Velero 1.18.2 in k3s, with a
@@ -19,7 +19,7 @@
# Defaults: the newest Completed daily-backup, and ntfy freshrss
# actual-budget. Not gatus: a restored Gatus would start sending alerts.
#
# It reads Velero's MinIO key from the cluster (velero/cloud-credentials)
# It reads Velero's S3 key from the cluster (velero/velero-s3-credentials)
# into a temporary directory, and the copied backup holds every Secret of
# the cluster; all of it is deleted at the end. KEEP=1 keeps the test
# cluster (kubeconfig in the printed directory) until you run it with
@@ -71,14 +71,14 @@ wait_for() {
done
}
# Endpoint, region and bucket as Velero itself uses them. MinIO answers a
# request signed for the wrong region (rclone defaults to us-east-1) with a
# bare 400 on HEAD.
# Endpoint, region and bucket as Velero itself uses them. The NAS rejects a
# request signed for the wrong region (rclone defaults to us-east-1); MinIO
# did so with a bare 400 on HEAD.
if [[ -n ${FROM_VOLUME:-} ]]; then
BUCKET=${BUCKET:-velero-backups}
else
bsl=$(pk -n velero get backupstoragelocations default -o json)
MINIO=${MINIO:-$(jq -r '.spec.config.s3Url' <<<"$bsl")}
S3URL=${S3URL:-$(jq -r '.spec.config.s3Url' <<<"$bsl")}
REGION=${REGION:-$(jq -r '.spec.config.region // "us-east-1"' <<<"$bsl")}
BUCKET=${BUCKET:-$(jq -r '.spec.objectStorage.bucket' <<<"$bsl")}
fi
@@ -101,10 +101,10 @@ if [[ -n ${FROM_VOLUME:-} ]]; then
docker run --rm -v "$FROM_VOLUME:/from:ro" -v "$VOL:/data" alpine:3.22 cp -a /from/. /data/
else
echo "== copy from the NAS (read-only)"
(umask 077; pk -n velero get secret cloud-credentials -o jsonpath='{.data.cloud}' | base64 -d |
(umask 077; pk -n velero get secret velero-s3-credentials -o jsonpath='{.data.cloud}' | base64 -d |
awk -F' *= *' '/aws_access_key_id/ {print "RCLONE_CONFIG_NAS_ACCESS_KEY_ID="$2}
/aws_secret_access_key/ {print "RCLONE_CONFIG_NAS_SECRET_ACCESS_KEY="$2}' > "$WORK/nas.env"
printf 'RCLONE_CONFIG_NAS_TYPE=s3\nRCLONE_CONFIG_NAS_PROVIDER=Minio\nRCLONE_CONFIG_NAS_ENDPOINT=%s\nRCLONE_CONFIG_NAS_REGION=%s\n' "$MINIO" "$REGION" >> "$WORK/nas.env")
printf 'RCLONE_CONFIG_NAS_TYPE=s3\nRCLONE_CONFIG_NAS_PROVIDER=Other\nRCLONE_CONFIG_NAS_ENDPOINT=%s\nRCLONE_CONFIG_NAS_REGION=%s\n' "$S3URL" "$REGION" >> "$WORK/nas.env")
copy() { docker run --rm --env-file "$WORK/nas.env" -v "$VOL:/data" "$RCLONE" copy --stats-one-line "nas:$BUCKET/$1" "/data/$BUCKET/$1"; }
copy "backups/$backup"
for ns in "${namespaces[@]}"; do copy "kopia/$ns"; done
+10 -4
View File
@@ -360,10 +360,16 @@ PR before Argo CD applies it.
cover every app folder, a regex manager reads `imageName` in
`*/postgresql-cluster*.yaml`, and major PostgreSQL updates are off
(those need pg_upgrade or a dump). Expect a batch of new PRs
(2026-10-25; `renovate-config-validator` passes). They only come once
fewer than 10 Renovate PRs are open (`config:recommended`'s
`prConcurrentLimit`); until then they wait under "Rate-Limited" on the
Dependency Dashboard (#2)
(2026-10-25). They only come once fewer than 10 Renovate PRs are open
(`config:recommended`'s `prConcurrentLimit`); until then they wait
under "Rate-Limited" on the Dependency Dashboard (#2)
- The other folders were left out with a `(?!…)` lookahead. Renovate
compiles these patterns with RE2, which has no lookahead, so it
rejected the config and stopped opening PRs (#32). Each manager now
takes every folder and skips the others through its own
`ignorePaths`. Renovate 44.181.6 with RE2 loaded fails the old config
with #32's error and passes the new one, and its matcher picks the
same 166 files as the old pattern did
- [x] Merged 2026-10-25 and checked after the sync: #28 cert-manager
v1.21.3 (security release; all three pods on v1.21.3), #24
external-snapshotter v8.7.0 (its manifests ship the v8.6.0 image, as
+14 -3
View File
@@ -127,6 +127,9 @@ spec:
TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet
MPLCONFIGDIR: /cache/matplotlib-config
# The optimized models of v3.3.0: faster, less memory,
# and the same results, so no jobs to run again.
MACHINE_LEARNING_MODEL_REVISION: v2
# Trust store without the compromised eMudhra
# intermediates; requests would otherwise use certifi.
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
@@ -139,11 +142,19 @@ spec:
globalMounts:
- path: /etc/ssl/certs
readOnly: true
# The models and their one-off preparation, kept across restarts.
# Downloaded again if lost, so no backups: the no-backup group
# keeps it out of Longhorn's default one (backup-daily), and
# Velero skips that label too (velero/skip-emptydir-policy.yaml).
cache:
enabled: true
# Optional: to avoid downloading the ML models every start, use
# a persistentVolumeClaim with size, accessMode and storageClass
type: emptyDir
type: persistentVolumeClaim
storageClass: longhorn
accessMode: ReadWriteOnce
size: 10Gi
labels:
recurring-job.longhorn.io/source: enabled
recurring-job-group.longhorn.io/no-backup: enabled
destination:
server: https://kubernetes.default.svc
+10 -1
View File
@@ -8,8 +8,17 @@ spec:
name: mbgwp-mariadb-credentials
key: root-password
image: mariadb:11.4.5
image: mariadb:11.8.9
imagePullPolicy: IfNotPresent
# Every X.Y is a major MariaDB release (11.4 to 11.8 too) and needs
# mariadb-upgrade on the datadir, or queries on mysql.proc fail. With this
# the image's entrypoint runs it on start; a no-op when there is nothing
# to migrate, so it stays. Operator 26.10.1 sets the same variable through
# updateStrategy.mariadbAutoUpgradeEnabled, but CI's MariaDB schema (datree
# catalog) doesn't know that field yet.
env:
- name: MARIADB_AUTO_UPGRADE
value: "true"
port: 3306
+35
View File
@@ -0,0 +1,35 @@
# Garage on the NAS
S3 storage for Velero, on the Synology (192.168.1.99) in Docker, outside
the cluster. Replaced MinIO, which is no longer maintained. One node, data
in `/volume1/docker/garage/{meta,data}`. S3 API on port 3900, admin API on
3903, both LAN only. Set up 2026-10-25 with Garage v2.5.0.
The NAS is reached as `ssh -p 34000 rubenhensen@192.168.1.99`; docker is
in `/usr/local/bin`, which a non-interactive SSH session doesn't have on
its PATH.
## Setup
`garage.toml` here has placeholders for the two secrets; on the NAS they
were filled in once with `openssl rand -hex 32` (`rpc_secret`) and
`openssl rand -base64 32` (`admin_token`), and only exist there.
```
cd /volume1/docker/garage
docker compose up -d
g="docker exec garage /garage"
$g layout assign -z nas -c 800G <node id from $g status>
$g layout apply --version 1
$g bucket create velero-backups
$g key create velero
$g bucket allow --read --write --key velero velero-backups
```
Velero's key is in Vault as `kv/velero-s3` (`access_key_id`,
`secret_access_key`) and reaches Velero through
`velero/external-secret.yaml`. Without the cluster, `$g key info velero
--show-secret` on the NAS shows it again.
Renovate updates the image tag in `compose.yaml`, but nothing deploys it:
after merging, `docker compose pull && docker compose up -d` on the NAS.
+12
View File
@@ -0,0 +1,12 @@
services:
garage:
image: dxflrs/garage:v2.5.0
container_name: garage
restart: unless-stopped
ports:
- "3900:3900" # S3 API
- "3903:3903" # admin API
volumes:
- ./garage.toml:/etc/garage.toml:ro
- ./meta:/var/lib/garage/meta
- ./data:/var/lib/garage/data
+29
View File
@@ -0,0 +1,29 @@
# One node on the Synology. Velero's kopia data is encrypted, so
# compression would only cost CPU.
#
# SQLite rather than the default LMDB: LMDB can be corrupted by an unclean
# shutdown, and a single node has no other node to repair from. With
# metadata_fsync SQLite runs WAL with synchronous=NORMAL, which only syncs
# at checkpoints; LMDB with metadata_fsync synced every transaction, ~270
# writes/s on the NAS's two disks with Garage idle. Converted 2026-10-25
# with `garage convert-db`.
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
metadata_fsync = true
data_fsync = true
metadata_auto_snapshot_interval = "6h"
replication_factor = 1
compression_level = "none"
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "@RPC_SECRET@"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
[admin]
api_bind_addr = "[::]:3903"
admin_token = "@ADMIN_TOKEN@"
+18 -2
View File
@@ -5,8 +5,16 @@
"customManagers:githubActionsVersions"
],
"argocd": {
"description": "Every app folder; Renovate's regexes are RE2, which has no lookahead, so the other folders are skipped through ignorePaths",
"managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/"
"/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
]
},
"helm-values": {
@@ -15,8 +23,16 @@
]
},
"kubernetes": {
"description": "Every app folder, as for argocd",
"managerFilePatterns": [
"/^(?!(nix-infra-machine|backup-testing|docs|tools|\\.github)/)[^/]+/.+\\.yaml$/"
"/^[^/]+/.+\\.yaml$/"
],
"ignorePaths": [
"nix-infra-machine/**",
"backup-testing/**",
"docs/**",
"tools/**",
".github/**"
]
},
"customManagers": [
+30
View File
@@ -0,0 +1,30 @@
# Velero's key for its bucket in Garage on the NAS (nas/garage), in the
# AWS credentials file format the aws plugin and kopia read.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: velero-s3-credentials
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: velero-s3-credentials
creationPolicy: Owner
template:
engineVersion: v2
data:
cloud: |
[default]
aws_access_key_id={{ .access_key_id }}
aws_secret_access_key={{ .secret_access_key }}
data:
- secretKey: access_key_id
remoteRef:
key: kv/velero-s3
property: access_key_id
- secretKey: secret_access_key
remoteRef:
key: kv/velero-s3
property: secret_access_key
-10
View File
@@ -1,10 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: nas-minio-external
spec:
type: ExternalName
externalName: 192.168.1.99
ports:
- port: 9000
protocol: TCP
+7
View File
@@ -1,6 +1,8 @@
# Volume policy for the backup schedules. emptyDir volumes hold nothing
# worth keeping, and backing them up fails when the pod is gone before its
# volume is read, as happens with Velero's own repository maintenance jobs.
# PVCs in Longhorn's no-backup group (caches, such as immich's ML models)
# are skipped too.
apiVersion: v1
kind: ConfigMap
metadata:
@@ -14,3 +16,8 @@ data:
- emptyDir
action:
type: skip
- conditions:
pvcLabels:
recurring-job-group.longhorn.io/no-backup: enabled
action:
type: skip
+8 -9
View File
@@ -27,20 +27,22 @@ spec:
- mountPath: /target
name: plugins
configuration:
# Garage on the NAS (nas/garage). Until 2026-10-25 this was MinIO on
# the same NAS; the bucket was copied over with rclone, so the
# existing backups and kopia repositories carried on.
backupStorageLocation:
- name: default
bucket: velero-backups
provider: aws
config:
region: minio
region: garage
s3ForcePathStyle: true
s3Url: http://192.168.1.99:9000
insecureSkipTLSVerify: "true"
s3Url: http://192.168.1.99:3900
volumeSnapshotLocation:
- name: default
provider: aws
config:
region: minio
region: garage
features: EnableCSI
# Kopia maintenance ran every hour for each of the 15 repositories,
# and once a day as a full run of several minutes that writes its
@@ -59,11 +61,8 @@ spec:
memoryRequest: 256Mi
memoryLimit: 2Gi
credentials:
# Use existing secret created by External Secrets Operator
# existingSecret: velero-minio-credentials
# extraEnvVars:
# AWS_SHARED_CREDENTIALS_FILE: /credentials/cloud
existingSecret: cloud-credentials
# From Vault, velero/external-secret.yaml
existingSecret: velero-s3-credentials
deployNodeAgent: true
nodeAgent:
containerSecurityContext: