Compare commits

...
Author SHA1 Message Date
Ruben Hensen c298ad8746 immich: remove the PostgreSQL 14 cluster 2026-10-28 15:40:14 +01:00
Ruben Hensen fe105a4fa7 Merge pull request #60 from rubenhensen/mariadb-11.4
MariaDB: newest 11.4 patch, and keep Renovate on 11.4
2026-10-28 15:36:16 +01:00
Ruben Hensen 1afd4b6bbd mariadb: newest 11.4 patch, and keep Renovate on 11.4 2026-10-28 15:31:40 +01:00
Ruben Hensen b4663216db Merge pull request #59 from rubenhensen/immich-share-people
immich: share the group account's people with everyone
2026-10-28 15:17:06 +01:00
Ruben Hensen 955c22c665 Merge pull request #58 from rubenhensen/memory-limits
Memory limits from a day of measurements
2026-10-28 15:16:55 +01:00
Ruben Hensen 1f894996e6 docs: memory limits on the roadmap 2026-10-28 14:58:03 +01:00
Ruben Hensen c380bfc0cf Memory limits from a day of measurements 2026-10-28 14:57:34 +01:00
22 changed files with 100 additions and 48 deletions

No files matched your search

+4
View File
@@ -103,6 +103,8 @@ spec:
requests:
cpu: 70m
memory: 560Mi
limits:
memory: 1536Mi
volumes:
- name: patched-ca-bundle
configMap:
@@ -192,6 +194,8 @@ spec:
requests:
cpu: 100m
memory: 336Mi
limits:
memory: 1Gi
volumes:
- name: patched-ca-bundle
configMap:
+6
View File
@@ -38,16 +38,22 @@ spec:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
cainjector:
resources:
requests:
cpu: 10m
memory: 80Mi
limits:
memory: 256Mi
webhook:
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
syncPolicy:
syncOptions:
- CreateNamespace=true
+2
View File
@@ -28,6 +28,8 @@ spec:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
destination:
server: https://kubernetes.default.svc
namespace: cnpg-system
@@ -24,16 +24,22 @@ spec:
requests:
cpu: 20m
memory: 64Mi
limits:
memory: 128Mi
certController:
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 128Mi
webhook:
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
destination:
server: https://kubernetes.default.svc
namespace: external-secrets
@@ -21,6 +21,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 128Mi
destination:
server: https://kubernetes.default.svc
namespace: cnpg-system
+16 -2
View File
@@ -1,6 +1,6 @@
# Requests from what the components used on 2026-10-25 (kubectl top every
# 5 minutes; highest value plus 10%). No limits until there are daytime
# numbers, see docs/roadmap.md.
# 5 minutes; highest value plus 10%). Memory limits at about twice the
# highest value seen then or on 2026-10-27, rounded up; no CPU limits.
apiVersion: apps/v1
kind: StatefulSet
metadata:
@@ -14,6 +14,8 @@ spec:
requests:
cpu: 150m
memory: 704Mi
limits:
memory: 2560Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -28,6 +30,8 @@ spec:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 512Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -42,6 +46,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 256Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -56,6 +62,8 @@ spec:
requests:
cpu: 20m
memory: 96Mi
limits:
memory: 256Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -70,6 +78,8 @@ spec:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -84,6 +94,8 @@ spec:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
---
apiVersion: apps/v1
kind: Deployment
@@ -98,3 +110,5 @@ spec:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
+2
View File
@@ -67,6 +67,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 512Mi
env:
# Newer boto3 adds checksums that S3-compatible stores can reject;
# the plugin's docs suggest sending them only when required.
+2
View File
@@ -18,6 +18,8 @@ spec:
requests:
cpu: 50m
memory: 256Mi
limits:
memory: 1Gi
bootstrap:
initdb:
database: authentik
+15 -9
View File
@@ -26,11 +26,12 @@ docs/longhorn-second-ssd.md:
- [x] Read `~/iosample.log` on opti1: the worst stretch was not the
backups but 8.5 hours of Longhorn merging immich-claim's deleted
snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md
- [ ] From 2026-10-27, if immich runs without problems: delete
`immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB),
the Longhorn snapshots and backups `immich-pg14-pre-vchord` and
`immich-pg17-pre-v3`, and both dumps in `~` on opti1. PostgreSQL 14
reaches end of life on 2026-11-12
- [x] From 2026-10-27, if immich runs without problems: delete
`immich/postgresql-cluster.yaml` (the PG14 cluster, 20 GiB), the
Longhorn snapshots and backups `immich-pg14-pre-vchord` and
`immich-pg17-pre-v3`, and both dumps in `~` on opti1. Done 2026-10-28
(PR #61), with the PG14 volume's other Longhorn backups.
`immich-pg17-pre-v3` only ever was a snapshot
- [ ] The upgrade's dqlite and backend copies in `/root/` on opti1
(`backend-*.tar`, `dqlite-pre-*`)
- [ ] Log in to immich as a jeugdlid (quota 0)
@@ -341,8 +342,13 @@ Counted 2026-10-25 over the 73 running containers:
requested. Still without requests: Longhorn (22 containers, its chart),
the Vault servers, the LDAP outpost, immich's PG14 cluster and three in
kube-system
- [ ] Memory limits, once there are daytime numbers: the night ones miss
immich's machine learning at work (it idles at 234Mi)
- [x] Memory limits: PR #58, for everything that got requests in #33.
About twice the highest memory from `kubectl top` every 5 minutes from
2026-10-26 22:29 to 2026-10-27 21:55 UTC or the night of #33, rounded
up; no CPU limits. No OOM kills that day. immich's machine learning
sat idle all day (240Mi), so on 2026-10-28 a test image went through
CLIP, face recognition and OCR inside the container: 884Mi with all
three loaded, limit 2Gi
- [x] WordPress (scoutingmbg.nl) was OOMKilled at 08:27:59 UTC on
2026-10-25, its third restart: a crawler asked for 88 pages and assets
within a few seconds (normally ~6 a minute), Apache started a PHP worker
@@ -503,7 +509,7 @@ critical / high:
| Image | Crit | High | What to do |
|---|---|---|---|
| tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; goes with the cleanup from 2026-10-27 |
| tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; deleted 2026-10-28 |
| cloudnative-pg/postgresql:17.4 | 27 | 260 | authentik's database, a Debian 11 image. Renovate now offers 17.7; a `17.x-standard-trixie` image is more current |
| tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag |
| longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below |
@@ -514,7 +520,7 @@ critical / high:
| rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image |
| coredns/coredns:1.10.1 | 5 | 70 | see below |
| dexidp/dex:v2.45.1, argocd v3.5.6, redis 8.2.3 | 4, 1, 2 | | come with Argo CD |
| mariadb:11.4.5 | 4 | 65 | the newest 11.4 patch; Renovate now covers it |
| mariadb:11.4.5 | 4 | 65 | 11.4.13 since PR #60; Renovate stays on 11.4 (it offered 11.8, PR #42) |
| calico v3.29.3 | 2-3 | ~50 | comes with microk8s |
| goauthentik ldap / server 2026.8.5 | 3, 1 | 74, 65 | the next authentik patch |
+4 -3
View File
@@ -169,8 +169,9 @@ pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:
v2.4.1 refused to start (it only accepts VectorChord <0.6); v2.5.0 and up
accept <2, so immich went to v2.7.5, the last 2.x. It rebuilt both
indexes as `vchordrq` in about 1.5 minutes
- [ ] After a few days without problems: delete the old cluster
- [x] After a few days without problems: delete the old cluster
(`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump.
Done 2026-10-28 (PR #61), with the PG14 volume's Longhorn backups.
Rolling back means reverting the old volume to the snapshot, since its
embeddings are now `real[]` without pgvecto.rs
- [x] immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on
@@ -189,8 +190,8 @@ pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:
(v3.3.0), for faster ML with less memory; numerically the same models,
so no jobs to rerun. The model cache is an emptyDir, so the one-off
preparation repeats on every restart
- [ ] After a few days without problems: delete the `immich-pg17-pre-v3`
snapshot and dump together with the PG14 leftovers above
- [x] After a few days without problems: delete the `immich-pg17-pre-v3`
snapshot and dump together with the PG14 leftovers above (2026-10-28)
## k8s-dqlite write stalls (found 2026-10-22)
+2
View File
@@ -83,6 +83,8 @@ spec:
requests:
cpu: 50m
memory: 176Mi
limits:
memory: 512Mi
# TCP, because mod_auth_openidc answers 401 to anything that isn't a
# browser.
readinessProbe:
+2
View File
@@ -54,6 +54,8 @@ spec:
requests:
cpu: 20m
memory: 336Mi
limits:
memory: 1Gi
# /api/healthz checks the database and the cache.
readinessProbe:
httpGet:
+8 -2
View File
@@ -67,6 +67,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 256Mi
image:
repository: docker.io/valkey/valkey
tag: 9.1.2-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
@@ -93,6 +95,8 @@ spec:
requests:
cpu: 100m
memory: 1280Mi
limits:
memory: 3Gi
# Node trusts its built-in roots, which still include the
# ones above the compromised eMudhra intermediates. Use
# patched-ca-bundle through OpenSSL until Node ships a fix.
@@ -117,12 +121,14 @@ spec:
image:
repository: ghcr.io/immich-app/immich-machine-learning
pullPolicy: IfNotPresent
# Idle at night (234Mi); the models take over a GiB when
# it works.
# Idle at 240Mi; 884Mi with CLIP, face recognition and
# OCR loaded (2026-10-28, a test image through each).
resources:
requests:
cpu: 50m
memory: 1Gi
limits:
memory: 2Gi
env:
TRANSFORMERS_CACHE: /cache
HF_XET_CACHE: /cache/huggingface-xet
+2
View File
@@ -71,6 +71,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 512Mi
env:
# Newer boto3 adds checksums that S3-compatible stores can reject;
# the plugin's docs suggest sending them only when required.
+6 -1
View File
@@ -13,6 +13,8 @@ spec:
requests:
cpu: 20m
memory: 320Mi
limits:
memory: 1Gi
postgresql:
shared_preload_libraries:
@@ -27,7 +29,10 @@ spec:
# Filled once from the PostgreSQL 14 cluster (immich-postgres) after its
# embedding columns were turned into real[] and pgvecto.rs was dropped,
# see docs/upgrade-k8s-1.33.md.
# see docs/upgrade-k8s-1.33.md. That cluster is gone since 2026-10-28, so
# creating this Cluster again fails here, on purpose: an empty database
# would start immich with an empty library. Restore from Garage instead
# (bootstrap.recovery, see postgresql-backup.yaml).
bootstrap:
initdb:
database: immich
-28
View File
@@ -1,28 +0,0 @@
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: immich-postgres
namespace: immich
spec:
instances: 1
# Use TensorChord's CNPG-compatible image with pgvecto.rs v0.2.1
imageName: ghcr.io/tensorchord/cloudnative-pgvecto.rs:14-v0.2.1
postgresql:
shared_preload_libraries:
- "vectors.so"
bootstrap:
initdb:
database: immich
owner: immich
postInitSQL:
- CREATE EXTENSION IF NOT EXISTS vectors;
- CREATE EXTENSION IF NOT EXISTS cube;
- CREATE EXTENSION IF NOT EXISTS earthdistance CASCADE;
- ALTER USER immich WITH SUPERUSER;
storage:
size: 20Gi
storageClass: longhorn
+6
View File
@@ -20,6 +20,8 @@ spec:
requests:
cpu: 10m
memory: 48Mi
limits:
memory: 128Mi
metrics:
enabled: true
serviceMonitor:
@@ -29,6 +31,8 @@ spec:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
serviceMonitor:
enabled: false
certController:
@@ -36,6 +40,8 @@ spec:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
serviceMonitor:
enabled: false
syncPolicy:
+1 -1
View File
@@ -8,7 +8,7 @@ spec:
name: mbgwp-mariadb-credentials
key: root-password
image: mariadb:11.4.5
image: mariadb:11.4.13
imagePullPolicy: IfNotPresent
port: 3306
+8 -2
View File
@@ -82,14 +82,20 @@
"description": "A PostgreSQL major upgrade needs pg_upgrade or a dump, not a tag bump",
"matchDepNames": [
"ghcr.io/cloudnative-pg/postgresql",
"ghcr.io/tensorchord/cloudnative-vectorchord",
"ghcr.io/tensorchord/cloudnative-pgvecto.rs"
"ghcr.io/tensorchord/cloudnative-vectorchord"
],
"matchUpdateTypes": [
"major"
],
"enabled": false
},
{
"description": "MariaDB goes from one LTS to the next by hand, after a dump: there is no way back. 11.4 has community support until May 2029, longer than 11.8 (June 2028); the next step is 12.3 (June 2029). Within 11.4 the patches come through",
"matchDepNames": [
"mariadb"
],
"allowedVersions": "<11.5"
},
{
"description": "Patch updates of apps that hold little state and are easy to roll back, and of CI's tools, merge themselves once CI passes: three days after the release, in the evening or at the weekend, when the cluster can be reached. Not the databases, operators, Vault, authentik, Longhorn, Velero, immich, Gitea or WordPress. Renovate merges itself; the repo has no branch protection for GitHub's auto-merge to wait on",
"matchFileNames": [
+2
View File
@@ -60,6 +60,8 @@ spec:
requests:
cpu: 10m
memory: 160Mi
limits:
memory: 512Mi
readinessProbe:
httpGet:
path: /login
+2
View File
@@ -7,6 +7,8 @@ injector:
requests:
cpu: 10m
memory: 32Mi
limits:
memory: 128Mi
# The chart's default anti-affinity forbids two injector pods on one
# node, so on this single node a rolling update can never start the new
# pod while the old one runs. Recreate removes the old pod first, which
+2
View File
@@ -71,6 +71,8 @@ spec:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 128Mi
# 512Mi got the server OOMKilled 314 times, including halfway
# through a backup of ~14.7k items.
resources: