Compare commits

...
Author SHA1 Message Date
Ruben Hensen c298ad8746 immich: remove the PostgreSQL 14 cluster 2026-10-28 15:40:14 +01:00
Ruben Hensen fe105a4fa7 Merge pull request #60 from rubenhensen/mariadb-11.4
MariaDB: newest 11.4 patch, and keep Renovate on 11.4
2026-10-28 15:36:16 +01:00
Ruben Hensen 1afd4b6bbd mariadb: newest 11.4 patch, and keep Renovate on 11.4 2026-10-28 15:31:40 +01:00
Ruben Hensen b4663216db Merge pull request #59 from rubenhensen/immich-share-people
immich: share the group account's people with everyone
2026-10-28 15:17:06 +01:00
Ruben Hensen 955c22c665 Merge pull request #58 from rubenhensen/memory-limits
Memory limits from a day of measurements
2026-10-28 15:16:55 +01:00
Ruben Hensen f7396b3cb7 immich: share the group account's people with everyone 2026-10-28 15:10:06 +01:00
8 changed files with 98 additions and 46 deletions

No files matched your search

+7 -2
View File
@@ -27,7 +27,9 @@ view, and everything is shared with everyone automatically.
- **immich has no user groups**; albums are shared per user, as viewer or - **immich has no user groups**; albums are shared per user, as viewer or
editor. The owner and editors of an album may add users and change editor. The owner and editors of an album may add users and change
their roles. Search only covers your own photos and those of partners their roles. Search only covers your own photos and those of partners
you show in your timeline. you show in your timeline. People (face recognition) are per user too,
and a partner's library doesn't bring its people along; since 3.3 a
user can share people with the users in their cluster group.
## Design ## Design
@@ -46,7 +48,10 @@ view, and everything is shared with everyone automatically.
every user into every album of the group account (`fotoupload@`), as every user into every album of the group account (`fotoupload@`), as
editor or viewer by quota; the group account's library as a partner of editor or viewer by quota; the group account's library as a partner of
everyone, shown in the timeline; album invitations for albums older than everyone, shown in the timeline; album invitations for albums older than
a week hidden; sessions of OAuth users older than 30 days ended. Albums a week hidden; sessions of OAuth users older than 30 days ended;
everyone in the group account's cluster group with its people shared
(names editable by those who may upload, added 2026-10-28, when nobody
but the group account could search by face). Albums
that kaderleden create get the group account as editor first (in the that kaderleden create get the group account as editor first (in the
database, since the API only lets an album's owner or editors share database, since the API only lets an album's owner or editors share
it), so they go to everyone too; names starting with "Privé"/"Prive" it), so they go to everyone too; names starting with "Privé"/"Prive"
+8 -7
View File
@@ -26,11 +26,12 @@ docs/longhorn-second-ssd.md:
- [x] Read `~/iosample.log` on opti1: the worst stretch was not the - [x] Read `~/iosample.log` on opti1: the worst stretch was not the
backups but 8.5 hours of Longhorn merging immich-claim's deleted backups but 8.5 hours of Longhorn merging immich-claim's deleted
snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md snapshots (2026-10-24 14:30-23:15), see docs/upgrade-k8s-1.33.md
- [ ] From 2026-10-27, if immich runs without problems: delete - [x] From 2026-10-27, if immich runs without problems: delete
`immich/postgresql-cluster.yaml` (the PG14 cluster still runs, 20 GiB), `immich/postgresql-cluster.yaml` (the PG14 cluster, 20 GiB), the
the Longhorn snapshots and backups `immich-pg14-pre-vchord` and Longhorn snapshots and backups `immich-pg14-pre-vchord` and
`immich-pg17-pre-v3`, and both dumps in `~` on opti1. PostgreSQL 14 `immich-pg17-pre-v3`, and both dumps in `~` on opti1. Done 2026-10-28
reaches end of life on 2026-11-12 (PR #61), with the PG14 volume's other Longhorn backups.
`immich-pg17-pre-v3` only ever was a snapshot
- [ ] The upgrade's dqlite and backend copies in `/root/` on opti1 - [ ] The upgrade's dqlite and backend copies in `/root/` on opti1
(`backend-*.tar`, `dqlite-pre-*`) (`backend-*.tar`, `dqlite-pre-*`)
- [ ] Log in to immich as a jeugdlid (quota 0) - [ ] Log in to immich as a jeugdlid (quota 0)
@@ -508,7 +509,7 @@ critical / high:
| Image | Crit | High | What to do | | Image | Crit | High | What to do |
|---|---|---|---| |---|---|---|---|
| tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; goes with the cleanup from 2026-10-27 | | tensorchord/cloudnative-pgvecto.rs:14-v0.2.1 | 38 | 342 | immich's old PG14 cluster; deleted 2026-10-28 |
| cloudnative-pg/postgresql:17.4 | 27 | 260 | authentik's database, a Debian 11 image. Renovate now offers 17.7; a `17.x-standard-trixie` image is more current | | cloudnative-pg/postgresql:17.4 | 27 | 260 | authentik's database, a Debian 11 image. Renovate now offers 17.7; a `17.x-standard-trixie` image is more current |
| tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag | | tensorchord/cloudnative-vectorchord:17.11-1.1.1 | 17 | 136 | immich's database; look for a newer tag |
| longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below | | longhornio/longhorn-instance-manager:v1.7.3 | 11 | 159 | still serves 16 volumes, see below |
@@ -519,7 +520,7 @@ critical / high:
| rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image | | rubenhensen/bolletjestrui:main-10268f1 | 5 | 76 | rebuild on a current base image |
| coredns/coredns:1.10.1 | 5 | 70 | see below | | coredns/coredns:1.10.1 | 5 | 70 | see below |
| dexidp/dex:v2.45.1, argocd v3.5.6, redis 8.2.3 | 4, 1, 2 | | come with Argo CD | | dexidp/dex:v2.45.1, argocd v3.5.6, redis 8.2.3 | 4, 1, 2 | | come with Argo CD |
| mariadb:11.4.5 | 4 | 65 | the newest 11.4 patch; Renovate now covers it | | mariadb:11.4.5 | 4 | 65 | 11.4.13 since PR #60; Renovate stays on 11.4 (it offered 11.8, PR #42) |
| calico v3.29.3 | 2-3 | ~50 | comes with microk8s | | calico v3.29.3 | 2-3 | ~50 | comes with microk8s |
| goauthentik ldap / server 2026.8.5 | 3, 1 | 74, 65 | the next authentik patch | | goauthentik ldap / server 2026.8.5 | 3, 1 | 74, 65 | the next authentik patch |
+4 -3
View File
@@ -169,8 +169,9 @@ pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:
v2.4.1 refused to start (it only accepts VectorChord <0.6); v2.5.0 and up v2.4.1 refused to start (it only accepts VectorChord <0.6); v2.5.0 and up
accept <2, so immich went to v2.7.5, the last 2.x. It rebuilt both accept <2, so immich went to v2.7.5, the last 2.x. It rebuilt both
indexes as `vchordrq` in about 1.5 minutes indexes as `vchordrq` in about 1.5 minutes
- [ ] After a few days without problems: delete the old cluster - [x] After a few days without problems: delete the old cluster
(`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump. (`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump.
Done 2026-10-28 (PR #61), with the PG14 volume's Longhorn backups.
Rolling back means reverting the old volume to the snapshot, since its Rolling back means reverting the old volume to the snapshot, since its
embeddings are now `real[]` without pgvecto.rs embeddings are now `real[]` without pgvecto.rs
- [x] immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on - [x] immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on
@@ -189,8 +190,8 @@ pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:
(v3.3.0), for faster ML with less memory; numerically the same models, (v3.3.0), for faster ML with less memory; numerically the same models,
so no jobs to rerun. The model cache is an emptyDir, so the one-off so no jobs to rerun. The model cache is an emptyDir, so the one-off
preparation repeats on every restart preparation repeats on every restart
- [ ] After a few days without problems: delete the `immich-pg17-pre-v3` - [x] After a few days without problems: delete the `immich-pg17-pre-v3`
snapshot and dump together with the PG14 leftovers above snapshot and dump together with the PG14 leftovers above (2026-10-28)
## k8s-dqlite write stalls (found 2026-10-22) ## k8s-dqlite write stalls (found 2026-10-22)
+4 -1
View File
@@ -29,7 +29,10 @@ spec:
# Filled once from the PostgreSQL 14 cluster (immich-postgres) after its # Filled once from the PostgreSQL 14 cluster (immich-postgres) after its
# embedding columns were turned into real[] and pgvecto.rs was dropped, # embedding columns were turned into real[] and pgvecto.rs was dropped,
# see docs/upgrade-k8s-1.33.md. # see docs/upgrade-k8s-1.33.md. That cluster is gone since 2026-10-28, so
# creating this Cluster again fails here, on purpose: an empty database
# would start immich with an empty library. Restore from Garage instead
# (bootstrap.recovery, see postgresql-backup.yaml).
bootstrap: bootstrap:
initdb: initdb:
database: immich database: immich
-28
View File
@@ -1,28 +0,0 @@
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: immich-postgres
namespace: immich
spec:
instances: 1
# Use TensorChord's CNPG-compatible image with pgvecto.rs v0.2.1
imageName: ghcr.io/tensorchord/cloudnative-pgvecto.rs:14-v0.2.1
postgresql:
shared_preload_libraries:
- "vectors.so"
bootstrap:
initdb:
database: immich
owner: immich
postInitSQL:
- CREATE EXTENSION IF NOT EXISTS vectors;
- CREATE EXTENSION IF NOT EXISTS cube;
- CREATE EXTENSION IF NOT EXISTS earthdistance CASCADE;
- ALTER USER immich WITH SUPERUSER;
storage:
size: 20Gi
storageClass: longhorn
+66 -2
View File
@@ -8,6 +8,10 @@
# - the group account's library is shared as a partner and shown in # - the group account's library is shared as a partner and shown in
# everyone's timeline, so search covers it (immich only searches your own # everyone's timeline, so search covers it (immich only searches your own
# and your timeline partners' photos); # and your timeline partners' photos);
# - everyone is put in the group account's cluster group and gets its
# people (person sharing, immich 3.3): people are per user, and a partner
# library doesn't bring its people along. Those who may upload can name
# them, for everyone;
# - the invitation notifications for albums older than a week are hidden, # - the invitation notifications for albums older than a week are hidden,
# or a new member would get one per album; # or a new member would get one per album;
# - sessions of OAuth users older than SESSION_MAX_AGE_DAYS are ended, so # - sessions of OAuth users older than SESSION_MAX_AGE_DAYS are ended, so
@@ -15,8 +19,11 @@
# Online. # Online.
# Albums and partners go through the API. Joining other people's albums # Albums and partners go through the API. Joining other people's albums
# (the API only lets an album's owner or editors share it), the timeline # (the API only lets an album's owner or editors share it), the timeline
# flag (which only the receiving user can set), the notifications and the # flag and the cluster group (which only the receiving user can set or
# sessions go straight to the database. # accept), sharing the people (the API wants person.update, which the
# group's key lacks, and with password login off nobody can log in as the
# group to add it), the notifications and the sessions go straight to the
# database.
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@@ -24,6 +31,7 @@ metadata:
namespace: immich namespace: immich
data: data:
sync.py: | sync.py: |
import collections
import json import json
import os import os
import subprocess import subprocess
@@ -129,6 +137,62 @@ data:
'UPDATE partner SET "inTimeline" = true WHERE "sharedById" = :\'group\' AND NOT "inTimeline"', 'UPDATE partner SET "inTimeline" = true WHERE "sharedById" = :\'group\' AND NOT "inTimeline"',
group=group_id, group=group_id,
) )
# What accepting an invitation to the group account's cluster group does
# (ClusterGroupService.acceptRequest): the person groups only the member
# has people in move along. Members with people in groups that others
# also have people in are left alone; that only happens after sharing.
for (email,) in sql(
"""
BEGIN;
CREATE TEMP TABLE moving ON COMMIT DROP AS
SELECT u.id FROM "user" u, "user" g
WHERE g.id = :'group' AND u.id = ANY (:'members'::uuid[])
AND u."clusterGroupId" <> g."clusterGroupId"
AND NOT EXISTS (
SELECT 1 FROM person p
JOIN person o ON o."personGroupId" = p."personGroupId" AND o."ownerId" <> p."ownerId"
WHERE p."ownerId" = u.id
);
UPDATE person_group SET "clusterGroupId" = (SELECT "clusterGroupId" FROM "user" WHERE id = :'group')
WHERE id IN (SELECT p."personGroupId" FROM person p JOIN moving m ON m.id = p."ownerId");
DELETE FROM cluster_group_request
WHERE "userId" IN (SELECT id FROM moving)
AND "clusterGroupId" = (SELECT "clusterGroupId" FROM "user" WHERE id = :'group');
UPDATE "user" SET "clusterGroupId" = (SELECT "clusterGroupId" FROM "user" WHERE id = :'group')
WHERE id IN (SELECT id FROM moving)
RETURNING email;
COMMIT;
""",
group=group_id,
members="{" + ",".join(members) + "}",
):
print(f"put {email} in the group's cluster group", flush=True)
# What PUT /people/users does for the group account: its people, with
# their names, in every member's people list and search filters. Write
# access (naming people, for everyone) for those who may upload.
shared = sql(
"""
INSERT INTO person_user ("personGroupId", "sharedById", "sharedWithId", role)
SELECT p."personGroupId", p."ownerId", m.id, m.role::person_user_role_enum
FROM person p
JOIN "user" g ON g.id = p."ownerId"
CROSS JOIN unnest(:'ids'::uuid[], :'roles'::text[]) AS m(id, role)
JOIN "user" u ON u.id = m.id AND u."clusterGroupId" = g."clusterGroupId"
WHERE p."ownerId" = :'group' AND NOT p."isHidden"
AND NOT EXISTS (
SELECT 1 FROM person_user pu
WHERE pu."personGroupId" = p."personGroupId" AND pu."sharedById" = p."ownerId"
AND pu."sharedWithId" = m.id AND pu.role = m.role::person_user_role_enum
)
ON CONFLICT ("personGroupId", "sharedById", "sharedWithId") DO UPDATE SET role = EXCLUDED.role
RETURNING "sharedWithId", role
""",
group=group_id,
ids="{" + ",".join(wanted) + "}",
roles="{" + ",".join("read" if role == "viewer" else "write" for role in wanted.values()) + "}",
)
for (uid, role), count in sorted(collections.Counter(map(tuple, shared)).items()):
print(f"share {count} people with {members[uid]} ({role})", flush=True)
sql( sql(
""" """
UPDATE notification SET "deletedAt" = now() UPDATE notification SET "deletedAt" = now()
+1 -1
View File
@@ -8,7 +8,7 @@ spec:
name: mbgwp-mariadb-credentials name: mbgwp-mariadb-credentials
key: root-password key: root-password
image: mariadb:11.4.5 image: mariadb:11.4.13
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
port: 3306 port: 3306
+8 -2
View File
@@ -82,14 +82,20 @@
"description": "A PostgreSQL major upgrade needs pg_upgrade or a dump, not a tag bump", "description": "A PostgreSQL major upgrade needs pg_upgrade or a dump, not a tag bump",
"matchDepNames": [ "matchDepNames": [
"ghcr.io/cloudnative-pg/postgresql", "ghcr.io/cloudnative-pg/postgresql",
"ghcr.io/tensorchord/cloudnative-vectorchord", "ghcr.io/tensorchord/cloudnative-vectorchord"
"ghcr.io/tensorchord/cloudnative-pgvecto.rs"
], ],
"matchUpdateTypes": [ "matchUpdateTypes": [
"major" "major"
], ],
"enabled": false "enabled": false
}, },
{
"description": "MariaDB goes from one LTS to the next by hand, after a dump: there is no way back. 11.4 has community support until May 2029, longer than 11.8 (June 2028); the next step is 12.3 (June 2029). Within 11.4 the patches come through",
"matchDepNames": [
"mariadb"
],
"allowedVersions": "<11.5"
},
{ {
"description": "Patch updates of apps that hold little state and are easy to roll back, and of CI's tools, merge themselves once CI passes: three days after the release, in the evening or at the weekend, when the cluster can be reached. Not the databases, operators, Vault, authentik, Longhorn, Velero, immich, Gitea or WordPress. Renovate merges itself; the repo has no branch protection for GitHub's auto-merge to wait on", "description": "Patch updates of apps that hold little state and are easy to roll back, and of CI's tools, merge themselves once CI passes: three days after the release, in the evening or at the weekend, when the cluster can be reached. Not the databases, operators, Vault, authentik, Longhorn, Velero, immich, Gitea or WordPress. Renovate merges itself; the repo has no branch protection for GitHub's auto-merge to wait on",
"matchFileNames": [ "matchFileNames": [