Compare commits

...
27 Commits
Author SHA1 Message Date
Ruben Hensen c5d2ebaefc docs: record the Envoy retry policy 2026-10-24 17:06:29 +02:00
Ruben Hensen 5f8394e0c0 docs: plan a second SSD for Longhorn
Hardware to buy, shrinking the data first (214 GiB of old snapshots), installing the NVMe drive, and moving every replica over with a disk eviction.
2026-10-24 15:17:15 +02:00
Ruben Hensen 396eb2665c docs: record why controllers still lost their leases
Heavy writes on the node's only SSD stall dqlite; the fixes so far, the running measurement and the second-SSD option.
2026-10-24 14:46:09 +02:00
Ruben Hensen 858904384c Give leader-elected controllers longer leases
The snapshot-controller (1,059 restarts), the CNPG operator and Envoy
Gateway restart whenever a lease renewal takes more than 10 seconds,
which happens when heavy writes stall dqlite on the node's SSD. Raise
lease duration, renew deadline and retry period from 15s/10s/2s to
60s/40s/5s so they ride out stalls of up to ~40 seconds. With one
replica, or a standby, a slower takeover costs nothing here.
2026-10-24 14:43:38 +02:00
Ruben Hensen 9014898fc5 velero: run kopia maintenance daily and throttled
Each of the 15 kopia repositories ran maintenance every hour, and about
once a day a full run of several minutes that writes its cache to the
node's only SSD. Those runs line up with dqlite stalling for seconds and
controllers losing their leases (vault 09:57-10:07, argocd and mbgwp
09:17-09:40 today). Make the default frequency 24h and cap the jobs at
half a CPU, so a full run writes slower and longer instead of in one
burst.
2026-10-24 14:43:38 +02:00
Ruben Hensen 111a9ddcea docs: foto@ is deleted 2026-10-24 14:20:07 +02:00
Ruben Hensen a8296a514e docs: foto@'s likes are deleted 2026-10-24 14:19:00 +02:00
Ruben Hensen 712fa1fe65 docs: sol-auth is gone 2026-10-24 14:15:14 +02:00
Ruben Hensen c3505bc8e3 immich: remove sol-auth
foto.scoutingmbg.nl goes straight to immich since the switch to Scouts
Online, so the Apache with mod_auth_openid (OpenID 2) in front of it has
no traffic left.
2026-10-24 14:14:43 +02:00
Ruben Hensen 6c6f473f63 docs: fotoupload@ sessions ended 2026-10-24 14:13:40 +02:00
Ruben Hensen a5478d60e6 docs: share-sync joins albums that kaderleden create 2026-10-24 14:11:07 +02:00
Ruben Hensen 88f68cd698 immich: share albums that kaderleden create automatically
Nobody uses the group account anymore, so new albums belong to the
leiding who make them, and share-sync only shared albums the group
account was in. It now first adds the group account as editor to every
album an uploader owns (straight in the database: the API only lets an
album's owner or editors share it), after which the album goes to
everyone like the others. Albums whose name starts with "Privé" or
"Prive" are left alone.
2026-10-24 14:10:30 +02:00
Ruben Hensen f69b790a58 docs: immich now logs in with Scouts Online only 2026-10-24 14:04:43 +02:00
Ruben Hensen 71af3a9342 immich: fix hiding album invitations in share-sync
immich stores a notification's data as a JSON string inside the jsonb column, so data->>'albumId' was always null and no invitation was hidden. Unwrap it first.
2026-10-24 14:03:02 +02:00
Ruben Hensen 348cce6d5d gatus: check immich's ping endpoint for foto.scoutingmbg.nl
Without sol-auth in front, /_app is no longer the open path to check;
/api/server/ping answers without a login.
2026-10-24 14:01:23 +02:00
Ruben Hensen abed11dbaa immich: log in with Scouts Online only
foto.scoutingmbg.nl now goes straight to immich instead of through
sol-auth. immich starts the Scouts Online login right away, the password
login is off and the login page no longer shows the password of the
shared viewing account. share-sync is on, so everyone gets every album
and the group's library in their timeline.

sol-auth stays deployed for now, so switching the route back is the way
back.
2026-10-24 14:01:23 +02:00
Ruben Hensen 86a0c59c6b immich: add share-sync, suspended
immich has no user groups, so a CronJob shares the group account's
albums with every user, as editor when they may upload (quota other than
0, which authentik sets from Scouts Online) and as viewer otherwise. It
also shares the group account's library as a partner and shows it in
every timeline, since immich only searches your own and your timeline
partners' photos; hides the album invitations for albums older than a
week; and ends OAuth sessions older than 30 days, so people who left the
group lose access at their next login.

Suspended until immich logs in through Scouts Online only.
2026-10-24 13:48:07 +02:00
Ruben Hensen ebf28c1bf1 docs: step B of the Scouts Online login is live 2026-10-24 13:43:24 +02:00
Ruben Hensen 135bca4a83 authentik: log in to immich with Scouts Online
Adds Scouts Online 3 (login.scouting.nl) as an OIDC source. Its userinfo
has the member's roles within the group; a property mapping puts every
member in "MBG leden" and those with an active Kaderlid role also in
"MBG kader", on every login. Members become external users. Family
members can share an email address and immich needs a unique one, so
those get a plus address with their member number.

The immich provider now authenticates through a flow whose only stage
is an identification stage with this one source and no fields, which
sends people straight on to Scouts Online. The source's own flows only
run through the source.
2026-10-24 13:37:12 +02:00
Ruben Hensen 1c6b42b20b docs: step A of the Scouts Online login is live 2026-10-24 13:25:02 +02:00
Ruben Hensen f3f36c3718 docs: plan the immich login with Scouts Online
Building blocks, design and the three steps towards personal accounts with upload rights from Scouts Online roles.
2026-10-24 13:19:52 +02:00
Ruben Hensen 399387bfb7 immich: configure through a config file with OAuth
immich gets its settings from a Secret that ESO renders with the OAuth
client secret from Vault. It keeps the settings that were in the database
(external domain, login page message, storage template) and adds OAuth
through authentik next to the password login, with auto register. With a
config file the settings are read-only in the admin UI.
2026-10-24 13:19:52 +02:00
Ruben Hensen 48b0873ca3 authentik: add an OIDC provider for immich
The Scouting MBG photos (immich) will log in through this authentik,
which later passes on Scouts Online. The provider issues immich_role and
immich_quota: admins and the members of "MBG kader" may upload (-1),
everyone else gets a quota of 0, which makes immich refuse uploads.
"MBG leden" and "MBG kader" are filled by the Scouts Online source in a
next step; until then only authentik Admins can use the app.

The scouting blueprints and their secrets are mounted as directories
rather than with subPath, and the blueprint reads the client secret with
!File instead of !Env, so later changes reach authentik without a restart.
2026-10-24 13:19:52 +02:00
Ruben Hensen 278f6fc55f authentik: limit personal apps to authentik Admins
Actual Budget, Argo CD, FreshRSS, Stalwart and Vault had no policy
binding, so any authentik user could sign in to them. Scouts Online
members are about to get accounts in this authentik for immich; bind
each app to the authentik Admins group, as Longhorn already was.
2026-10-24 13:04:33 +02:00
Ruben Hensen f4e74a9503 docs: record the immich 3 upgrade
v3.3.5 with chart 0.13.4 is running; note the safety net to clean up later and the optional revised ML models.
2026-10-24 12:33:43 +02:00
Ruben Hensen c270bcc12b immich: upgrade to v3.3.5 and chart 0.13.4
immich 3 drops pgvecto.rs, which the database no longer uses since the
move to PostgreSQL 17 with VectorChord 1.1.1 (supported range >=0.3 <2)
and pgvector 0.8.6 (>=0.7 <0.9). None of the removed environment
variables are set here. The mobile app cannot pass sol-auth, so there are
no app clients to upgrade first.

Charts after 0.12.0 are only published to the OCI registry. Since 0.13.0
the chart validates its values, and emptyDir volumes no longer accept
size, accessMode or storageClass; they did nothing anyway. Valkey follows
the chart's new default, 9.1.2. Each component now gets its own
ServiceAccount without a mounted token.

Safety net: Longhorn snapshot immich-pg17-pre-v3 and a pg_dump in
~/immich-pg17-pre-v3.dump on opti1, since immich cannot be downgraded.
2026-10-24 12:29:35 +02:00
Ruben Hensen 14e61ee5ee gateway: retry requests on dropped backend connections
When a backend closes a kept-alive connection just as Envoy sends a
request over it, Envoy returns a 503 (response flag UC) and does not
retry by default. ingress-nginx retried these, so the race never showed.

The Apache in front of immich (sol-auth) recycles its processes after 30
connections, which made this happen during page loads: three chunks got
a 503 and immich failed with "error loading dynamically imported module".
authentik and ntfy saw it too, less often.

Retry up to twice on connection failures and resets, for every route on
the Gateway.
2026-10-24 12:19:37 +02:00
26 changed files with 1110 additions and 225 deletions

No files matched your search

+56
View File
@@ -127,6 +127,29 @@ spec:
- name: blueprint-longhorn - name: blueprint-longhorn
configMap: configMap:
name: authentik-blueprint-longhorn name: authentik-blueprint-longhorn
# Directory mounts, not subPath, so changes reach the pods
# without a restart. The secrets are optional: a blueprint
# that needs a missing one fails until it shows up.
- name: blueprint-scouting
configMap:
name: authentik-blueprint-scouting
- name: scouting-secrets
projected:
sources:
- secret:
name: immich-oidc-client-secret
optional: true
items:
- key: client_secret
path: immich-oidc-client-secret
- secret:
name: scouts-online-client
optional: true
items:
- key: client_id
path: scouts-online-client-id
- key: client_secret
path: scouts-online-client-secret
volumeMounts: volumeMounts:
- name: blueprint-vault - name: blueprint-vault
mountPath: /blueprints/custom/vault-oidc.yaml mountPath: /blueprints/custom/vault-oidc.yaml
@@ -152,6 +175,11 @@ spec:
- name: blueprint-longhorn - name: blueprint-longhorn
mountPath: /blueprints/custom/longhorn-oidc.yaml mountPath: /blueprints/custom/longhorn-oidc.yaml
subPath: longhorn-oidc.yaml subPath: longhorn-oidc.yaml
- name: blueprint-scouting
mountPath: /blueprints/custom/scouting
- name: scouting-secrets
mountPath: /secrets/scouting
readOnly: true
- name: patched-ca-bundle - name: patched-ca-bundle
mountPath: /etc/ssl/certs mountPath: /etc/ssl/certs
readOnly: true readOnly: true
@@ -184,6 +212,29 @@ spec:
- name: blueprint-longhorn - name: blueprint-longhorn
configMap: configMap:
name: authentik-blueprint-longhorn name: authentik-blueprint-longhorn
# Directory mounts, not subPath, so changes reach the pods
# without a restart. The secrets are optional: a blueprint
# that needs a missing one fails until it shows up.
- name: blueprint-scouting
configMap:
name: authentik-blueprint-scouting
- name: scouting-secrets
projected:
sources:
- secret:
name: immich-oidc-client-secret
optional: true
items:
- key: client_secret
path: immich-oidc-client-secret
- secret:
name: scouts-online-client
optional: true
items:
- key: client_id
path: scouts-online-client-id
- key: client_secret
path: scouts-online-client-secret
volumeMounts: volumeMounts:
- name: blueprint-vault - name: blueprint-vault
mountPath: /blueprints/custom/vault-oidc.yaml mountPath: /blueprints/custom/vault-oidc.yaml
@@ -209,6 +260,11 @@ spec:
- name: blueprint-longhorn - name: blueprint-longhorn
mountPath: /blueprints/custom/longhorn-oidc.yaml mountPath: /blueprints/custom/longhorn-oidc.yaml
subPath: longhorn-oidc.yaml subPath: longhorn-oidc.yaml
- name: blueprint-scouting
mountPath: /blueprints/custom/scouting
- name: scouting-secrets
mountPath: /secrets/scouting
readOnly: true
- name: patched-ca-bundle - name: patched-ca-bundle
mountPath: /etc/ssl/certs mountPath: /etc/ssl/certs
readOnly: true readOnly: true
+5
View File
@@ -19,6 +19,11 @@ spec:
# upgrades instead of restarting them. Both clusters have a # upgrades instead of restarting them. Both clusters have a
# single instance, so a restart means downtime. # single instance, so a restart means downtime.
ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES: "true" ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES: "true"
# Ride out dqlite stalls of up to ~40s instead of restarting when a
# lease renewal takes over 10s (docs/upgrade-k8s-1.33.md).
additionalArgs:
- --leader-lease-duration=60
- --leader-renew-deadline=40
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: cnpg-system namespace: cnpg-system
+13
View File
@@ -13,6 +13,19 @@ spec:
chart: gateway-helm chart: gateway-helm
# v1.9 supports Kubernetes 1.33 to 1.36 # v1.9 supports Kubernetes 1.33 to 1.36
targetRevision: v1.9.2 targetRevision: v1.9.2
helm:
values: |
config:
envoyGateway:
provider:
kubernetes:
# Ride out dqlite stalls of up to ~40s instead of restarting
# when a lease renewal takes over 10s
# (docs/upgrade-k8s-1.33.md).
leaderElection:
leaseDuration: 60s
renewDeadline: 40s
retryPeriod: 5s
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: envoy-gateway-system namespace: envoy-gateway-system
@@ -9,6 +9,23 @@ spec:
repoURL: https://github.com/kubernetes-csi/external-snapshotter.git repoURL: https://github.com/kubernetes-csi/external-snapshotter.git
targetRevision: v8.2.1 targetRevision: v8.2.1
path: deploy/kubernetes/snapshot-controller path: deploy/kubernetes/snapshot-controller
kustomize:
patches:
# Ride out dqlite stalls of up to ~40s instead of restarting when a
# lease renewal takes over 10s (docs/upgrade-k8s-1.33.md).
- target:
kind: Deployment
name: snapshot-controller
patch: |-
- op: add
path: /spec/template/spec/containers/0/args/-
value: --leader-election-lease-duration=60s
- op: add
path: /spec/template/spec/containers/0/args/-
value: --leader-election-renew-deadline=40s
- op: add
path: /spec/template/spec/containers/0/args/-
value: --leader-election-retry-period=5s
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
namespace: kube-system namespace: kube-system
@@ -38,3 +38,12 @@ data:
name: Actual Budget name: Actual Budget
provider: !KeyOf actualbudget-provider provider: !KeyOf actualbudget-provider
meta_launch_url: https://ynab.rubenhensen.nl meta_launch_url: https://ynab.rubenhensen.nl
# Only admins: Scouts Online members get accounts in this authentik
# too (for immich) and must not reach personal apps.
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf actualbudget-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
+9
View File
@@ -38,3 +38,12 @@ data:
name: ArgoCD name: ArgoCD
provider: !KeyOf argocd-provider provider: !KeyOf argocd-provider
meta_launch_url: https://argocd.rubenhensen.nl meta_launch_url: https://argocd.rubenhensen.nl
# Only admins: Scouts Online members get accounts in this authentik
# too (for immich) and must not reach personal apps.
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf argocd-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
+9
View File
@@ -40,3 +40,12 @@ data:
name: FreshRSS name: FreshRSS
provider: !KeyOf freshrss-provider provider: !KeyOf freshrss-provider
meta_launch_url: https://rss.rubenhensen.nl meta_launch_url: https://rss.rubenhensen.nl
# Only admins: Scouts Online members get accounts in this authentik
# too (for immich) and must not reach personal apps.
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf freshrss-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
+9
View File
@@ -38,3 +38,12 @@ data:
name: Stalwart Mail name: Stalwart Mail
provider: !KeyOf stalwart-provider provider: !KeyOf stalwart-provider
meta_launch_url: https://mail.rubenhensen.nl meta_launch_url: https://mail.rubenhensen.nl
# Only admins: Scouts Online members get accounts in this authentik
# too (for immich) and must not reach personal apps.
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf stalwart-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
+278
View File
@@ -0,0 +1,278 @@
# Login for the Scouting MBG photos (immich on foto.scoutingmbg.nl).
# Mounted as a directory, unlike the other blueprints, so authentik picks up
# changes without a restart; secrets come from files in /secrets/scouting
# for the same reason (see apps/templates/authentik-helm.yaml).
apiVersion: v1
kind: ConfigMap
metadata:
name: authentik-blueprint-scouting
data:
immich-oidc.yaml: |
version: 1
metadata:
name: immich OIDC
entries:
# The source and the login flow below come from scouts-online.yaml.
- model: authentik_blueprints.metaapplyblueprint
attrs:
identifiers:
path: custom/scouting/scouts-online.yaml
required: true
# Filled by the Scouts Online source on every login: every member, and
# the members with an active Kaderlid role.
- model: authentik_core.group
id: mbg-leden
state: present
identifiers:
name: MBG leden
- model: authentik_core.group
id: mbg-kader
state: present
identifiers:
name: MBG kader
# immich syncs both claims on every login. A quota of 0 makes immich
# refuse uploads, -1 means unlimited.
- model: authentik_providers_oauth2.scopemapping
id: immich-scope
state: present
identifiers:
name: "immich: role and quota"
attrs:
name: "immich: role and quota"
scope_name: immich
description: Upload rights in the Scouting MBG photos
expression: |
admin = ak_is_group_member(request.user, name="authentik Admins")
uploader = admin or ak_is_group_member(request.user, name="MBG kader")
return {
"immich_role": "admin" if admin else "user",
"immich_quota": -1 if uploader else 0,
}
- model: authentik_providers_oauth2.oauth2provider
id: immich-provider
state: present
identifiers:
name: immich
attrs:
name: immich
authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
# Straight on to Scouts Online. Admins with a password log in at
# authentik first.
authentication_flow: !Find [authentik_flows.flow, [slug, scouts-online-login]]
invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
client_type: confidential
client_id: immich
client_secret: !File /secrets/scouting/immich-oidc-client-secret
redirect_uris:
- matching_mode: strict
url: https://foto.scoutingmbg.nl/auth/login
# "Link to OAuth" in immich's account settings
- matching_mode: strict
url: https://foto.scoutingmbg.nl/user-settings
- matching_mode: strict
url: app.immich:///oauth-callback
signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Self-signed Certificate"]]
property_mappings:
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-openid]]
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-email]]
- !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-profile]]
- !KeyOf immich-scope
- model: authentik_core.application
id: immich-app
state: present
identifiers:
slug: immich
attrs:
name: Scouting MBG foto's
provider: !KeyOf immich-provider
meta_launch_url: https://foto.scoutingmbg.nl
policy_engine_mode: any
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf immich-app
attrs:
group: !KeyOf mbg-leden
- model: authentik_policies.policybinding
state: present
identifiers:
order: 1
target: !KeyOf immich-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
scouts-online.yaml: |
version: 1
metadata:
name: Scouts Online
entries:
- model: authentik_sources_oauth.oauthsourcepropertymapping
id: user-mapping
state: present
identifiers:
name: "Scouts Online: user"
attrs:
name: "Scouts Online: user"
expression: |
# info is the userinfo of Scouts Online (https://login.scouting.nl/docs/api).
# It only holds the roles within the organisations of this connection,
# which is just the group.
from datetime import date
today = date.today().isoformat()
roles = {role.get("id"): role for role in (info.get("roles") or {}).values()}
kader = False
for assignment in info.get("role_assignments") or []:
start = (assignment.get("start_date") or "")[:10]
end = (assignment.get("end_date") or "")[:10]
if (start and start > today) or (end and end < today):
continue
role = roles.get(assignment.get("role_id")) or {}
if "Kaderlid" in (role.get("member_type") or []):
kader = True
member_id = info.get("member_id")
email = info.get("email") or ""
# Family members can share an email address; immich needs one per account.
if email and info.get("multiple_accounts"):
local, _, domain = email.partition("@")
email = f"{local}+{member_id}@{domain}"
return {
"username": f"sol-{member_id}",
"email": email,
"name": info.get("name") or "",
"groups": ["MBG leden", "MBG kader"] if kader else ["MBG leden"],
"attributes": {"sol_member_id": member_id},
}
# The source's flows can only be reached through the source.
- model: authentik_policies_expression.expressionpolicy
id: sso-only
state: present
identifiers:
name: scouts-online-sso-only
attrs:
expression: return ak_is_sso_flow
# New members become external users: they can use the apps they are
# let into, not authentik's own interface.
- model: authentik_stages_user_write.userwritestage
id: user-write
state: present
identifiers:
name: scouts-online-user-write
attrs:
user_creation_mode: always_create
user_type: external
- model: authentik_stages_user_login.userloginstage
id: user-login
state: present
identifiers:
name: scouts-online-user-login
- model: authentik_flows.flow
id: enrollment-flow
state: present
identifiers:
slug: scouts-online-enrollment
attrs:
name: Scouts Online enrollment
title: Scouting MBG
designation: enrollment
authentication: none
- model: authentik_flows.flowstagebinding
state: present
identifiers:
order: 0
target: !KeyOf enrollment-flow
stage: !KeyOf user-write
- model: authentik_flows.flowstagebinding
state: present
identifiers:
order: 1
target: !KeyOf enrollment-flow
stage: !KeyOf user-login
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf enrollment-flow
policy: !KeyOf sso-only
- model: authentik_flows.flow
id: authentication-flow
state: present
identifiers:
slug: scouts-online-authentication
attrs:
name: Scouts Online authentication
title: Scouting MBG
designation: authentication
authentication: none
- model: authentik_flows.flowstagebinding
state: present
identifiers:
order: 0
target: !KeyOf authentication-flow
stage: !KeyOf user-login
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf authentication-flow
policy: !KeyOf sso-only
# SOL only lets in members of the organisations picked in its
# connection, and only sends roles within them. The callback URL is
# https://authentik.rubenhensen.nl/source/oauth/callback/scouts-online/
- model: authentik_sources_oauth.oauthsource
id: source
state: present
identifiers:
slug: scouts-online
attrs:
name: Scouts Online
slug: scouts-online
enabled: true
provider_type: openidconnect
consumer_key: !File /secrets/scouting/scouts-online-client-id
consumer_secret: !File /secrets/scouting/scouts-online-client-secret
authorization_url: https://login.scouting.nl/authorize
access_token_url: https://login.scouting.nl/token
profile_url: https://login.scouting.nl/oidc/userinfo
oidc_well_known_url: https://login.scouting.nl/.well-known/openid-configuration
oidc_jwks_url: https://login.scouting.nl/.well-known/openid-jwks.json
additional_scopes: membership roles
pkce: S256
authorization_code_auth_method: basic_auth
user_matching_mode: identifier
# Links to the groups that immich-oidc.yaml creates
group_matching_mode: name_link
user_path_template: goauthentik.io/sources/scouts-online
enrollment_flow: !KeyOf enrollment-flow
authentication_flow: !KeyOf authentication-flow
user_property_mappings:
- !KeyOf user-mapping
# Where immich sends people who aren't logged in. An identification
# stage without fields and with one source goes straight to it.
- model: authentik_stages_identification.identificationstage
id: identification
state: present
identifiers:
name: scouts-online-identification
attrs:
user_fields: []
sources:
- !KeyOf source
show_source_labels: true
- model: authentik_flows.flow
id: login-flow
state: present
identifiers:
slug: scouts-online-login
attrs:
name: Scouts Online login
title: Scouting MBG
designation: authentication
- model: authentik_flows.flowstagebinding
state: present
identifiers:
order: 0
target: !KeyOf login-flow
stage: !KeyOf identification
+9
View File
@@ -40,3 +40,12 @@ data:
name: Vault name: Vault
provider: !KeyOf vault-provider provider: !KeyOf vault-provider
meta_launch_url: https://vault.rubenhensen.nl meta_launch_url: https://vault.rubenhensen.nl
# Only admins: Scouts Online members get accounts in this authentik
# too (for immich) and must not reach personal apps.
- model: authentik_policies.policybinding
state: present
identifiers:
order: 0
target: !KeyOf vault-app
attrs:
group: !Find [authentik_core.group, [name, "authentik Admins"]]
+43
View File
@@ -153,3 +153,46 @@ spec:
remoteRef: remoteRef:
key: kv/longhorn-oidc key: kv/longhorn-oidc
property: client_secret property: client_secret
---
# Mounted as a file for blueprint-scouting.yaml, not as an env variable.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: immich-oidc-client-secret
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: immich-oidc-client-secret
creationPolicy: Owner
data:
- secretKey: client_secret
remoteRef:
key: kv/immich-oidc
property: client_secret
---
# The Scouts Online connection (login.scouting.nl, "Beheerde websites"),
# mounted as files for blueprint-scouting.yaml.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: scouts-online-client
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: scouts-online-client
creationPolicy: Owner
data:
- secretKey: client_id
remoteRef:
key: kv/scouts-online
property: client_id
- secretKey: client_secret
remoteRef:
key: kv/scouts-online
property: client_secret
+115
View File
@@ -0,0 +1,115 @@
# immich: log in with Scouts Online
foto.scoutingmbg.nl sits behind sol-auth, an Apache with mod_auth_openid
that lets any Scouting Nederland account through (OpenID 2, which Scouting
Nederland is phasing out). Behind it, people use two shared immich
accounts: `fotoupload@` owns all 45,703 photos and 504 albums, `foto@`
views them (its password is on the login page).
Goal: everyone logs in with their own Scouts Online 3 account and gets an
immich account of their own; kaderleden can upload, everyone else can only
view, and everything is shared with everyone automatically.
## Building blocks
- **Scouts Online 3** (login.scouting.nl) is an OIDC provider. A connection
is registered under "Beheerde websites" by someone with the Webmaster
role in SOL 3, and only members of the organisations picked there can
log in. The `roles` scope returns the member's organisation units
(`Board`, `Game Section` with speltak, committees, ...), roles
(`member_type` `Jeugdlid`/`Kaderlid`, a category such as
`Leidinggevende`) and role assignments with start and end dates, only
from the userinfo endpoint and nested (https://login.scouting.nl/docs/api).
- **immich** can't read that: it takes the claims `immich_quota` (GiB; `0`
blocks uploads, `-1` is unlimited) and `immich_role` (`user`/`admin`)
and syncs both on every login. It links an OAuth login to an existing
account by email, or creates one (auto register).
- **immich has no user groups**; albums are shared per user, as viewer or
editor. The owner and editors of an album may add users and change
their roles. Search only covers your own photos and those of partners
you show in your timeline.
## Design
- **authentik** (authentik.rubenhensen.nl for now; auth.scoutingmbg.nl
later, once the group's DNS has a record for it) brokers the login: a
Scouts Online source turns the roles into the groups `MBG leden` and
`MBG kader` (active `Kaderlid` role) on every login, and the immich
provider turns those into `immich_quota` and `immich_role`
(`authentik/blueprint-scouting.yaml`). Members get `external` users;
every personal app in this authentik is bound to `authentik Admins`.
- **immich** gets its settings from a config file rendered by ESO
(`immich/immich-config.yaml`), OAuth with auto register and, after the
switch, auto launch and no password login.
- **share-sync** (`immich/share-sync.yaml`), a CronJob every 10 minutes:
every user into every album of the group account (`fotoupload@`), as
editor or viewer by quota; the group account's library as a partner of
everyone, shown in the timeline; album invitations for albums older than
a week hidden; sessions of OAuth users older than 30 days ended. Albums
that kaderleden create get the group account as editor first (in the
database, since the API only lets an album's owner or editors share
it), so they go to everyone too; names starting with "Privé"/"Prive"
are skipped. Photos outside albums stay with their uploader.
## Steps
A. immich OAuth through authentik, next to the password login
- [x] Vault `kv/immich-oidc`: `client_secret` (random, e.g. `openssl rand -hex 32`)
- [x] Push the authentik and immich changes. Done 2026-10-24: both rolled
over without downtime. authentik only found the new blueprint after
`blueprints_discovery.send()` in `ak shell`; the claims came out as
admin/-1 for authentik Admins, user/-1 for MBG kader, user/0 for MBG
leden, and other users get no access
- [x] Test with an authentik admin account. The login created a new immich
admin (`ik@rubenhensen.nl`, the authentik email) instead of linking the
old one (`ruben.hensen@protonmail.com`): linking an existing account
takes a password login first, then "Link to OAuth" in its settings
B. Scouts Online source
- [x] SOL connection, by someone with the Webmaster role: redirect URI
`https://authentik.rubenhensen.nl/source/oauth/callback/scouts-online/`,
post logout `https://foto.scoutingmbg.nl/`, scopes E-mail, Persoonlijk,
Lidmaatschap, Rollen en lidmaatschappen; only the group, without
sub-organisations; confidential client with PKCE required
- [x] Vault `kv/scouts-online`: `client_id`, `client_secret`
- [x] Source, enrollment and authentication flows, and a login flow that
goes straight to Scouts Online (an identification stage with no fields
and one source redirects on its own). Done 2026-10-24; the user mapping
was tested beforehand with made-up userinfo (active and ended Kaderlid
roles, a shared family email, no roles). A kaderlid's login made an
external authentik user `sol-<member id>` in MBG leden and MBG kader, and
an immich user with unlimited quota. Use a private window to test: an
existing authentik session skips the login flow
- [ ] Test with a jeugdlid (quota 0, so no uploads)
C. Switch
- [x] Vault `kv/immich-share`: `api_key` of `fotoupload@` (album.read,
albumUser.create, albumUser.update, partner.read, partner.create);
share-sync dry run (`DRY_RUN=1`): 504 albums, no errors
- [x] Switched 2026-10-24 12:00 UTC: HTTPRoute straight to
helm-immich-server (tested beforehand with a temporary route behind a
secret header: page, API and a websocket upgrade), auto launch on,
password login off, login page message gone, immich restarted to read
the config file, the 25 sessions of `foto@` ended (those of
`fotoupload@`, phones backing up to the group account, kept),
share-sync on, gatus on s1 and s2 checking `/api/server/ping`. The first
run hid no invitations: immich stores a notification's `data` as a JSON
string inside the jsonb column; fixed, 1008 hidden, the 6 for albums of
the last week kept
- [x] Removed sol-auth 2026-10-24 (`immich/sol-auth*`,
`immich/httpd.conf-cm.yml`). The immich folder app doesn't prune, so its
Deployment, Service and ConfigMap went through a sync limited to those
three with prune on
- [x] `foto@`'s activity (155 likes, no comments) deleted 2026-10-24, and
the account marked deleted the way immich's admin delete does (status
`deleted`, `deletedAt`); immich removes it for good after 7 days
- [x] Nobody uses `fotoupload@` anymore: its 9 (browser) sessions ended
2026-10-24. Its API key stays for share-sync
## Open points
- `foto.scoutingmbg.nl` is a CNAME to `mbg.hensen.io`; hensen.io is about
to expire. Keep it until the group's DNS points foto at scoutingmbg.nl.
- Public shared links become public for real once sol-auth is gone.
- Photos leiding back up from their phones stay private until they add
them to a shared album.
+124
View File
@@ -0,0 +1,124 @@
# A second SSD for Longhorn
## Why
Root, the dqlite database, every Longhorn replica, container images and
emptyDirs all sit on one Apacer AS350 1TB, a DRAM-less 2.5" SATA SSD. When
something writes to it for minutes (Longhorn backups at 00:00, the Velero
backup at 01:00, kopia maintenance, image pulls), its write latency climbs
and dqlite, which syncs every commit, stalls for seconds. Controllers then
lose their leases and restart ("Still stalling after the VACUUM" in
docs/upgrade-k8s-1.33.md). Longer leases and throttled maintenance soften
that; moving Longhorn's data to its own NVMe SSD takes away most of the
writes dqlite competes with.
The alternative, moving dqlite to the new SSD, is less work in data but
not in risk: its directory lives under the microk8s snap's revision
directory, which snapd copies on every refresh, so a bind mount there is
fragile. Longhorn supports several disks per node and moves replicas
between them live.
## Hardware
- opti1 is a Dell OptiPlex 5060 (BIOS 1.29.0); the i5-8600T points to the
Micro, but check the case. Every 5060 has one M.2 2280 slot that takes
PCIe NVMe (Gen3 x4); Dell sells 2280 Gen3 x4 drives for the Micro. The
only storage controller is the SATA AHCI one and no NVMe device is
present, so the slot should be empty.
- Buy an M.2 2280 NVMe SSD **with a DRAM cache**, 2 TB. Gen4 drives work and
run at Gen3 speed, which is plenty and cooler. Examples with DRAM:
Samsung 990 Pro, WD Black SN850X, Kingston KC3000, Crucial T500. Avoid
DRAM-less and QLC drives (WD SN770, Samsung 990 EVO, Crucial P3); they
have the same weakness as the current SSD. In the Micro, skip a heatsink
that doesn't fit under the cover.
- Size: Longhorn holds 339 GiB on disk now, of which 297 GiB is the immich
library (126 GB in use inside the volume; the rest is snapshots, see
below). Personal immich accounts will add uploads; volumes can be sized
up to 551 GiB in total today. 2 TB leaves room; 1 TB would fit for now.
## Before: shrink what has to move
- [ ] Delete stale snapshots. 188 snapshots hold 334 GiB, 214 GiB of it in
snapshots from before October 2026, nearly all on immich-claim
(pvc-75d269dd): a 72.5 GiB one from 2024-12-20, 71.3 GiB from
2025-03-02, daily ones from 2025-03-13 to 03-20 (a recurring job that no
longer exists), 56.6 GiB from 2026-01-20. The Longhorn backups on the NAS
don't depend on them. Deleting a snapshot merges it into the next one,
which writes a lot, so do this in a quiet hour, one volume at a time.
- [ ] Trim immich-claim's filesystem (Longhorn UI, volume, "Trim Filesystem")
so blocks of deleted files are released.
- [ ] Fresh backups: let a nightly Longhorn and Velero backup run, plus a
Longhorn system backup.
- [ ] Set Longhorn's `concurrent-replica-rebuild-per-node-limit` from 5 to 1
for the move, so only one replica copies at a time. It is not in the
Helm values, so set it in the UI (Settings, General) or with
`kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type merge -p '{"value":"1"}'`.
## Install (downtime: everything, about 15 minutes)
- [ ] `sudo shutdown -h now` on opti1. microk8s stops, the volumes detach.
- [ ] Unplug, open the case, put the SSD in the M.2 slot and screw it down.
- [ ] Boot into the BIOS (F2): the NVMe drive shows up under storage, SATA
operation stays AHCI, and the boot sequence still starts "ubuntu" from
the SATA SSD.
- [ ] Boot. `lsblk` shows `nvme0n1`; wait until everything is green again
(gatus on s1 and s2).
## Prepare the disk (sudo on opti1)
```bash
lsblk -o NAME,SIZE,MODEL /dev/nvme0n1 # the new, empty drive
sudo wipefs -a /dev/nvme0n1
sudo parted -s /dev/nvme0n1 mklabel gpt mkpart longhorn ext4 1MiB 100%
sudo mkfs.ext4 -L longhorn-nvme /dev/nvme0n1p1
sudo mkdir -p /var/lib/longhorn-nvme
echo "UUID=$(sudo blkid -s UUID -o value /dev/nvme0n1p1) /var/lib/longhorn-nvme ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload && sudo mount /var/lib/longhorn-nvme
df -h /var/lib/longhorn-nvme
```
Longhorn wants an extent-based filesystem (ext4 or xfs) mounted at a
directory, no symlink. With `nofail` a missing drive doesn't block the
boot; Longhorn then sees an empty directory on the root filesystem with
the wrong filesystem ID and won't use it. Ubuntu's weekly `fstrim.timer`
covers the new filesystem too.
## Move Longhorn's data (no downtime)
- [ ] Add the disk to the Longhorn node:
```bash
kubectl -n longhorn-system patch nodes.longhorn.io opti1 --type merge -p '{"spec":{"disks":{"nvme":{"path":"/var/lib/longhorn-nvme","diskType":"filesystem","allowScheduling":true,"evictionRequested":false,"storageReserved":53687091200,"tags":[]}}}}'
```
(50 GiB reserved.) Wait until the disk is Ready and Schedulable.
- [ ] On the old disk (`default-disk-5db8685563bf9bf9`, `/var/lib/longhorn/`)
set `allowScheduling: false`, then `evictionRequested: true`. Longhorn
builds a new replica of each volume on the NVMe drive and drops the old
one once the rebuild succeeds; it attaches detached volumes for this
itself. All 20 volumes have one replica, so each briefly has two.
- [ ] Watch the old disk's replica count go to 0 (UI, Node, or
`kubectl -n longhorn-system get nodes.longhorn.io opti1 -o yaml`). The
immich library takes longest. Copying reads from the SATA SSD, so expect
a few slow API moments meanwhile; best in the evening.
- [ ] Remove the old disk from `spec.disks` once it has no replicas, and set
the rebuild limit back to 5.
To stop halfway: set `evictionRequested` back to false; replicas already
moved stay on the NVMe drive, the rest on the SATA SSD, and both disks
work.
## Afterwards
- [ ] Compare with before: lease updates over 5s
(`apiserver_request_duration_seconds` for PUT on leases), controller
restarts, and sda's latency during the nightly backups
(`~/iosample.sh`).
- [ ] Optional, if stalls remain: container images
(`/var/snap/microk8s/common/var/lib/containerd`) and the kubelet's
emptyDirs, such as kopia's scratch space, still write to the SATA SSD.
## Sources
- Dell parts for the OptiPlex 5060 Micro, with M.2 2280 PCIe NVMe Gen3 x4
drives: https://dell.com/en-au/shop/pfydresults/6163895
- Longhorn, eviction: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/disks-or-nodes-eviction/
- Longhorn, multiple disks: https://longhorn.io/docs/1.13.0/nodes-and-volumes/nodes/multidisk/
+65 -1
View File
@@ -138,6 +138,13 @@ snapshots after every step. Fresh Velero backup before starting.
dropped mbg.hensen.io; that reissue went through the Gateway's HTTP-01 dropped mbg.hensen.io; that reissue went through the Gateway's HTTP-01
solver. The Ingresses were pruned with `PrunePropagationPolicy=orphan` solver. The Ingresses were pruned with `PrunePropagationPolicy=orphan`
so the Certificates stayed. Gatus now expects a 302 from Longhorn so the Certificates stayed. Gatus now expects a 302 from Longhorn
- [x] Retries (`gateway/retry-policy.yaml`, 2026-10-24). When a backend
closes a kept-alive connection just as Envoy sends a request over it,
Envoy answers 503 (flag UC) and, unlike nginx, does not retry. Behind
sol-auth's Apache, which recycled its processes after 30 connections,
that cost immich chunks during page loads ("error loading dynamically
imported module"). A BackendTrafficPolicy on the Gateway retries twice
on connect failures and resets
## 9. immich: PostgreSQL 14 with pgvecto.rs → 17 with VectorChord ## 9. immich: PostgreSQL 14 with pgvecto.rs → 17 with VectorChord
@@ -166,7 +173,24 @@ pgvecto.rs. Done 2026-10-24, about an hour of immich downtime:
(`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump. (`immich/postgresql-cluster.yaml`), the snapshot/backup and the dump.
Rolling back means reverting the old volume to the snapshot, since its Rolling back means reverting the old volume to the snapshot, since its
embeddings are now `real[]` without pgvecto.rs embeddings are now `real[]` without pgvecto.rs
- [ ] immich 3.x, separately (pgvecto.rs no longer matters for it) - [x] immich 3.x: v2.7.5 → v3.3.5 with chart 0.10.3 → 0.13.4 on
2026-10-24, about three minutes of downtime (two of them pulling the
image). Charts after 0.12.0 are only in the OCI registry
(`ghcr.io/immich-app/immich-charts`), and since 0.13.0 the values
schema rejects size, accessMode and storageClass on emptyDir volumes.
None of the env variables v3 removed were set; the ML image needs
x86-64-v2, and the i5-8600T is v3. The migrations took 16s, both vector
indexes stayed, and `immich-admin schema-check` reports no drift (the
geodata_places_pkey warning at startup came from the geodata import
running at the same time). Safety net, since immich can't be
downgraded: Longhorn snapshot `immich-pg17-pre-v3` and
`~/immich-pg17-pre-v3.dump` (440 MB) on opti1
- [ ] Optional: `MACHINE_LEARNING_MODEL_REVISION=v2` on the ML container
(v3.3.0), for faster ML with less memory; numerically the same models,
so no jobs to rerun. The model cache is an emptyDir, so the one-off
preparation repeats on every restart
- [ ] After a few days without problems: delete the `immich-pg17-pre-v3`
snapshot and dump together with the PG14 leftovers above
## k8s-dqlite write stalls (found 2026-10-22) ## k8s-dqlite write stalls (found 2026-10-22)
@@ -212,6 +236,46 @@ do the first three steps: `ssh -t ruben@192.168.1.204 ./dqlite-tuning.sh`
`--cgroups-per-qos=false`, the likely trigger of the 1.32->1.33 kubelite `--cgroups-per-qos=false`, the likely trigger of the 1.32->1.33 kubelite
panic (#5057), is not set in `args/kubelet`. panic (#5057), is not set in `args/kubelet`.
### Still stalling after the VACUUM (2026-10-24)
The smaller database fixed the snapshot stalls, but controllers kept
losing their leases a few times a day: since the API server started,
256 of ~278k lease updates took over 5s and 44 over 10s. dqlite's log
shows short complete stops (at 10:00:42 even its own health query hit
its deadline). Every lease loss that day lined up with minutes of heavy
writes to the one SSD, which holds root, dqlite, every Longhorn replica,
image layers and emptyDirs:
| Lease losses | Writing at the time |
|---|---|
| 00:02 | Longhorn backups of every volume (from 00:00) |
| 01:09 | Velero backup (kopia reading and caching 133 GB of photos) |
| 09:22 | full kopia maintenance of mbgwp (6 min) and argocd (11 min) |
| 10:00:43 | full kopia maintenance of vault (9 min) |
| 10:31, 10:38 | immich upgrade: image pulls, pg_dump, migrations |
Velero ran maintenance every hour for each of the 15 kopia repositories
(15 pods an hour), without resource limits and with the kopia cache in
an emptyDir; about once a day a run is a full maintenance of minutes, at
whatever time of day the hourly schedule had drifted to.
- [x] Velero: `defaultRepoMaintainFrequency: 24h`, the 15 existing
BackupRepositories patched to 24h by hand, and maintenance jobs capped
at 500m CPU (2Gi memory)
- [x] Leases of 60s/40s/5s instead of 15s/10s/2s (duration, renew deadline,
retry period) for the snapshot-controller (kustomize patch in its Argo
CD app), the CNPG operator (`additionalArgs`) and Envoy Gateway
(`provider.kubernetes.leaderElection`). Longhorn's CSI sidecars and
global manager have no setting for it
- [ ] `~/iosample.sh` on opti1 logs sda's counters and io pressure every 2s
for 26h (from 2026-10-24 12:35 UTC) into `~/iosample.log`, to put
numbers on the disk latency during tonight's backups
- [ ] Structural fix: a second SSD, see docs/longhorn-second-ssd.md. The OptiPlex 5060 (Micro, going by the
i5-8600T) has an M.2 slot that takes NVMe, and no NVMe device is
present; the Apacer AS350 is a DRAM-less 2.5" SATA SSD. An NVMe SSD with
DRAM for Longhorn's data (replicas move over live by evicting the old
disk) would leave the SATA SSD to dqlite and the system
## On to 1.36 ## On to 1.36
microk8s goes up one minor per refresh. 1.34 brings k8s-dqlite 1.8.x with microk8s goes up one minor per refresh. 1.34 brings k8s-dqlite 1.8.x with
+25
View File
@@ -0,0 +1,25 @@
# Retry a request when the backend drops a kept-alive connection just as
# Envoy sends a request over it. Envoy answers that with a 503 (flag UC) and
# does not retry by default; ingress-nginx did. It broke immich's page loads
# behind the Apache of sol-auth (since removed), which recycled its
# processes after 30 connections, and shows up less often elsewhere.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: retry-on-reset
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: homelab
retry:
numRetries: 2
retryOn:
triggers:
- connect-failure
- reset
perRetry:
backOff:
baseInterval: 25ms
maxInterval: 250ms
+1 -1
View File
@@ -123,7 +123,7 @@ endpoints:
conditions: *http-ok conditions: *http-ok
- name: foto.scoutingmbg.nl - name: foto.scoutingmbg.nl
group: Scouting MBG group: Scouting MBG
url: https://foto.scoutingmbg.nl/_app url: https://foto.scoutingmbg.nl/api/server/ping
interval: 60s interval: 60s
conditions: *http-ok conditions: *http-ok
+14 -22
View File
@@ -15,8 +15,10 @@ spec:
project: default project: default
sources: sources:
- chart: immich - chart: immich
repoURL: https://immich-app.github.io/immich-charts # No scheme, so Argo CD pulls the chart from the OCI registry. Charts
targetRevision: 0.10.3 # after 0.12.0 are only published there.
repoURL: ghcr.io/immich-app/immich-charts
targetRevision: 0.13.4
helm: helm:
values: | values: |
controllers: controllers:
@@ -24,7 +26,7 @@ spec:
containers: containers:
main: main:
image: image:
tag: v2.7.5 tag: v3.3.5
env: env:
REDIS_HOSTNAME: '{{ printf "%s-valkey" .Release.Name }}' REDIS_HOSTNAME: '{{ printf "%s-valkey" .Release.Name }}'
IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}' IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'
@@ -48,16 +50,10 @@ spec:
# Automatically creating the library volume is not supported by this chart # Automatically creating the library volume is not supported by this chart
# You have to specify an existing PVC to use # You have to specify an existing PVC to use
existingClaim: immich-claim existingClaim: immich-claim
# configuration is immich-config.json converted to yaml # The config file is a Secret rendered by ESO, since it holds the
# ref: https://immich.app/docs/install/config-file/ # OAuth client secret (immich/immich-config.yaml).
# configurationKind: Secret
configuration: {} existingConfiguration: immich-config
# trash:
# enabled: false
# days: 30
# storageTemplate:
# enabled: true
# template: "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"
# Dependencies # Dependencies
@@ -69,16 +65,14 @@ spec:
main: main:
image: image:
repository: docker.io/valkey/valkey repository: docker.io/valkey/valkey
tag: 9.0-alpine@sha256:b4ee67d73e00393e712accc72cfd7003b87d0fcd63f0eba798b23251bfc9c394 tag: 9.1.2-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
persistence: persistence:
data: data:
enabled: true enabled: true
size: 1Gi # Optional: to keep job queues persistent, use a
# Optional: Set this to persistentVolumeClaim to keep job queues persistent # persistentVolumeClaim with size, accessMode and storageClass
type: emptyDir type: emptyDir
accessMode: ReadWriteOnce
storageClass: longhorn
# Immich components # Immich components
@@ -133,11 +127,9 @@ spec:
readOnly: true readOnly: true
cache: cache:
enabled: true enabled: true
size: 10Gi # Optional: to avoid downloading the ML models every start, use
# Optional: Set this to persistentVolumeClaim to avoid downloading the ML models every start. # a persistentVolumeClaim with size, accessMode and storageClass
type: emptyDir type: emptyDir
accessMode: ReadWriteMany
# storageClass: longhorn
destination: destination:
server: https://kubernetes.default.svc server: https://kubernetes.default.svc
-132
View File
@@ -1,132 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: httpd-conf
data:
httpd.conf: |
ServerRoot "/usr/local/apache2"
Listen 8002
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule headers_module modules/mod_headers.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule http2_module modules/mod_http2.so
LoadModule proxy_http2_module modules/mod_proxy_http2.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule authopenid_module /usr/lib/apache2/modules/mod_auth_openid.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule rewrite_module modules/mod_rewrite.so
MaxConnectionsPerChild 30
<IfModule unixd_module>
User www-data
Group www-data
</IfModule>
ServerAdmin webmaster@scoutingmbg.nl
ServerName foto.scoutingmbg.nl
<Directory />
AllowOverride none
Require all denied
</Directory>
DocumentRoot "/usr/local/apache2/htdocs"
<Directory "/usr/local/apache2/htdocs">
Options Indexes FollowSymLinks
AllowOverride None
</Directory>
<Location />
AuthType OpenID
require valid-user
AuthOpenIDDBLocation /tmp/mod_auth_openid.db
AuthOpenIDTrusted ^https://login.scouting.nl/.*$ ^https://login.scouting.nl/user/$
AuthOpenIDUseCookie On
AuthOpenIDCookieName open_id_session_id
AuthOpenIDCookieLifespan 31536000
AuthOpenIDServerName https://foto.scoutingmbg.nl
AuthOpenIDSingleIdP https://login.scouting.nl/user/
AuthOpenIDSecureCookie On
</Location>
<Location "/_app">
AuthType None
Require all granted
</Location>
AllowEncodedSlashes NoDecode
ProxyPreserveHost Off
RewriteEngine on
RewriteRule ^/?photos/?$ /sharing [R=301,L]
ProxyPass / http://helm-immich-server:2283/
ProxyPassReverse / http://helm-immich-server:2283/
RewriteEngine on
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "ws://helm-immich-server:2283/$1" [P,L]
# must be bigger than pingInterval (25s by default) + pingTimeout (20s by default)
ProxyTimeout 60
ProxyRequests off
Header always set Access-Control-Allow-Origin "*"
Header always set Access-Control-Max-Age "1000"
Header always set Access-Control-Allow-Headers "x-requested-with, Content-Type, origin, authorization, accept, client-security-token"
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
<Files ".ht*">
Require all denied
</Files>
ErrorLog /proc/self/fd/2
LogLevel warn
<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
CustomLog /proc/self/fd/1 common
</IfModule>
<IfModule alias_module>
ScriptAlias /cgi-bin/ "/usr/local/apache2/cgi-bin/"
</IfModule>
<Directory "/usr/local/apache2/cgi-bin">
AllowOverride None
Options None
Require all granted
</Directory>
<IfModule headers_module>
RequestHeader unset Proxy early
</IfModule>
<IfModule mime_module>
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
</IfModule>
<IfModule proxy_html_module>
Include conf/extra/proxy-html.conf
</IfModule>
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
+4 -2
View File
@@ -9,10 +9,12 @@ spec:
sectionName: https-foto sectionName: https-foto
hostnames: hostnames:
- foto.scoutingmbg.nl - foto.scoutingmbg.nl
# Straight to immich, which logs people in with Scouts Online through
# authentik (docs/immich-scouts-online.md).
rules: rules:
- backendRefs: - backendRefs:
- name: sol-auth-svc - name: helm-immich-server
port: 8002 port: 2283
# No limit on the whole request, as with nginx. Envoy's default of 15s # No limit on the whole request, as with nginx. Envoy's default of 15s
# cuts off uploads and long-lived connections. # cuts off uploads and long-lived connections.
timeouts: timeouts:
+46
View File
@@ -0,0 +1,46 @@
# immich's settings, rendered with the OAuth client secret from Vault. With
# a config file the settings are read-only in the admin UI. `{{`...`}}` keeps
# immich's own template syntax away from ESO's.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: immich-config
namespace: immich
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: immich-config
creationPolicy: Owner
template:
engineVersion: v2
data:
immich-config.yaml: |
server:
externalDomain: https://foto.scoutingmbg.nl
storageTemplate:
enabled: true
template: "{{`{{y}}/{{#if album}}{{album}}{{else}}Other/{{MM}}{{/if}}/{{filename}}`}}"
# authentik passes on the Scouts Online login and sets the
# immich_role and immich_quota claims (authentik/blueprint-scouting.yaml).
oauth:
enabled: true
issuerUrl: https://authentik.rubenhensen.nl/application/o/immich/
clientId: immich
clientSecret: "{{ .client_secret }}"
scope: openid email profile immich
buttonText: Inloggen met Scouts Online
autoRegister: true
# /auth/login?autoLaunch=0 still shows the login page
autoLaunch: true
# Admins log in through authentik too. To get back in without it,
# enable this again (immich-admin can't change a config file).
passwordLogin:
enabled: false
data:
- secretKey: client_secret
remoteRef:
key: kv/immich-oidc
property: client_secret
+242
View File
@@ -0,0 +1,242 @@
# Shares the Scouting MBG photos with every immich user. immich has no user
# groups (on its roadmap as "Soon"), so this does it per user:
# - albums that uploaders (kaderleden) create get the group account as
# editor, unless their name starts with "Privé" or "Prive";
# - every album the group account owns or edits gets every user, as editor
# when they may upload (quota other than 0, which authentik sets from
# Scouts Online) and as viewer otherwise;
# - the group account's library is shared as a partner and shown in
# everyone's timeline, so search covers it (immich only searches your own
# and your timeline partners' photos);
# - the invitation notifications for albums older than a week are hidden,
# or a new member would get one per album;
# - sessions of OAuth users older than SESSION_MAX_AGE_DAYS are ended, so
# someone who left the group loses access at the next login at Scouts
# Online.
# Albums and partners go through the API. Joining other people's albums
# (the API only lets an album's owner or editors share it), the timeline
# flag (which only the receiving user can set), the notifications and the
# sessions go straight to the database.
apiVersion: v1
kind: ConfigMap
metadata:
name: share-sync
namespace: immich
data:
sync.py: |
import json
import os
import subprocess
import urllib.request
API = os.environ["IMMICH_URL"].rstrip("/") + "/api"
KEY = os.environ["IMMICH_API_KEY"]
GROUP_EMAIL = os.environ["GROUP_EMAIL"]
SKIP_EMAILS = set(os.environ.get("SKIP_EMAILS", "").split())
SESSION_MAX_AGE_DAYS = int(os.environ.get("SESSION_MAX_AGE_DAYS", "30"))
DRY_RUN = os.environ.get("DRY_RUN") == "1"
def api(method, path, body=None):
req = urllib.request.Request(
API + path,
method=method,
data=None if body is None else json.dumps(body).encode(),
headers={"x-api-key": KEY, "Accept": "application/json", "Content-Type": "application/json"},
)
with urllib.request.urlopen(req, timeout=60) as resp:
data = resp.read()
return json.loads(data) if data else None
def sql(query, **params):
"""Runs query through psql; :'name' in it is params[name], quoted."""
args = ["psql", "-X", "-q", "-At", "-F", "\t", "-v", "ON_ERROR_STOP=1"]
for name, value in params.items():
args += ["-v", f"{name}={value}"]
out = subprocess.run(args, input=query, capture_output=True, text=True, check=True).stdout
return [line.split("\t") for line in out.splitlines() if line]
def change(what, call):
print(("would " if DRY_RUN else "") + what, flush=True)
if not DRY_RUN:
call()
users = sql('SELECT id, email, name, coalesce("quotaSizeInBytes"::text, \'\') FROM "user" WHERE "deletedAt" IS NULL')
group = next(u for u in users if u[1] == GROUP_EMAIL)
group_id, group_name = group[0], group[2]
members = {u[0]: u[1] for u in users if u[0] != group_id and u[1] not in SKIP_EMAILS}
# A quota of 0 bytes is how immich (and authentik's immich_quota claim)
# says: no uploads.
wanted = {u[0]: "viewer" if u[3] == "0" else "editor" for u in users if u[0] in members}
# Albums of uploaders that the group account isn't in yet
UNJOINED = """
FROM album_user au
JOIN "user" u ON u.id = au."userId"
JOIN album a ON a.id = au."albumId"
WHERE au.role = 'owner' AND u.id != :'group'
AND u."deletedAt" IS NULL AND a."deletedAt" IS NULL
AND (u."quotaSizeInBytes" IS NULL OR u."quotaSizeInBytes" > 0)
AND a."albumName" !~* '^\\s*priv[eé]'
AND NOT EXISTS (SELECT 1 FROM album_user g WHERE g."albumId" = a.id AND g."userId" = :'group')
"""
if DRY_RUN:
for (name,) in sql("SELECT a.\"albumName\"" + UNJOINED, group=group_id):
print(f"would join {name}", flush=True)
else:
for (name,) in sql(
f"""
WITH joined AS (
INSERT INTO album_user ("albumId", "userId", role)
SELECT a.id, :'group', 'editor' {UNJOINED}
RETURNING "albumId"
)
SELECT a."albumName" FROM joined JOIN album a ON a.id = joined."albumId"
""",
group=group_id,
):
print(f"join {name}", flush=True)
for album in api("GET", "/albums"):
roles = {au["user"]["id"]: au["role"] for au in album["albumUsers"]}
if roles.get(group_id) not in ("owner", "editor"):
continue
add = [{"userId": uid, "role": role} for uid, role in wanted.items() if uid not in roles]
if add:
change(
f"add {len(add)} users to {album['albumName']}",
lambda: api("PUT", f"/albums/{album['id']}/users", {"albumUsers": add}),
)
for uid, role in wanted.items():
if roles.get(uid) in ("viewer", "editor") and roles[uid] != role:
change(
f"make {members[uid]} {role} of {album['albumName']}",
lambda: api("PUT", f"/albums/{album['id']}/user/{uid}", {"role": role}),
)
partners = {p["id"] for p in api("GET", "/partners?direction=shared-by")}
for uid, email in members.items():
if uid not in partners:
change(f"share the library with {email}", lambda: api("POST", "/partners", {"sharedWithId": uid}))
if DRY_RUN:
raise SystemExit(0)
sql(
'UPDATE partner SET "inTimeline" = true WHERE "sharedById" = :\'group\' AND NOT "inTimeline"',
group=group_id,
)
sql(
"""
UPDATE notification SET "deletedAt" = now()
WHERE type = 'AlbumInvite' AND "deletedAt" IS NULL AND "readAt" IS NULL
AND description LIKE :'sender' || ' shared an album%'
-- immich stores data as a JSON string inside the jsonb
AND ((data #>> '{}')::jsonb ->> 'albumId')::uuid IN (SELECT id FROM album WHERE "createdAt" < now() - interval '7 days')
""",
sender=group_name,
)
sql(
"""
DELETE FROM session
WHERE "createdAt" < now() - make_interval(days => :days)
AND "userId" IN (SELECT id FROM "user" WHERE "oauthId" IS NOT NULL)
""",
days=SESSION_MAX_AGE_DAYS,
)
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: immich-share
namespace: immich
spec:
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
refreshInterval: 15m
target:
name: immich-share
creationPolicy: Owner
data:
# API key of the group account, with album.read, albumUser.create,
# albumUser.update, partner.read and partner.create
- secretKey: api_key
remoteRef:
key: kv/immich-share
property: api_key
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: share-sync
namespace: immich
spec:
schedule: "*/10 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 0
activeDeadlineSeconds: 1800
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 26
containers:
- name: sync
# The database's own image: it has python3 and psql.
image: ghcr.io/tensorchord/cloudnative-vectorchord:17.11-1.1.1
command: ["python3", "/script/sync.py"]
env:
- name: IMMICH_URL
value: http://helm-immich-server:2283
- name: IMMICH_API_KEY
valueFrom:
secretKeyRef:
name: immich-share
key: api_key
- name: GROUP_EMAIL
value: fotoupload@scoutingmbg.nl
# The shared viewing account from before Scouts Online
- name: SKIP_EMAILS
value: foto@scoutingmbg.nl
- name: PGHOST
value: immich-postgres-17-rw
- name: PGSSLMODE
value: require
# album names are UTF-8 (and so is the Privé check)
- name: PGCLIENTENCODING
value: UTF8
- name: PGDATABASE
value: immich
- name: PGUSER
valueFrom:
secretKeyRef:
name: immich-postgres-17-app
key: username
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: immich-postgres-17-app
key: password
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 256Mi
volumeMounts:
- name: script
mountPath: /script
volumes:
- name: script
configMap:
name: share-sync
-40
View File
@@ -1,40 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: sol-auth
spec:
replicas: 1
selector:
matchLabels:
app: sol-auth
template:
metadata:
labels:
app: sol-auth
spec:
# securityContext:
# runAsUser: 1000
# runAsGroup: 1000
# fsGroup: 1000
containers:
- image: hebury/sol-auth:2.3
name: sol-auth
resources:
requests:
memory: "50Mi"
limits:
memory: "500Mi"
ports:
- name: http
containerPort: 8002
hostPort: 8002
protocol: TCP
volumeMounts:
- name: config-volume
mountPath: /config/
volumes:
- name: config-volume
configMap:
name: httpd-conf
restartPolicy: Always
terminationGracePeriodSeconds: 5
-13
View File
@@ -1,13 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: sol-auth-svc
spec:
ports:
- name: http-svc
port: 8002
targetPort: 8002
selector:
app: sol-auth
status:
loadBalancer: {}
-13
View File
@@ -1,13 +0,0 @@
FROM httpd:2.4-bullseye
RUN apt-get update && \
apt-get install --no-install-recommends -y \
ca-certificates libapache2-mod-auth-openid
RUN touch /tmp/mod_auth_openid.db && chown www-data /tmp/mod_auth_openid.db
RUN mkdir /config
RUN touch /config/httpd.conf
RUN rm /usr/local/apache2/conf/httpd.conf
RUN ln -s /config/httpd.conf /usr/local/apache2/conf/httpd.conf
# how to build: docker build . -t hebury/sol-auth:2.3
# how to push: docker push hebury/sol-auth:2.3
# do not remember how to build for arm atm.
+1 -1
View File
@@ -90,7 +90,7 @@ let
conditions = [ "[CONNECTED] == true" "[CERTIFICATE_EXPIRATION] > 168h" ]; conditions = [ "[CONNECTED] == true" "[CERTIFICATE_EXPIRATION] > 168h" ];
} }
(gatusHttp "Scouting MBG" "scoutingmbg.nl" "https://scoutingmbg.nl" // gatusAlert) (gatusHttp "Scouting MBG" "scoutingmbg.nl" "https://scoutingmbg.nl" // gatusAlert)
(gatusHttp "Scouting MBG" "foto.scoutingmbg.nl" "https://foto.scoutingmbg.nl/_app" // gatusAlert) (gatusHttp "Scouting MBG" "foto.scoutingmbg.nl" "https://foto.scoutingmbg.nl/api/server/ping" // gatusAlert)
({ ({
name = "s2 (homelab)"; name = "s2 (homelab)";
group = "watchers"; group = "watchers";
+16
View File
@@ -42,6 +42,22 @@ spec:
config: config:
region: minio region: minio
features: EnableCSI features: EnableCSI
# Kopia maintenance ran every hour for each of the 15 repositories,
# and once a day as a full run of several minutes that writes its
# cache to the node's only SSD, which stalls dqlite and makes
# controllers lose their leases (docs/upgrade-k8s-1.33.md). Daily
# and throttled instead. Existing BackupRepositories keep their own
# maintenanceFrequency; they were patched to 24h by hand.
defaultRepoMaintainFrequency: 24h
repositoryMaintenanceJob:
repositoryConfigData:
global:
keepLatestMaintenanceJobs: 3
podResources:
cpuRequest: 100m
cpuLimit: 500m
memoryRequest: 256Mi
memoryLimit: 2Gi
credentials: credentials:
# Use existing secret created by External Secrets Operator # Use existing secret created by External Secrets Operator
# existingSecret: velero-minio-credentials # existingSecret: velero-minio-credentials