TriliumNext: User=0 for keep-id, and our own health check

The first run of this playbook against emily failed, and all three reasons are
worth recording because none of them is visible from the files alone.

keep-id runs the container *process* as the uid it maps, so
UserNS=keep-id:uid=1000,gid=1000 started the container as `node` rather than root
-- and the image's start-docker.sh is written to start as root, ending with
`exec su -c "node ./main.mjs" node`. `su` from a non-root uid asks for a password
the node account does not have, so the container died with "su: Authentication
failure" and Restart=always turned that into a crash loop. User=0 forces the
process back to root and changes nothing about the mapping: node is still tanshu,
so the app is still what writes the data directory and it stays tanshu's.
nextcloud.container carries the same pair for the same reason.

The health check had to be defined rather than inherited. `latest` floated between
the two pulls -- monoco runs v0.95.0 (built 2025-06-15) and emily pulled v0.106.0
(built 2026-09-25) -- and v0.106.0 ships no HEALTHCHECK where v0.95.0 had
docker_healthcheck.cjs, so `podman healthcheck run` failed with "no defined
healthcheck". The unit now defines HealthCmd= against /api/health-check, the
endpoint the app advertises itself in the healthcheck-url file it writes into the
data directory, and the image has curl for it (but not wget).

The verify and health-check steps are folded in with the start, because they fail
the same way and want the same evidence. `systemctl start` reports success as soon
as systemd holds the unit, so a container that dies a second later surfaced only
as "activating" from `is-active`; the reason -- that "su: Authentication failure"
-- was in the journal and nowhere the play looked. The rescue now dumps status,
journal and podman logs for any of the three, which is how the su failure was
found in the first place.

The version float becomes a documented risk rather than a hypothetical: the
cutover upgrades the app as well as the runtime, and the first start migrates
document.db forward. That is one-way for the copy on emily, so the readme compares
digests before the first start, keeps monoco's own tree as the rollback, and notes
that pinning a tag here would decouple the two changes.

Verified on emily rather than by inspection: the rendered unit was run first as a
throwaway Quadlet unit against a scratch volume, then the playbook itself ran to
ok=22/changed=6/failed=0 with the mount assertion passing, `healthcheck run`
returning 0, podman reporting healthy, and the app's files landing in
~/podman-data/trilium owned by tanshu. The data directory was emptied afterwards
and the unit left stopped, ready for the copy from monoco, which has not run yet.

trilium/roles/trilium/tasks/main.yaml

@@ -64,7 +64,16 @@
     - Reload systemd user units
     - Restart Trilium
 
-- name: Start Trilium
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+#
+# All three steps are in one block because they fail the same way and want the
+# same evidence. `systemctl start` reports success as soon as it has handed the
+# unit to systemd, so a container that dies a second later — `User=0` missing, a
+# network that was never attached, an unwritable data directory — surfaces here
+# as `activating`, or as a health check that never passes, with the real reason
+# only in the journal. Restart=always makes that a crash loop, not an error.
+- name: Start and verify Trilium
   block:
     - name: Start Trilium
       ansible.builtin.systemd_service:
@@ -72,10 +81,35 @@
         scope: user
         state: started
         daemon_reload: true
+
+    # Retried rather than checked once: the container is coming up, and podman
+    # reports `activating` for the interval between systemd starting it and the
+    # container's own readiness notification.
+    - name: Verify Trilium is running
+      ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
+      register: trilium_active
+      until: trilium_active.stdout == 'active'
+      retries: 12
+      delay: 5
+      changed_when: false
+
+    # `is-active` says the container is running; it does not say the app can
+    # serve. The unit defines the probe (HealthCmd=, see the template) because
+    # the image ships none, so this runs exactly that command once,
+    # synchronously, and fails while the app is still migrating or not yet
+    # listening.
+    - name: Wait for Trilium to pass its health check
+      ansible.builtin.command: "podman healthcheck run {{ container_name }}"
+      register: trilium_health
+      until: trilium_health.rc == 0
+      retries: 24
+      delay: 5
+      changed_when: false
   rescue:
     # systemd reports "control process exited with error code" and nothing
-    # else, which is useless for the usual failures (an unattached network, an
-    # unwritable data directory, a bad volume). Pull the real reason out.
+    # else, which is useless for the usual failures. The journal is where the
+    # entrypoint's own output is — that is where "su: Authentication failure"
+    # or a failed bind mount shows up.
     - name: Collect Trilium diagnostics
       ansible.builtin.command: "{{ item }}"
       loop:
@@ -94,30 +128,6 @@
       ansible.builtin.fail:
         msg: "Trilium did not start — see the diagnostics above."
 
-# Quadlet honours [Install] WantedBy= when it generates the unit, so
-# `systemctl --user enable` is neither needed nor possible on a generated unit.
-- name: Verify Trilium is running
-  ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
-  register: trilium_active
-  changed_when: false
-  failed_when: trilium_active.stdout != 'active'
-
-# `is-active` says the container is running; it does not say the app can serve.
-# `Restart=always` means even a crash-looping container is reported active.
-#
-# Unlike audiobookshelf, this image ships its own HEALTHCHECK
-# (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`, which the
-# Docker-era container reported healthy with), so there is no HealthCmd= in the
-# unit to run here — `podman healthcheck run` executes the image's own probe and
-# this depends on no path the play has to know about.
-- name: Wait for Trilium to pass its own health check
-  ansible.builtin.command: "podman healthcheck run {{ container_name }}"
-  register: trilium_health
-  until: trilium_health.rc == 0
-  retries: 24
-  delay: 5
-  changed_when: false
-
 # The check that catches a missing or mistyped Volume= line, which is the one
 # failure here that is silent: the app would run perfectly well with document.db
 # in the container's writable layer — or in an anonymous volume, if the image

trilium/roles/trilium/templates/trilium.container.j2

@@ -15,14 +15,24 @@ Wants=network-online.target
 ContainerName={{ container_name }}
 Image={{ image }}
 
-# The entrypoint starts as container-root and drops to the image's `node` user,
-# and `node` is uid 1000 — which is also the uid that owns every file in the
-# Docker-era data tree on monoco (docs/podman-compat-audit.md §3.2). Mapping
-# container uid 1000 onto {{ podman_user }} keeps {{ data_dir }} owned by
-# {{ podman_user }} on the host, so tar/rsync backups keep working and no
-# subuid-owned files appear.
+# The image's start-docker.sh drops privileges to the `node` user, and `node` is
+# uid 1000 — which is also the uid that owns every file in the Docker-era data
+# tree on monoco (docs/podman-compat-audit.md §3.2). Mapping container uid 1000
+# onto {{ podman_user }} keeps {{ data_dir }} owned by {{ podman_user }} on the
+# host, so tar/rsync backups keep working and no subuid-owned files appear.
 UserNS=keep-id:uid=1000,gid=1000
 
+# keep-id also runs the container *process* as that mapped uid, i.e. as `node`,
+# while start-docker.sh is written to start as root: it ends with
+# `exec su -c "node ./main.mjs" node`, and `su` from a non-root uid asks for a
+# password the `node` account does not have. The container then dies with
+# "su: Authentication failure" and Restart=always turns that into a crash loop.
+# Forcing the process back to root restores what the script expects and changes
+# nothing about the mapping above: `node` is still {{ podman_user }}, so the app
+# is still what writes {{ data_dir }} and it stays {{ podman_user }}'s.
+# (nextcloud.container needs the same two lines, for the same reason.)
+User=0
+
 Network={{ network_name }}.network
 
 # The image's data directory. This line is the only thing keeping state out of
@@ -37,9 +47,18 @@ Volume={{ data_dir }}:{{ container_data_dir }}
 # `Environment=TZ=Asia/Kolkata` here if you want local-time log filenames; until
 # then the container runs on UTC, exactly as it does on monoco.
 #
-# No HealthCmd= either, unlike audiobookshelf: this image ships its own
-# HEALTHCHECK (docker_healthcheck.cjs), so `podman healthcheck run` executes the
-# image's own probe and the unit only has to not get in the way.
+# The image ships no HEALTHCHECK — v0.106.0 dropped the docker_healthcheck.cjs
+# that the v0.95.0 still running on monoco had — so without this podman has
+# nothing to run and the play's `podman healthcheck run` fails with "no defined
+# healthcheck". This probes the endpoint the app advertises itself: it writes
+# healthcheck-url, pointing at /api/health-check, into the data directory on
+# startup. Deliberately no HealthOnFailure=: that would let a temporarily slow
+# start kill and restart a healthy server.
+HealthCmd={{ health_cmd }}
+HealthInterval=30s
+HealthTimeout=5s
+HealthRetries=3
+HealthStartPeriod=30s
 
 [Service]
 Restart=always