TriliumNext: User=0 for keep-id, and our own health check

The first run of this playbook against emily failed, and all three reasons are
worth recording because none of them is visible from the files alone.

keep-id runs the container *process* as the uid it maps, so
UserNS=keep-id:uid=1000,gid=1000 started the container as `node` rather than root
-- and the image's start-docker.sh is written to start as root, ending with
`exec su -c "node ./main.mjs" node`. `su` from a non-root uid asks for a password
the node account does not have, so the container died with "su: Authentication
failure" and Restart=always turned that into a crash loop. User=0 forces the
process back to root and changes nothing about the mapping: node is still tanshu,
so the app is still what writes the data directory and it stays tanshu's.
nextcloud.container carries the same pair for the same reason.

The health check had to be defined rather than inherited. `latest` floated between
the two pulls -- monoco runs v0.95.0 (built 2025-06-15) and emily pulled v0.106.0
(built 2026-09-25) -- and v0.106.0 ships no HEALTHCHECK where v0.95.0 had
docker_healthcheck.cjs, so `podman healthcheck run` failed with "no defined
healthcheck". The unit now defines HealthCmd= against /api/health-check, the
endpoint the app advertises itself in the healthcheck-url file it writes into the
data directory, and the image has curl for it (but not wget).

The verify and health-check steps are folded in with the start, because they fail
the same way and want the same evidence. `systemctl start` reports success as soon
as systemd holds the unit, so a container that dies a second later surfaced only
as "activating" from `is-active`; the reason -- that "su: Authentication failure"
-- was in the journal and nowhere the play looked. The rescue now dumps status,
journal and podman logs for any of the three, which is how the su failure was
found in the first place.

The version float becomes a documented risk rather than a hypothetical: the
cutover upgrades the app as well as the runtime, and the first start migrates
document.db forward. That is one-way for the copy on emily, so the readme compares
digests before the first start, keeps monoco's own tree as the rollback, and notes
that pinning a tag here would decouple the two changes.

Verified on emily rather than by inspection: the rendered unit was run first as a
throwaway Quadlet unit against a scratch volume, then the playbook itself ran to
ok=22/changed=6/failed=0 with the mount assertion passing, `healthcheck run`
returning 0, podman reporting healthy, and the app's files landing in
~/podman-data/trilium owned by tanshu. The data directory was emptied afterwards
and the unit left stopped, ready for the copy from monoco, which has not run yet.

trilium/readme.md

@@ -52,10 +52,23 @@ misspelling survives in exactly one place, deliberately: the **source path**
 `/var/lib/trillium` in the copy commands below, which is what the old host
 actually has.
 
-The image's data directory is `/home/node/trilium-data`, and the image ships its
-own `HEALTHCHECK` (`docker_healthcheck.cjs`), which the Docker-era container
-reported healthy with. The unit therefore defines no `HealthCmd=` — unlike
-[audiobookshelf/](../audiobookshelf/readme.md), whose image has none.
+Two things about the image had to be settled against `emily` rather than read off
+`monoco`, because `latest` floated by far more than a patch release between the
+two pulls:
+
+| | monoco (running) | emily (pulled) |
+|---|---|---|
+| version | v0.95.0, built 2025-06-15 | **v0.106.0**, built 2026-09-25 |
+| `HEALTHCHECK` | ships one (`docker_healthcheck.cjs`) | **ships none** |
+
+So the unit defines its own `HealthCmd=`, probing the endpoint the app advertises
+in the `healthcheck-url` file it writes into the data directory, and the cutover
+is **also an app upgrade**: TriliumNext migrates `document.db` forward on first
+start. That migration is one-way for the copy on emily, which is why the digests
+below are compared *before* the first start and monoco's own tree is left alone.
+
+The image also runs as root and drops to `node` with `su`, which is what `User=0`
+exists for — see [Settings](#settings).
 
 ## Ordering
 
@@ -69,7 +82,10 @@ this app has no database.
 
 This playbook does **not** move the data. Copy it first (see
 ["Migrating the data"](#migrating-the-data-from-monoco)) so the first start opens
-the real `document.db`, rather than creating an empty document beside it.
+the real `document.db`, rather than creating an empty document beside it. If it
+does start against an empty directory — which the play warns about — that empty
+document and its log have to be cleared before the copy, and the copy's steps say
+how.
 
 ## Layout
 
@@ -92,6 +108,7 @@ Quadlet units and state:
 | `~/podman-data/trilium/document.db` | the document, with `-wal` / `-shm` beside it |
 | `~/podman-data/trilium/config.ini` | the app's settings — `port`, `trustedReverseProxy`, `noBackup` |
 | `~/podman-data/trilium/session_secret.txt` | the session signing secret |
+| `~/podman-data/trilium/healthcheck-url` | the app's own health endpoint, written on every start. `HealthCmd=` in the unit probes it |
 | `~/podman-data/trilium/log/` | one log file per day, 91 MB of the 106 MB tree |
 | `~/podman-data/trilium/backup/` | Trilium's own zipped `document.db` backups |
 | `~/podman-data/trilium/tmp/` | scratch, empty at rest |
@@ -163,6 +180,7 @@ port-floor work Caddy itself needs.
 | `container_data_dir` | `/home/node/trilium-data` — the image's data directory |
 | `log_dir` / `tmp_dir` / `backup_dir` | The three app-managed subdirectories, created at 0700 |
 | `container_port` | `8080`, matching `config.ini`'s `port` |
+| `health_cmd` | The unit's `HealthCmd=`, since the image ships none |
 | `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |
 
 The three `podman_*` / `quadlet_dir` vars are **duplicated** from
@@ -178,12 +196,23 @@ Like Caddy, `podman_user` is derived from the connection and the play carries no
 connecting as `root`**; the prerequisites, including that check, live in
 `caddy/playbook.yml`.
 
-`UserNS=keep-id:uid=1000,gid=1000` — the entrypoint starts as container-root and
-drops to the image's `node` user, which is uid 1000, and uid 1000 is what owns
-every file in the Docker-era tree on monoco
-([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). Mapping
-container uid 1000 onto `podman_user` keeps `~/podman-data/trilium` owned by
-`tanshu` and leaves no subuid-owned files to trip over backups.
+Two lines decide who the container is, and **both are needed** — this is the one
+thing that actually broke the first deployment of this playbook, and it is the
+same pair [nextcloud/](../nextcloud/readme.md) needs:
+
+- **`UserNS=keep-id:uid=1000,gid=1000`** — `1000` is `node`, the user the app runs
+  as, and it is also what owns every file in the Docker-era tree on monoco
+  ([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `keep-id`
+  maps that one uid onto `podman_user`, so `~/podman-data/trilium` stays owned by
+  `tanshu` and plain `tar`, `rm -rf` and the backup scripts keep working on it.
+- **`User=0`** — `keep-id` also runs the container *process* as that mapped uid,
+  i.e. as `node`, while the image's `start-docker.sh` is written to start as root:
+  it ends with `exec su -c "node ./main.mjs" node`, and `su` from a non-root uid
+  asks for a password the `node` account does not have. Without `User=0` the
+  container dies with `su: Authentication failure` and `Restart=always` turns that
+  into a crash loop. Forcing the process back to root restores what the script
+  expects and changes nothing about the mapping above: `node` is still `tanshu`,
+  so the app is still what writes the data directory and it stays `tanshu`'s.
 
 **There is no environment file here**, unlike `bitwarden/` and `tandoor/`. This
 deployment holds no secrets Ansible has to pass in: `config.ini` and
@@ -243,6 +272,11 @@ ssh monoco 'docker stop trillium'
 # 3. Create the destination — this is the mount target, /home/node/trilium-data.
 ssh emily 'mkdir -p ~/podman-data/trilium'
 
+#    If this playbook has already run against that directory — it warns when it
+#    does — the app has left a document of its own there: an empty one, plus its
+#    log and healthcheck-url. Clear it, or step 5's digests will not match:
+# ssh emily 'systemctl --user stop trilium.service; find ~/podman-data/trilium -mindepth 1 -delete'
+
 # 4. Copy. `sudo` on the source side is not optional here: log/, tmp/ and
 #    backup/ are 0700 and unreadable as tanshu, and backup/ is state.
 #
@@ -317,7 +351,7 @@ journalctl --user -u trilium -f
 podman logs trilium
 
 # What the playbook checks for you:
-podman healthcheck run trilium          # the image's own probe
+podman healthcheck run trilium          # the unit's own HealthCmd=, not the image's
 
 # The unit's mount is real and not the container's writable layer:
 podman inspect trilium --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'
@@ -358,24 +392,41 @@ Then reboot the host and confirm the unit returns unattended — without
   `/root/trilium-data` and predates TriliumNext's schema migrations; pointed at
   this `document.db` it is at best a failure to start. The unit pins
   `docker.io/triliumnext/notes:latest`.
-- **The image's own health check is used.** There is no `HealthCmd=` in the unit
-  and there does not need to be: the image ships one
-  (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`), which is why the
-  Docker-era container showed `healthy`. `is-active` alone cannot tell a serving
-  server from a crash-looping one under `Restart=always`, which is why the play
-  runs the healthcheck rather than trusting it.
+- **The unit's health check is its own.** Unlike the v0.95.0 on monoco, the
+  v0.106.0 image ships no `HEALTHCHECK`, so `podman healthcheck run` would fail
+  with "no defined healthcheck"; the unit probes the endpoint the app advertises
+  in `healthcheck-url` instead. `is-active` alone cannot tell a serving server
+  from a crash-looping one under `Restart=always`, which is why the play runs the
+  healthcheck rather than trusting it.
+- **`keep-id` runs the container as the mapped uid, and this image needs root.**
+  The full explanation is in [Settings](#settings); the short version is that
+  without `User=0` the image's own `su node` cannot work and the unit crash-loops.
+  The three failures this playbook has actually produced, and what each means:
+
+  | In the journal | Cause |
+  |---|---|
+  | `Password:` then `su: Authentication failure` | the container process is not root, so the image's privilege drop fails — `User=0` is missing |
+  | `no defined healthcheck` from `podman healthcheck run` | the image ships none and the unit's `HealthCmd=` is missing |
+  | `DB not initialized, please visit setup page` | the app is serving a **new, empty** document — the copy from monoco has not run (or was wiped) |
+
 - **The mount check is a mount check, not a file check.** Audiobookshelf can
   assert its database appeared on the host, because the app creates it. Here the
   migration puts `document.db` on the host *before* the app runs, so its presence
-  proves nothing — the play asserts against `podman inspect` instead. That is the
-  one failure that is otherwise silent: with a missing `Volume=` line the app
-  runs perfectly and loses the document on the next recreate.
-- **`latest` floats.** The image measured for this playbook is **v0.95.0**
-  (`org.opencontainers.image.version`), built 2025-06-15, revision `b9ae2265`.
-  Pin a digest if reproducible deploys matter —
-  `podman image inspect docker.io/triliumnext/notes:latest --format '{{.Digest}}'`
-  — since an upgrade can run schema migrations on `document.db` on first start.
-  Take Trilium's own backup (or stop-and-copy) before upgrading.
+  proves nothing — the play asserts against `podman inspect` instead, and checks
+  the *source* as well as the destination, because a destination-only check is
+  satisfied by an anonymous volume mounted where the bind mount should be. That is
+  the one failure that is otherwise silent: the app runs perfectly and loses the
+  document on the next recreate.
+- **`latest` floats, and it moved here.** monoco runs **v0.95.0** (built
+  2025-06-15, revision `b9ae2265`) and the pull on emily is **v0.106.0** (built
+  2026-09-25, revision `c0fdceba`) — so this cutover upgrades the app as well as
+  the runtime, and the first start migrates `document.db` forward. That is
+  one-way *for the copy on emily*; monoco's own tree is untouched and remains the
+  rollback. Pin a version tag or a digest in `vars/default.yml` if the migration
+  should change only the runtime and leave the upgrade to a separate, deliberate
+  step (`podman image inspect docker.io/triliumnext/notes:latest --format
+  '{{.Digest}}'`). Take Trilium's own backup, or a stop-and-copy, before any
+  upgrade.
 - **`trustedReverseProxy=false` in `config.ini`.** Trilium does not trust
   `X-Forwarded-For`, so its own logs see the proxy rather than the client. That
   is the existing behaviour, carried across unchanged; set it to `true` (or to
@@ -403,7 +454,9 @@ Then reboot the host and confirm the unit returns unattended — without
 - **DNS.** `notes.tanshu.com` must resolve to `emily` before Caddy can obtain a
   certificate for it. The certificates and ACME account key from monoco's
   `/var/lib/caddy/data` are **not** part of this migration — Caddy's own playbook
-  and readme cover that.
+  and readme cover that. Until the name moves, emily's Caddy retries the ACME
+  order and logs the failures; that is expected while monoco still holds the name,
+  and the site keeps being served from monoco.
 - **Root's crontab cannot `podman exec`** into another user's rootless
   containers, so anything scheduled belongs in a **systemd user timer** next to
   the Quadlet units. Trilium's own `backup/` mechanism runs *inside* the app

trilium/roles/trilium/tasks/main.yaml

@@ -64,7 +64,16 @@
     - Reload systemd user units
     - Restart Trilium
 
-- name: Start Trilium
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+#
+# All three steps are in one block because they fail the same way and want the
+# same evidence. `systemctl start` reports success as soon as it has handed the
+# unit to systemd, so a container that dies a second later — `User=0` missing, a
+# network that was never attached, an unwritable data directory — surfaces here
+# as `activating`, or as a health check that never passes, with the real reason
+# only in the journal. Restart=always makes that a crash loop, not an error.
+- name: Start and verify Trilium
   block:
     - name: Start Trilium
       ansible.builtin.systemd_service:
@@ -72,10 +81,35 @@
         scope: user
         state: started
         daemon_reload: true
+
+    # Retried rather than checked once: the container is coming up, and podman
+    # reports `activating` for the interval between systemd starting it and the
+    # container's own readiness notification.
+    - name: Verify Trilium is running
+      ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
+      register: trilium_active
+      until: trilium_active.stdout == 'active'
+      retries: 12
+      delay: 5
+      changed_when: false
+
+    # `is-active` says the container is running; it does not say the app can
+    # serve. The unit defines the probe (HealthCmd=, see the template) because
+    # the image ships none, so this runs exactly that command once,
+    # synchronously, and fails while the app is still migrating or not yet
+    # listening.
+    - name: Wait for Trilium to pass its health check
+      ansible.builtin.command: "podman healthcheck run {{ container_name }}"
+      register: trilium_health
+      until: trilium_health.rc == 0
+      retries: 24
+      delay: 5
+      changed_when: false
   rescue:
     # systemd reports "control process exited with error code" and nothing
-    # else, which is useless for the usual failures (an unattached network, an
-    # unwritable data directory, a bad volume). Pull the real reason out.
+    # else, which is useless for the usual failures. The journal is where the
+    # entrypoint's own output is — that is where "su: Authentication failure"
+    # or a failed bind mount shows up.
     - name: Collect Trilium diagnostics
       ansible.builtin.command: "{{ item }}"
       loop:
@@ -94,30 +128,6 @@
       ansible.builtin.fail:
         msg: "Trilium did not start — see the diagnostics above."
 
-# Quadlet honours [Install] WantedBy= when it generates the unit, so
-# `systemctl --user enable` is neither needed nor possible on a generated unit.
-- name: Verify Trilium is running
-  ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
-  register: trilium_active
-  changed_when: false
-  failed_when: trilium_active.stdout != 'active'
-
-# `is-active` says the container is running; it does not say the app can serve.
-# `Restart=always` means even a crash-looping container is reported active.
-#
-# Unlike audiobookshelf, this image ships its own HEALTHCHECK
-# (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`, which the
-# Docker-era container reported healthy with), so there is no HealthCmd= in the
-# unit to run here — `podman healthcheck run` executes the image's own probe and
-# this depends on no path the play has to know about.
-- name: Wait for Trilium to pass its own health check
-  ansible.builtin.command: "podman healthcheck run {{ container_name }}"
-  register: trilium_health
-  until: trilium_health.rc == 0
-  retries: 24
-  delay: 5
-  changed_when: false
-
 # The check that catches a missing or mistyped Volume= line, which is the one
 # failure here that is silent: the app would run perfectly well with document.db
 # in the container's writable layer — or in an anonymous volume, if the image

trilium/roles/trilium/templates/trilium.container.j2

@@ -15,14 +15,24 @@ Wants=network-online.target
 ContainerName={{ container_name }}
 Image={{ image }}
 
-# The entrypoint starts as container-root and drops to the image's `node` user,
-# and `node` is uid 1000 — which is also the uid that owns every file in the
-# Docker-era data tree on monoco (docs/podman-compat-audit.md §3.2). Mapping
-# container uid 1000 onto {{ podman_user }} keeps {{ data_dir }} owned by
-# {{ podman_user }} on the host, so tar/rsync backups keep working and no
-# subuid-owned files appear.
+# The image's start-docker.sh drops privileges to the `node` user, and `node` is
+# uid 1000 — which is also the uid that owns every file in the Docker-era data
+# tree on monoco (docs/podman-compat-audit.md §3.2). Mapping container uid 1000
+# onto {{ podman_user }} keeps {{ data_dir }} owned by {{ podman_user }} on the
+# host, so tar/rsync backups keep working and no subuid-owned files appear.
 UserNS=keep-id:uid=1000,gid=1000
 
+# keep-id also runs the container *process* as that mapped uid, i.e. as `node`,
+# while start-docker.sh is written to start as root: it ends with
+# `exec su -c "node ./main.mjs" node`, and `su` from a non-root uid asks for a
+# password the `node` account does not have. The container then dies with
+# "su: Authentication failure" and Restart=always turns that into a crash loop.
+# Forcing the process back to root restores what the script expects and changes
+# nothing about the mapping above: `node` is still {{ podman_user }}, so the app
+# is still what writes {{ data_dir }} and it stays {{ podman_user }}'s.
+# (nextcloud.container needs the same two lines, for the same reason.)
+User=0
+
 Network={{ network_name }}.network
 
 # The image's data directory. This line is the only thing keeping state out of
@@ -37,9 +47,18 @@ Volume={{ data_dir }}:{{ container_data_dir }}
 # `Environment=TZ=Asia/Kolkata` here if you want local-time log filenames; until
 # then the container runs on UTC, exactly as it does on monoco.
 #
-# No HealthCmd= either, unlike audiobookshelf: this image ships its own
-# HEALTHCHECK (docker_healthcheck.cjs), so `podman healthcheck run` executes the
-# image's own probe and the unit only has to not get in the way.
+# The image ships no HEALTHCHECK — v0.106.0 dropped the docker_healthcheck.cjs
+# that the v0.95.0 still running on monoco had — so without this podman has
+# nothing to run and the play's `podman healthcheck run` fails with "no defined
+# healthcheck". This probes the endpoint the app advertises itself: it writes
+# healthcheck-url, pointing at /api/health-check, into the data directory on
+# startup. Deliberately no HealthOnFailure=: that would let a temporarily slow
+# start kill and restart a healthy server.
+HealthCmd={{ health_cmd }}
+HealthInterval=30s
+HealthTimeout=5s
+HealthRetries=3
+HealthStartPeriod=30s
 
 [Service]
 Restart=always

trilium/vars/default.yml

@@ -38,6 +38,14 @@ unit_name: "{{ container_name }}.service"
 # playbook pulled: monoco has been running TriliumNext since June 2025, and its
 # document.db carries the schema migrations that came with it. Pulling the old
 # image against that database would not work.
+#
+# `latest` has floated, and by more than a patch: monoco runs v0.95.0 (built
+# 2025-06-15) and the pull on emily is v0.106.0 (2026-09-25). TriliumNext
+# migrates document.db forward on first start, so the cutover is also an app
+# upgrade -- one-way for the copy on emily, which is why the readme says to
+# compare digests before the first start and to leave monoco's own tree alone.
+# Pin a digest or a version tag here if the migration should move the runtime
+# only and leave the app version to a separate, deliberate step.
 image: docker.io/triliumnext/notes:latest
 
 # The Quadlet file is <network_name>.network. caddy/roles/register pins the
@@ -67,6 +75,13 @@ backup_dir: "{{ data_dir }}/backup"
 # it; nothing is published on the host.
 container_port: 8080
 
+# The unit's own HEALTHCHECK, because the image does not ship one: v0.106.0
+# dropped the docker_healthcheck.cjs that v0.95.0 had, so `podman healthcheck
+# run` otherwise fails with "no defined healthcheck". /api/health-check is the
+# endpoint the app advertises itself, writing it to `healthcheck-url` in the data
+# directory at startup. `curl` is in the image; `wget` is not.
+health_cmd: "curl -fsS -o /dev/null http://127.0.0.1:{{ container_port }}/api/health-check"
+
 # The shared Caddy unit and its Caddyfile, both owned by caddy/playbook.yml.
 caddy_name: caddy
 caddyfile_path: "{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile"