diff --git a/trilium/readme.md b/trilium/readme.md index 37d61e4..41103ca 100644 --- a/trilium/readme.md +++ b/trilium/readme.md @@ -52,10 +52,23 @@ misspelling survives in exactly one place, deliberately: the **source path** `/var/lib/trillium` in the copy commands below, which is what the old host actually has. -The image's data directory is `/home/node/trilium-data`, and the image ships its -own `HEALTHCHECK` (`docker_healthcheck.cjs`), which the Docker-era container -reported healthy with. The unit therefore defines no `HealthCmd=` — unlike -[audiobookshelf/](../audiobookshelf/readme.md), whose image has none. +Two things about the image had to be settled against `emily` rather than read off +`monoco`, because `latest` floated by far more than a patch release between the +two pulls: + +| | monoco (running) | emily (pulled) | +|---|---|---| +| version | v0.95.0, built 2025-06-15 | **v0.106.0**, built 2026-09-25 | +| `HEALTHCHECK` | ships one (`docker_healthcheck.cjs`) | **ships none** | + +So the unit defines its own `HealthCmd=`, probing the endpoint the app advertises +in the `healthcheck-url` file it writes into the data directory, and the cutover +is **also an app upgrade**: TriliumNext migrates `document.db` forward on first +start. That migration is one-way for the copy on emily, which is why the digests +below are compared *before* the first start and monoco's own tree is left alone. + +The image also runs as root and drops to `node` with `su`, which is what `User=0` +exists for — see [Settings](#settings). ## Ordering @@ -69,7 +82,10 @@ this app has no database. This playbook does **not** move the data. Copy it first (see ["Migrating the data"](#migrating-the-data-from-monoco)) so the first start opens -the real `document.db`, rather than creating an empty document beside it. +the real `document.db`, rather than creating an empty document beside it. If it +does start against an empty directory — which the play warns about — that empty +document and its log have to be cleared before the copy, and the copy's steps say +how. ## Layout @@ -92,6 +108,7 @@ Quadlet units and state: | `~/podman-data/trilium/document.db` | the document, with `-wal` / `-shm` beside it | | `~/podman-data/trilium/config.ini` | the app's settings — `port`, `trustedReverseProxy`, `noBackup` | | `~/podman-data/trilium/session_secret.txt` | the session signing secret | +| `~/podman-data/trilium/healthcheck-url` | the app's own health endpoint, written on every start. `HealthCmd=` in the unit probes it | | `~/podman-data/trilium/log/` | one log file per day, 91 MB of the 106 MB tree | | `~/podman-data/trilium/backup/` | Trilium's own zipped `document.db` backups | | `~/podman-data/trilium/tmp/` | scratch, empty at rest | @@ -163,6 +180,7 @@ port-floor work Caddy itself needs. | `container_data_dir` | `/home/node/trilium-data` — the image's data directory | | `log_dir` / `tmp_dir` / `backup_dir` | The three app-managed subdirectories, created at 0700 | | `container_port` | `8080`, matching `config.ini`'s `port` | +| `health_cmd` | The unit's `HealthCmd=`, since the image ships none | | `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile | The three `podman_*` / `quadlet_dir` vars are **duplicated** from @@ -178,12 +196,23 @@ Like Caddy, `podman_user` is derived from the connection and the play carries no connecting as `root`**; the prerequisites, including that check, live in `caddy/playbook.yml`. -`UserNS=keep-id:uid=1000,gid=1000` — the entrypoint starts as container-root and -drops to the image's `node` user, which is uid 1000, and uid 1000 is what owns -every file in the Docker-era tree on monoco -([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). Mapping -container uid 1000 onto `podman_user` keeps `~/podman-data/trilium` owned by -`tanshu` and leaves no subuid-owned files to trip over backups. +Two lines decide who the container is, and **both are needed** — this is the one +thing that actually broke the first deployment of this playbook, and it is the +same pair [nextcloud/](../nextcloud/readme.md) needs: + +- **`UserNS=keep-id:uid=1000,gid=1000`** — `1000` is `node`, the user the app runs + as, and it is also what owns every file in the Docker-era tree on monoco + ([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `keep-id` + maps that one uid onto `podman_user`, so `~/podman-data/trilium` stays owned by + `tanshu` and plain `tar`, `rm -rf` and the backup scripts keep working on it. +- **`User=0`** — `keep-id` also runs the container *process* as that mapped uid, + i.e. as `node`, while the image's `start-docker.sh` is written to start as root: + it ends with `exec su -c "node ./main.mjs" node`, and `su` from a non-root uid + asks for a password the `node` account does not have. Without `User=0` the + container dies with `su: Authentication failure` and `Restart=always` turns that + into a crash loop. Forcing the process back to root restores what the script + expects and changes nothing about the mapping above: `node` is still `tanshu`, + so the app is still what writes the data directory and it stays `tanshu`'s. **There is no environment file here**, unlike `bitwarden/` and `tandoor/`. This deployment holds no secrets Ansible has to pass in: `config.ini` and @@ -243,6 +272,11 @@ ssh monoco 'docker stop trillium' # 3. Create the destination — this is the mount target, /home/node/trilium-data. ssh emily 'mkdir -p ~/podman-data/trilium' +# If this playbook has already run against that directory — it warns when it +# does — the app has left a document of its own there: an empty one, plus its +# log and healthcheck-url. Clear it, or step 5's digests will not match: +# ssh emily 'systemctl --user stop trilium.service; find ~/podman-data/trilium -mindepth 1 -delete' + # 4. Copy. `sudo` on the source side is not optional here: log/, tmp/ and # backup/ are 0700 and unreadable as tanshu, and backup/ is state. # @@ -317,7 +351,7 @@ journalctl --user -u trilium -f podman logs trilium # What the playbook checks for you: -podman healthcheck run trilium # the image's own probe +podman healthcheck run trilium # the unit's own HealthCmd=, not the image's # The unit's mount is real and not the container's writable layer: podman inspect trilium --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' @@ -358,24 +392,41 @@ Then reboot the host and confirm the unit returns unattended — without `/root/trilium-data` and predates TriliumNext's schema migrations; pointed at this `document.db` it is at best a failure to start. The unit pins `docker.io/triliumnext/notes:latest`. -- **The image's own health check is used.** There is no `HealthCmd=` in the unit - and there does not need to be: the image ships one - (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`), which is why the - Docker-era container showed `healthy`. `is-active` alone cannot tell a serving - server from a crash-looping one under `Restart=always`, which is why the play - runs the healthcheck rather than trusting it. +- **The unit's health check is its own.** Unlike the v0.95.0 on monoco, the + v0.106.0 image ships no `HEALTHCHECK`, so `podman healthcheck run` would fail + with "no defined healthcheck"; the unit probes the endpoint the app advertises + in `healthcheck-url` instead. `is-active` alone cannot tell a serving server + from a crash-looping one under `Restart=always`, which is why the play runs the + healthcheck rather than trusting it. +- **`keep-id` runs the container as the mapped uid, and this image needs root.** + The full explanation is in [Settings](#settings); the short version is that + without `User=0` the image's own `su node` cannot work and the unit crash-loops. + The three failures this playbook has actually produced, and what each means: + + | In the journal | Cause | + |---|---| + | `Password:` then `su: Authentication failure` | the container process is not root, so the image's privilege drop fails — `User=0` is missing | + | `no defined healthcheck` from `podman healthcheck run` | the image ships none and the unit's `HealthCmd=` is missing | + | `DB not initialized, please visit setup page` | the app is serving a **new, empty** document — the copy from monoco has not run (or was wiped) | + - **The mount check is a mount check, not a file check.** Audiobookshelf can assert its database appeared on the host, because the app creates it. Here the migration puts `document.db` on the host *before* the app runs, so its presence - proves nothing — the play asserts against `podman inspect` instead. That is the - one failure that is otherwise silent: with a missing `Volume=` line the app - runs perfectly and loses the document on the next recreate. -- **`latest` floats.** The image measured for this playbook is **v0.95.0** - (`org.opencontainers.image.version`), built 2025-06-15, revision `b9ae2265`. - Pin a digest if reproducible deploys matter — - `podman image inspect docker.io/triliumnext/notes:latest --format '{{.Digest}}'` - — since an upgrade can run schema migrations on `document.db` on first start. - Take Trilium's own backup (or stop-and-copy) before upgrading. + proves nothing — the play asserts against `podman inspect` instead, and checks + the *source* as well as the destination, because a destination-only check is + satisfied by an anonymous volume mounted where the bind mount should be. That is + the one failure that is otherwise silent: the app runs perfectly and loses the + document on the next recreate. +- **`latest` floats, and it moved here.** monoco runs **v0.95.0** (built + 2025-06-15, revision `b9ae2265`) and the pull on emily is **v0.106.0** (built + 2026-09-25, revision `c0fdceba`) — so this cutover upgrades the app as well as + the runtime, and the first start migrates `document.db` forward. That is + one-way *for the copy on emily*; monoco's own tree is untouched and remains the + rollback. Pin a version tag or a digest in `vars/default.yml` if the migration + should change only the runtime and leave the upgrade to a separate, deliberate + step (`podman image inspect docker.io/triliumnext/notes:latest --format + '{{.Digest}}'`). Take Trilium's own backup, or a stop-and-copy, before any + upgrade. - **`trustedReverseProxy=false` in `config.ini`.** Trilium does not trust `X-Forwarded-For`, so its own logs see the proxy rather than the client. That is the existing behaviour, carried across unchanged; set it to `true` (or to @@ -403,7 +454,9 @@ Then reboot the host and confirm the unit returns unattended — without - **DNS.** `notes.tanshu.com` must resolve to `emily` before Caddy can obtain a certificate for it. The certificates and ACME account key from monoco's `/var/lib/caddy/data` are **not** part of this migration — Caddy's own playbook - and readme cover that. + and readme cover that. Until the name moves, emily's Caddy retries the ACME + order and logs the failures; that is expected while monoco still holds the name, + and the site keeps being served from monoco. - **Root's crontab cannot `podman exec`** into another user's rootless containers, so anything scheduled belongs in a **systemd user timer** next to the Quadlet units. Trilium's own `backup/` mechanism runs *inside* the app diff --git a/trilium/roles/trilium/tasks/main.yaml b/trilium/roles/trilium/tasks/main.yaml index bd1eb5a..62a287c 100644 --- a/trilium/roles/trilium/tasks/main.yaml +++ b/trilium/roles/trilium/tasks/main.yaml @@ -64,7 +64,16 @@ - Reload systemd user units - Restart Trilium -- name: Start Trilium +# Quadlet honours [Install] WantedBy= when it generates the unit, so +# `systemctl --user enable` is neither needed nor possible on a generated unit. +# +# All three steps are in one block because they fail the same way and want the +# same evidence. `systemctl start` reports success as soon as it has handed the +# unit to systemd, so a container that dies a second later — `User=0` missing, a +# network that was never attached, an unwritable data directory — surfaces here +# as `activating`, or as a health check that never passes, with the real reason +# only in the journal. Restart=always makes that a crash loop, not an error. +- name: Start and verify Trilium block: - name: Start Trilium ansible.builtin.systemd_service: @@ -72,10 +81,35 @@ scope: user state: started daemon_reload: true + + # Retried rather than checked once: the container is coming up, and podman + # reports `activating` for the interval between systemd starting it and the + # container's own readiness notification. + - name: Verify Trilium is running + ansible.builtin.command: "systemctl --user is-active {{ unit_name }}" + register: trilium_active + until: trilium_active.stdout == 'active' + retries: 12 + delay: 5 + changed_when: false + + # `is-active` says the container is running; it does not say the app can + # serve. The unit defines the probe (HealthCmd=, see the template) because + # the image ships none, so this runs exactly that command once, + # synchronously, and fails while the app is still migrating or not yet + # listening. + - name: Wait for Trilium to pass its health check + ansible.builtin.command: "podman healthcheck run {{ container_name }}" + register: trilium_health + until: trilium_health.rc == 0 + retries: 24 + delay: 5 + changed_when: false rescue: # systemd reports "control process exited with error code" and nothing - # else, which is useless for the usual failures (an unattached network, an - # unwritable data directory, a bad volume). Pull the real reason out. + # else, which is useless for the usual failures. The journal is where the + # entrypoint's own output is — that is where "su: Authentication failure" + # or a failed bind mount shows up. - name: Collect Trilium diagnostics ansible.builtin.command: "{{ item }}" loop: @@ -94,30 +128,6 @@ ansible.builtin.fail: msg: "Trilium did not start — see the diagnostics above." -# Quadlet honours [Install] WantedBy= when it generates the unit, so -# `systemctl --user enable` is neither needed nor possible on a generated unit. -- name: Verify Trilium is running - ansible.builtin.command: "systemctl --user is-active {{ unit_name }}" - register: trilium_active - changed_when: false - failed_when: trilium_active.stdout != 'active' - -# `is-active` says the container is running; it does not say the app can serve. -# `Restart=always` means even a crash-looping container is reported active. -# -# Unlike audiobookshelf, this image ships its own HEALTHCHECK -# (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`, which the -# Docker-era container reported healthy with), so there is no HealthCmd= in the -# unit to run here — `podman healthcheck run` executes the image's own probe and -# this depends on no path the play has to know about. -- name: Wait for Trilium to pass its own health check - ansible.builtin.command: "podman healthcheck run {{ container_name }}" - register: trilium_health - until: trilium_health.rc == 0 - retries: 24 - delay: 5 - changed_when: false - # The check that catches a missing or mistyped Volume= line, which is the one # failure here that is silent: the app would run perfectly well with document.db # in the container's writable layer — or in an anonymous volume, if the image diff --git a/trilium/roles/trilium/templates/trilium.container.j2 b/trilium/roles/trilium/templates/trilium.container.j2 index 15d669c..33171c6 100644 --- a/trilium/roles/trilium/templates/trilium.container.j2 +++ b/trilium/roles/trilium/templates/trilium.container.j2 @@ -15,14 +15,24 @@ Wants=network-online.target ContainerName={{ container_name }} Image={{ image }} -# The entrypoint starts as container-root and drops to the image's `node` user, -# and `node` is uid 1000 — which is also the uid that owns every file in the -# Docker-era data tree on monoco (docs/podman-compat-audit.md §3.2). Mapping -# container uid 1000 onto {{ podman_user }} keeps {{ data_dir }} owned by -# {{ podman_user }} on the host, so tar/rsync backups keep working and no -# subuid-owned files appear. +# The image's start-docker.sh drops privileges to the `node` user, and `node` is +# uid 1000 — which is also the uid that owns every file in the Docker-era data +# tree on monoco (docs/podman-compat-audit.md §3.2). Mapping container uid 1000 +# onto {{ podman_user }} keeps {{ data_dir }} owned by {{ podman_user }} on the +# host, so tar/rsync backups keep working and no subuid-owned files appear. UserNS=keep-id:uid=1000,gid=1000 +# keep-id also runs the container *process* as that mapped uid, i.e. as `node`, +# while start-docker.sh is written to start as root: it ends with +# `exec su -c "node ./main.mjs" node`, and `su` from a non-root uid asks for a +# password the `node` account does not have. The container then dies with +# "su: Authentication failure" and Restart=always turns that into a crash loop. +# Forcing the process back to root restores what the script expects and changes +# nothing about the mapping above: `node` is still {{ podman_user }}, so the app +# is still what writes {{ data_dir }} and it stays {{ podman_user }}'s. +# (nextcloud.container needs the same two lines, for the same reason.) +User=0 + Network={{ network_name }}.network # The image's data directory. This line is the only thing keeping state out of @@ -37,9 +47,18 @@ Volume={{ data_dir }}:{{ container_data_dir }} # `Environment=TZ=Asia/Kolkata` here if you want local-time log filenames; until # then the container runs on UTC, exactly as it does on monoco. # -# No HealthCmd= either, unlike audiobookshelf: this image ships its own -# HEALTHCHECK (docker_healthcheck.cjs), so `podman healthcheck run` executes the -# image's own probe and the unit only has to not get in the way. +# The image ships no HEALTHCHECK — v0.106.0 dropped the docker_healthcheck.cjs +# that the v0.95.0 still running on monoco had — so without this podman has +# nothing to run and the play's `podman healthcheck run` fails with "no defined +# healthcheck". This probes the endpoint the app advertises itself: it writes +# healthcheck-url, pointing at /api/health-check, into the data directory on +# startup. Deliberately no HealthOnFailure=: that would let a temporarily slow +# start kill and restart a healthy server. +HealthCmd={{ health_cmd }} +HealthInterval=30s +HealthTimeout=5s +HealthRetries=3 +HealthStartPeriod=30s [Service] Restart=always diff --git a/trilium/vars/default.yml b/trilium/vars/default.yml index b341305..40bdaca 100644 --- a/trilium/vars/default.yml +++ b/trilium/vars/default.yml @@ -38,6 +38,14 @@ unit_name: "{{ container_name }}.service" # playbook pulled: monoco has been running TriliumNext since June 2025, and its # document.db carries the schema migrations that came with it. Pulling the old # image against that database would not work. +# +# `latest` has floated, and by more than a patch: monoco runs v0.95.0 (built +# 2025-06-15) and the pull on emily is v0.106.0 (2026-09-25). TriliumNext +# migrates document.db forward on first start, so the cutover is also an app +# upgrade -- one-way for the copy on emily, which is why the readme says to +# compare digests before the first start and to leave monoco's own tree alone. +# Pin a digest or a version tag here if the migration should move the runtime +# only and leave the app version to a separate, deliberate step. image: docker.io/triliumnext/notes:latest # The Quadlet file is .network. caddy/roles/register pins the @@ -67,6 +75,13 @@ backup_dir: "{{ data_dir }}/backup" # it; nothing is published on the host. container_port: 8080 +# The unit's own HEALTHCHECK, because the image does not ship one: v0.106.0 +# dropped the docker_healthcheck.cjs that v0.95.0 had, so `podman healthcheck +# run` otherwise fails with "no defined healthcheck". /api/health-check is the +# endpoint the app advertises itself, writing it to `healthcheck-url` in the data +# directory at startup. `curl` is in the image; `wget` is not. +health_cmd: "curl -fsS -o /dev/null http://127.0.0.1:{{ container_port }}/api/health-check" + # The shared Caddy unit and its Caddyfile, both owned by caddy/playbook.yml. caddy_name: caddy caddyfile_path: "{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile"