playbooks/trilium/readme.md

# TriliumNext Notes (rootless Podman + Quadlet)

Runs [TriliumNext Notes](https://github.com/TriliumNext/Notes) on `emily` as a
**rootless Podman** container managed by a **systemd user unit** (Quadlet),
behind the shared [rootless Caddy](../caddy/readme.md). Replaces the container
running on `monoco`, which `docker inspect` reports as:

```command
docker run -d \
  --name trillium \
  --network trillium_net \
  --volume /var/lib/trillium:/home/node/trilium-data \
  --restart=unless-stopped \
  triliumnext/notes:latest
```

with no published ports: Caddy is on the same network and proxies to
`trillium:8080`.

Unlike [bitwarden/](../bitwarden/readme.md) and [tandoor/](../tandoor/readme.md),
**there is no database to attach.** Trilium's state is one SQLite file
(`document.db`, with its WAL sidecars), `config.ini`, `session_secret.txt`, and
an app-managed `backup/` directory — so this play has no `postgres` role and
nothing to restore from the shared cluster.

## What the old playbook gets wrong

`trilium/playbook.yml` was last touched in 2022 and **is not what runs on
monoco.** Three differences shape everything below, and all three were read off
the live host rather than the playbook:

| On monoco (measured) | In the old playbook |
|---|---|
| `triliumnext/notes:latest` | `zadam/trilium:latest` — the retired original. Its `document.db` predates TriliumNext's schema migrations |
| `/var/lib/trillium` → `/home/node/trilium-data` | `/var/lib/trilium` → `/root/trilium-data` |
| `trillium_net`, Caddy proxies to `trillium:8080` | nginx on the host, proxying to `127.0.0.1:8888` |

So this playbook pins the **TriliumNext** image against a data directory at
`/home/node/trilium-data` — pulling `zadam/trilium` against the migrated
database would not work.

### Naming: `trilium`, not `trillium`

The live container on monoco is `trillium`, with an extra `l` — a typo that
exists only in the hand-made `docker run` that created it. This directory, the
Docker-era playbook, and the app itself all say `trilium`, and **everything on
emily is `trilium`**: directory, container, unit, Podman network, Caddyfile
upstream.

The site `notes.tanshu.com` is unchanged, so nothing client-facing moves. The
misspelling survives in exactly one place, deliberately: the **source path**
`/var/lib/trillium` in the copy commands below, which is what the old host
actually has.

The image's data directory is `/home/node/trilium-data`, and the image ships its
own `HEALTHCHECK` (`docker_healthcheck.cjs`), which the Docker-era container
reported healthy with. The unit therefore defines no `HealthCmd=` — unlike
[audiobookshelf/](../audiobookshelf/readme.md), whose image has none.

## Ordering

Run [caddy/playbook.yml](../caddy/playbook.yml) first. It is the reverse proxy,
owns the Caddyfile this site appends to, supplies the `caddy.service` handlers,
and is the one play that applies the rootless-Podman prerequisites (installing
podman, `loginctl enable-linger`, the low-port sysctl, the Quadlet directory).

There is no second prerequisite to wait for — no cluster to bring across, since
this app has no database.

This playbook does **not** move the data. Copy it first (see
["Migrating the data"](#migrating-the-data-from-monoco)) so the first start opens
the real `document.db`, rather than creating an empty document beside it.

## Layout

```
trilium/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── trilium/         # the data directory, the Quadlet unit, start + verify
    └── caddy/           # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/trilium.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/trilium.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-trilium.conf` | attaches Caddy to that network |
| `~/podman-data/trilium/document.db` | the document, with `-wal` / `-shm` beside it |
| `~/podman-data/trilium/config.ini` | the app's settings — `port`, `trustedReverseProxy`, `noBackup` |
| `~/podman-data/trilium/session_secret.txt` | the session signing secret |
| `~/podman-data/trilium/log/` | one log file per day, 91 MB of the 106 MB tree |
| `~/podman-data/trilium/backup/` | Trilium's own zipped `document.db` backups |
| `~/podman-data/trilium/tmp/` | scratch, empty at rest |

## What is state, and what is not

Per [docs/backup-plan.md §2.1](../docs/backup-plan.md), the tree is **106 MB**
and almost all of it is dispensable:

| Path | Size | Keep? | Why |
|---|---|---|---|
| `document.db` | 3.8 MB | **required** | every note, revision, attribute and attachment reference |
| `config.ini` | 3 KB | **required** | the instance's settings; `port=8080` is why the unit proxies to 8080 |
| `session_secret.txt` | 64 B | **required** | the session signing secret |
| `backup/` | 12 MB | worth keeping | Trilium's own zipped `document.db` backups (`noBackup=false`) |
| `log/` | 91 MB | **no** | daily app logs. Regenerable, and 86% of the tree |
| `tmp/` | 0 | no | scratch |

So the real state is **~4 MB**, and the migration carries all of it.

**A live `document.db` must not be `tar`-ed.** It is SQLite in WAL mode, so the
main file and its `-wal` sidecar are copied at different instants and the result
can be torn — a database that will not open
([docs/backup-plan.md §3.1](../docs/backup-plan.md)). Stop the container, as
[§7.3](../docs/backup-plan.md) lists this among the stop-copy-start SQLite
services, or use Trilium's own **Settings → Backup**.

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Trilium's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`trilium.container` references `trilium.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet.

`network_name` is `trilium`, so the *file* is `trilium.network` and the *Podman
network* is `trilium_net` (`NetworkName=` in the network unit pins it).

[roles/caddy](roles/caddy/tasks/main.yaml) appends the site block to the shared
Caddyfile with `blockinfile`, exactly as the other services do. Caddy handles the
WebSocket upgrade the live-updating editor needs, and imposes no request-body
limit, so there is nothing like the old nginx front end to configure.

**Nothing is published on the host.** The container listens on 8080 and Caddy
reaches it as `trilium:8080` over `trilium_net`, so this play needs none of the
port-floor work Caddy itself needs.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `notes.tanshu.com` — the site block Caddy serves |
| `container_name` | `trilium`, and the Quadlet unit / container name |
| `image` | `docker.io/triliumnext/notes:latest` — fully qualified, and the TriliumNext image |
| `network_name` | `trilium` → `trilium.network` / `trilium_net` |
| `data_dir` | `~/podman-data/trilium` — the one and only mount source |
| `container_data_dir` | `/home/node/trilium-data` — the image's data directory |
| `log_dir` / `tmp_dir` / `backup_dir` | The three app-managed subdirectories, created at 0700 |
| `container_port` | `8080`, matching `config.ini`'s `port` |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

`UserNS=keep-id:uid=1000,gid=1000` — the entrypoint starts as container-root and
drops to the image's `node` user, which is uid 1000, and uid 1000 is what owns
every file in the Docker-era tree on monoco
([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). Mapping
container uid 1000 onto `podman_user` keeps `~/podman-data/trilium` owned by
`tanshu` and leaves no subuid-owned files to trip over backups.

**There is no environment file here**, unlike `bitwarden/` and `tandoor/`. This
deployment holds no secrets Ansible has to pass in: `config.ini` and
`session_secret.txt` live inside the data directory, and the Docker-era container
set no environment at all. That is also why no `TZ` is set — the container runs
on UTC, exactly as it does today. Add `Environment=TZ=Asia/Kolkata` to the
template if you want local-time log filenames.

## Running

```command
ansible-playbook -u tanshu trilium/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

This play applies **only** this app's units and data. Its `pre_tasks` run Caddy's
`register` include and nothing else. The rootless-Podman prerequisites are the
shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml) and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.

`systemctl --user enable trilium` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

## Migrating the data from monoco

Everything lives in **`/var/lib/trillium`** — note the extra `l`; see
["Naming"](#naming-trilium-not-trillium) — owned by **uid 1000**, which is the
`ubuntu` account on monoco and the uid the app writes as inside the container. It
moves to `~/podman-data/trilium`, and the whole tree is 106 MB, of which 91 MB is
regenerable logs.

Two properties of the tree decide how the copy has to be done:

- **`document.db` is live SQLite in WAL mode**, with `-wal` and `-shm` beside it
  ([docs/backup-plan.md §3.1](../docs/backup-plan.md)). **Stop the container
  first**; a hot copy can be torn.
- **`log/`, `tmp/` and `backup/` are mode 0700 and owned by uid 1000**, not by
  the `tanshu` account you SSH as. A copy made as `tanshu` cannot read them at
  all — `backup/` included, and that one is state.

`emily` cannot SSH to `monoco`, nor `monoco` to `emily` (neither resolves the
other's name), so the copy streams through the machine that holds both aliases —
and `rsync` cannot copy remote-to-remote in a single command anyway.

```bash
# 1. Look before you copy: size, contents, numeric ownership. `ls -ln` shows
#    uid:gid, which is what matters here — the tree is uid 1000, not root.
ssh monoco 'du -sh /var/lib/trillium; ls -ln /var/lib/trillium'

# 2. Stop the old container so nothing writes while you copy, and leave it
#    stopped: monoco's tree becomes a stale snapshot the moment emily serves.
ssh monoco 'docker stop trillium'

# 3. Create the destination — this is the mount target, /home/node/trilium-data.
ssh emily 'mkdir -p ~/podman-data/trilium'

# 4. Copy. `sudo` on the source side is not optional here: log/, tmp/ and
#    backup/ are 0700 and unreadable as tanshu, and backup/ is state.
#
#    `set -o pipefail` matters: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
#
#    `--no-same-owner` is explicit because the source files are uid 1000
#    (`ubuntu`) on monoco. keep-id maps container uid 1000 onto tanshu, so files
#    that arrive owned by tanshu are already correct — restoring the old
#    ownership would be wrong.
set -o pipefail
ssh monoco 'sudo tar -C /var/lib/trillium -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/trilium -xf -'

# 5. Verify by content, over the whole tree on both sides. The paths are relative
#    on both sides, so the digests are comparable. `sudo` again on the source:
#    the 0700 directories are part of what is being compared.
#
#    Do this *before* the deploy below — Trilium may run schema migrations
#    against document.db on first start, and the two trees stop being comparable
#    at that point.
ssh monoco 'sudo sh -c "cd /var/lib/trillium && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum"'
ssh emily  'cd ~/podman-data/trilium && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by a subuid, or by root.
ssh emily 'find ~/podman-data/trilium ! -user tanshu'    # empty = correct

# 7. Deploy — after the copy, so the first start opens the real document rather
#    than creating an empty one beside it.
cd prepare-server && ansible-playbook -u tanshu ../trilium/playbook.yml
```

Ownership needs no `chown` pass on emily: the container writes as uid 1000, which
`keep-id` maps onto `tanshu`, and the extraction already produced `tanshu`-owned
files. That is also why `--no-same-owner` is explicit on the receiving side.

If you would rather not use `sudo`, or the 91 MB of logs are not worth carrying,
exclude the three app-managed directories — the role recreates them at the right
mode, and the play reads `log/` back as evidence afterwards:

```bash
set -o pipefail
ssh monoco 'tar --exclude=./log --exclude=./tmp --exclude=./backup -C /var/lib/trillium -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/trilium -xf -'
```

That drops the 91 MB of regenerable logs **and** the 12 MB of `backup/` —
Trilium's own zipped `document.db` backups. Nothing in it is unique against the
migration itself, but that is worth saying deliberately rather than by accident.
If you use this variant, apply the same three exclusions to the `find` in step 5.

If the link is slow or drops, copy in two hops instead: resumable, and it shows
progress. The first hop needs `--rsync-path` because of the 0700 directories.

```bash
rsync -aP --rsync-path="sudo rsync" monoco:/var/lib/trillium/ ~/trilium-copy/
rsync -aP ~/trilium-copy/ emily:~/podman-data/trilium/
```

Rolling back means rolling back on monoco: `ssh monoco 'docker start trillium'`
is meaningful only while monoco's tree is still the one you copied. Once emily
has served and written to `document.db`, the two have diverged — restore monoco's
copy rather than starting it in place.

## Verification

```bash
systemctl --user status trilium
journalctl --user -u trilium -f
podman logs trilium

# What the playbook checks for you:
podman healthcheck run trilium          # the image's own probe

# The unit's mount is real and not the container's writable layer:
podman inspect trilium --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'

# The health the unit reports on its own timer, not just once:
podman inspect trilium --format '{{.State.Health.Status}}'

# The document came across: a migrated document.db is a few megabytes, not the
# kilobytes of one the app creates on an empty install.
ssh emily 'ls -ln ~/podman-data/trilium/document.db'
ssh emily 'command -v sqlite3 >/dev/null && sqlite3 ~/podman-data/trilium/document.db "pragma integrity_check"'

# The site, end to end.
curl -sSI https://notes.tanshu.com
```

Then open `https://notes.tanshu.com` **in a browser**, log in, and open a note.
A `curl` cannot prove the migrated document is *readable*; only the app can.

Also exercise a **write** — edit a note — and confirm on the host that what the
app wrote is owned by `tanshu`:

```bash
ssh emily 'ls -ln ~/podman-data/trilium/document.db* ~/podman-data/trilium/log/'
```

An edit touches `document.db-wal` and appends to today's log; both must be owned
by `tanshu`. That is the check that catches a wrong `keep-id`; a read-only smoke
test will not. The play reports the same thing from the other direction, by
looking for a `trilium-*.log` touched since the unit started.

Then reboot the host and confirm the unit returns unattended — without
`loginctl enable-linger tanshu` it will not, and the failure is silent.

## Notes and gotchas

- **Do not deploy the old image.** The retired `zadam/trilium` reads
  `/root/trilium-data` and predates TriliumNext's schema migrations; pointed at
  this `document.db` it is at best a failure to start. The unit pins
  `docker.io/triliumnext/notes:latest`.
- **The image's own health check is used.** There is no `HealthCmd=` in the unit
  and there does not need to be: the image ships one
  (`exec gosu node node /usr/src/app/docker_healthcheck.cjs`), which is why the
  Docker-era container showed `healthy`. `is-active` alone cannot tell a serving
  server from a crash-looping one under `Restart=always`, which is why the play
  runs the healthcheck rather than trusting it.
- **The mount check is a mount check, not a file check.** Audiobookshelf can
  assert its database appeared on the host, because the app creates it. Here the
  migration puts `document.db` on the host *before* the app runs, so its presence
  proves nothing — the play asserts against `podman inspect` instead. That is the
  one failure that is otherwise silent: with a missing `Volume=` line the app
  runs perfectly and loses the document on the next recreate.
- **`latest` floats.** The image measured for this playbook is **v0.95.0**
  (`org.opencontainers.image.version`), built 2025-06-15, revision `b9ae2265`.
  Pin a digest if reproducible deploys matter —
  `podman image inspect docker.io/triliumnext/notes:latest --format '{{.Digest}}'`
  — since an upgrade can run schema migrations on `document.db` on first start.
  Take Trilium's own backup (or stop-and-copy) before upgrading.
- **`trustedReverseProxy=false` in `config.ini`.** Trilium does not trust
  `X-Forwarded-For`, so its own logs see the proxy rather than the client. That
  is the existing behaviour, carried across unchanged; set it to `true` (or to
  the Caddy network's CIDR) in `config.ini` if you want real client addresses
  there. It is app state, not something this playbook sets.
- **The copy needs `sudo` on monoco** because `log/`, `tmp/` and `backup/` are
  0700, and `backup/` is the one of the three that is state. `tanshu` cannot read
  them, and a copy that silently skips them looks like it worked.
- **Verify before deploying, not after.** The digest comparison in step 5 stops
  being meaningful once the app has opened `document.db`, because it may migrate
  it.
- **No `TZ`.** The Docker-era container set no environment at all, so the app
  runs on UTC — that is where the log filenames' dates come from. Trilium renders
  dates in the browser, so this is cosmetic, but it is a deliberate carry-over
  rather than an omission.
- **Registering restarts all of Caddy.** Writing the network unit and the drop-in
  changes Caddy's generated unit, and Quadlet has no live equivalent of
  `docker network connect`, so Caddy is recreated — a brief restart of *every*
  hosted site, not just this one. See
  [caddy/readme.md](../caddy/readme.md), "Trade-off worth knowing".
- **If the Caddyfile does not exist yet**, this playbook creates it containing
  only the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a
  missing or zero-length file. Deploy Caddy first on a fresh host.
- **DNS.** `notes.tanshu.com` must resolve to `emily` before Caddy can obtain a
  certificate for it. The certificates and ACME account key from monoco's
  `/var/lib/caddy/data` are **not** part of this migration — Caddy's own playbook
  and readme cover that.
- **Root's crontab cannot `podman exec`** into another user's rootless
  containers, so anything scheduled belongs in a **systemd user timer** next to
  the Quadlet units. Trilium's own `backup/` mechanism runs *inside* the app
  (`noBackup=false` in `config.ini`), so it keeps working without cron.
- **Never back up a hot `document.db`.** Same WAL problem as the migration: stop
  the container, or let the app's own backup do it.
- **Do not restart the container on monoco.** Its tree is a stale snapshot of a
  document emily has moved past.

## Not done here

- Pinning the image by digest or version tag, or wiring up
  `AutoUpdate=registry` with `podman auto-update`.
- A backup script or systemd user timer for `document.db` — the app's own
  **Settings → Backup** into `backup/` is the mechanism, and the 12 MB it has
  produced so far comes across with the copy.
- Setting `TZ`, or turning on `trustedReverseProxy`, both of which are runtime
  settings inside `config.ini` rather than Ansible variables.
- Renaming `trillium` to `trilium` on monoco, or updating the docs that call this
  service `trillium`: `docs/podman-migration.md` (§4.1, §4.2, §4.3, §5.5) and
  `docs/podman-compat-audit.md` (§3.2, §4).