playbooks/trilium/readme.md

TriliumNext Notes (rootless Podman + Quadlet)

Runs TriliumNext Notes on emily as a rootless Podman container managed by a systemd user unit (Quadlet), behind the shared rootless Caddy. Replaces the container running on monoco, which docker inspect reports as:

docker run -d \
  --name trillium \
  --network trillium_net \
  --volume /var/lib/trillium:/home/node/trilium-data \
  --restart=unless-stopped \
  triliumnext/notes:latest

with no published ports: Caddy is on the same network and proxies to trillium:8080.

Unlike bitwarden/ and tandoor/, there is no database to attach. Trilium's state is one SQLite file (document.db, with its WAL sidecars), config.ini, session_secret.txt, and an app-managed backup/ directory — so this play has no postgres role and nothing to restore from the shared cluster.

What the old playbook gets wrong

trilium/playbook.yml was last touched in 2022 and is not what runs on monoco. Three differences shape everything below, and all three were read off the live host rather than the playbook:

On monoco (measured)	In the old playbook
triliumnext/notes:latest	zadam/trilium:latest — the retired original. Its document.db predates TriliumNext's schema migrations
/var/lib/trillium → /home/node/trilium-data	/var/lib/trilium → /root/trilium-data
trillium_net, Caddy proxies to trillium:8080	nginx on the host, proxying to 127.0.0.1:8888

So this playbook pins the TriliumNext image against a data directory at /home/node/trilium-data — pulling zadam/trilium against the migrated database would not work.

Naming: trilium, not trillium

The live container on monoco is trillium, with an extra l — a typo that exists only in the hand-made docker run that created it. This directory, the Docker-era playbook, and the app itself all say trilium, and everything on emily is trilium: directory, container, unit, Podman network, Caddyfile upstream.

The site notes.tanshu.com is unchanged, so nothing client-facing moves. The misspelling survives in exactly one place, deliberately: the source path /var/lib/trillium in the copy commands below, which is what the old host actually has.

Two things about the image had to be settled against emily rather than read off monoco, because latest floated by far more than a patch release between the two pulls:

	monoco (running)	emily (pulled)
version	v0.95.0, built 2025-06-15	v0.106.0, built 2026-09-25
HEALTHCHECK	ships one (docker_healthcheck.cjs)	ships none

So the unit defines its own HealthCmd=, probing the endpoint the app advertises in the healthcheck-url file it writes into the data directory, and the cutover is also an app upgrade: TriliumNext migrates document.db forward on first start. That migration is one-way for the copy on emily, which is why the digests below are compared before the first start and monoco's own tree is left alone.

The image also runs as root and drops to node with su, which is what User=0 exists for — see Settings.

Ordering

Run caddy/playbook.yml first. It is the reverse proxy, owns the Caddyfile this site appends to, supplies the caddy.service handlers, and is the one play that applies the rootless-Podman prerequisites (installing podman, loginctl enable-linger, the low-port sysctl, the Quadlet directory).

There is no second prerequisite to wait for — no cluster to bring across, since this app has no database.

This playbook does not move the data. Copy it first (see "Migrating the data") so the first start opens the real document.db, rather than creating an empty document beside it. If it does start against an empty directory — which the play warns about — that empty document and its log have to be cleared before the copy, and the copy's steps say how.

Layout

trilium/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── trilium/         # the data directory, the Quadlet unit, start + verify
    └── caddy/           # appends this site's block to the shared Caddyfile

Quadlet units and state:

On the host	What
~/.config/containers/systemd/trilium.container	the generated unit (Ansible-templated)
~/.config/containers/systemd/trilium.network	the app network, written by caddy/roles/register
~/.config/containers/systemd/caddy.container.d/50-trilium.conf	attaches Caddy to that network
~/podman-data/trilium/document.db	the document, with -wal / -shm beside it
~/podman-data/trilium/config.ini	the app's settings — port, trustedReverseProxy, noBackup
~/podman-data/trilium/session_secret.txt	the session signing secret
~/podman-data/trilium/healthcheck-url	the app's own health endpoint, written on every start. HealthCmd= in the unit probes it
~/podman-data/trilium/log/	one log file per day, 91 MB of the 106 MB tree
~/podman-data/trilium/backup/	Trilium's own zipped document.db backups
~/podman-data/trilium/tmp/	scratch, empty at rest

What is state, and what is not

Per docs/backup-plan.md §2.1, the tree is 106 MB and almost all of it is dispensable:

Path	Size	Keep?	Why
document.db	3.8 MB	required	every note, revision, attribute and attachment reference
config.ini	3 KB	required	the instance's settings; port=8080 is why the unit proxies to 8080
session_secret.txt	64 B	required	the session signing secret
backup/	12 MB	worth keeping	Trilium's own zipped document.db backups (noBackup=false)
log/	91 MB	no	daily app logs. Regenerable, and 86% of the tree
tmp/	0	no	scratch

So the real state is ~4 MB, and the migration carries all of it.

A live document.db must not be tar-ed. It is SQLite in WAL mode, so the main file and its -wal sidecar are copied at different instants and the result can be torn — a database that will not open (docs/backup-plan.md §3.1). Stop the container, as §7.3 lists this among the stop-copy-start SQLite services, or use Trilium's own Settings → Backup.

How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls the shared include in caddy/roles/register, which writes the network unit and the drop-in that pulls it into Caddy:

- name: Register Trilium's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"

It runs in pre_tasks, before the app unit is started, because trilium.container references trilium.network: Quadlet fails unit generation — so systemctl --user start reports "unit not found" — when a referenced .network file does not exist yet.

network_name is trilium, so the file is trilium.network and the Podman network is trilium_net (NetworkName= in the network unit pins it).

roles/caddy appends the site block to the shared Caddyfile with blockinfile, exactly as the other services do. Caddy handles the WebSocket upgrade the live-updating editor needs, and imposes no request-body limit, so there is nothing like the old nginx front end to configure.

Nothing is published on the host. The container listens on 8080 and Caddy reaches it as trilium:8080 over trilium_net, so this play needs none of the port-floor work Caddy itself needs.

Settings

Variable	Value / purpose
podman_user	Derived from the connection (ansible_facts['user_id']), not set by hand
podman_data_dir	~/podman-data
quadlet_dir	~/.config/containers/systemd
host	notes.tanshu.com — the site block Caddy serves
container_name	trilium, and the Quadlet unit / container name
image	docker.io/triliumnext/notes:latest — fully qualified, and the TriliumNext image
network_name	trilium → trilium.network / trilium_net
data_dir	~/podman-data/trilium — the one and only mount source
container_data_dir	/home/node/trilium-data — the image's data directory
log_dir / tmp_dir / backup_dir	The three app-managed subdirectories, created at 0700
container_port	8080, matching config.ini's port
health_cmd	The unit's HealthCmd=, since the image ships none
caddy_name / caddyfile_path	The shared Caddy unit and its Caddyfile

The three podman_* / quadlet_dir vars are duplicated from caddy/vars/default.yml, as in every service playbook. That is a deliberate trade: a play cannot include the Caddy vars file wholesale, because it also sets host_directory (to the Caddy data directory) and every service defines that for itself. Keep them in sync. Caddy's fourth, podman_unprivileged_port_start, belongs to the podman tasks this play does not run.

Like Caddy, podman_user is derived from the connection and the play carries no become. Because the podman tasks are not run here, there is no guard against connecting as root; the prerequisites, including that check, live in caddy/playbook.yml.

Two lines decide who the container is, and both are needed — this is the one thing that actually broke the first deployment of this playbook, and it is the same pair nextcloud/ needs:

• UserNS=keep-id:uid=1000,gid=1000 — 1000 is node, the user the app runs as, and it is also what owns every file in the Docker-era tree on monoco (docs/podman-compat-audit.md §3.2). keep-id maps that one uid onto podman_user, so ~/podman-data/trilium stays owned by tanshu and plain tar, rm -rf and the backup scripts keep working on it.
• User=0 — keep-id also runs the container process as that mapped uid, i.e. as node, while the image's start-docker.sh is written to start as root: it ends with exec su -c "node ./main.mjs" node, and su from a non-root uid asks for a password the node account does not have. Without User=0 the container dies with su: Authentication failure and Restart=always turns that into a crash loop. Forcing the process back to root restores what the script expects and changes nothing about the mapping above: node is still tanshu, so the app is still what writes the data directory and it stays tanshu's.

There is no environment file here, unlike bitwarden/ and tandoor/. This deployment holds no secrets Ansible has to pass in: config.ini and session_secret.txt live inside the data directory, and the Docker-era container set no environment at all. That is also why no TZ is set — the container runs on UTC, exactly as it does today. Add Environment=TZ=Asia/Kolkata to the template if you want local-time log filenames.

Running

ansible-playbook -u tanshu trilium/playbook.yml

The play targets emily directly, so the inventory must contain it — the one in prepare-server/ does, and its ansible.cfg names it.

This play applies only this app's units and data. Its pre_tasks run Caddy's register include and nothing else. The rootless-Podman prerequisites are the shared task file at caddy/roles/podman/tasks/main.yaml and are not run here. A host that has already run caddy/playbook.yml has them.

systemctl --user enable trilium is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Migrating the data from monoco

Everything lives in /var/lib/trillium — note the extra l; see "Naming" — owned by uid 1000, which is the ubuntu account on monoco and the uid the app writes as inside the container. It moves to ~/podman-data/trilium, and the whole tree is 106 MB, of which 91 MB is regenerable logs.

Two properties of the tree decide how the copy has to be done:

• document.db is live SQLite in WAL mode, with -wal and -shm beside it (docs/backup-plan.md §3.1). Stop the container first; a hot copy can be torn.
• log/, tmp/ and backup/ are mode 0700 and owned by uid 1000, not by the tanshu account you SSH as. A copy made as tanshu cannot read them at all — backup/ included, and that one is state.

emily cannot SSH to monoco, nor monoco to emily (neither resolves the other's name), so the copy streams through the machine that holds both aliases — and rsync cannot copy remote-to-remote in a single command anyway.

# 1. Look before you copy: size, contents, numeric ownership. `ls -ln` shows
#    uid:gid, which is what matters here — the tree is uid 1000, not root.
ssh monoco 'du -sh /var/lib/trillium; ls -ln /var/lib/trillium'

# 2. Stop the old container so nothing writes while you copy, and leave it
#    stopped: monoco's tree becomes a stale snapshot the moment emily serves.
ssh monoco 'docker stop trillium'

# 3. Create the destination — this is the mount target, /home/node/trilium-data.
ssh emily 'mkdir -p ~/podman-data/trilium'

#    If this playbook has already run against that directory — it warns when it
#    does — the app has left a document of its own there: an empty one, plus its
#    log and healthcheck-url. Clear it, or step 5's digests will not match:
# ssh emily 'systemctl --user stop trilium.service; find ~/podman-data/trilium -mindepth 1 -delete'

# 4. Copy. `sudo` on the source side is not optional here: log/, tmp/ and
#    backup/ are 0700 and unreadable as tanshu, and backup/ is state.
#
#    `set -o pipefail` matters: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
#
#    `--no-same-owner` is explicit because the source files are uid 1000
#    (`ubuntu`) on monoco. keep-id maps container uid 1000 onto tanshu, so files
#    that arrive owned by tanshu are already correct — restoring the old
#    ownership would be wrong.
set -o pipefail
ssh monoco 'sudo tar -C /var/lib/trillium -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/trilium -xf -'

# 5. Verify by content, over the whole tree on both sides. The paths are relative
#    on both sides, so the digests are comparable. `sudo` again on the source:
#    the 0700 directories are part of what is being compared.
#
#    Do this *before* the deploy below — Trilium may run schema migrations
#    against document.db on first start, and the two trees stop being comparable
#    at that point.
ssh monoco 'sudo sh -c "cd /var/lib/trillium && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum"'
ssh emily  'cd ~/podman-data/trilium && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by a subuid, or by root.
ssh emily 'find ~/podman-data/trilium ! -user tanshu'    # empty = correct

# 7. Deploy — after the copy, so the first start opens the real document rather
#    than creating an empty one beside it.
cd prepare-server && ansible-playbook -u tanshu ../trilium/playbook.yml

Ownership needs no chown pass on emily: the container writes as uid 1000, which keep-id maps onto tanshu, and the extraction already produced tanshu-owned files. That is also why --no-same-owner is explicit on the receiving side.

If you would rather not use sudo, or the 91 MB of logs are not worth carrying, exclude the three app-managed directories — the role recreates them at the right mode, and the play reads log/ back as evidence afterwards:

set -o pipefail
ssh monoco 'tar --exclude=./log --exclude=./tmp --exclude=./backup -C /var/lib/trillium -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/trilium -xf -'

That drops the 91 MB of regenerable logs and the 12 MB of backup/ — Trilium's own zipped document.db backups. Nothing in it is unique against the migration itself, but that is worth saying deliberately rather than by accident. If you use this variant, apply the same three exclusions to the find in step 5.

If the link is slow or drops, copy in two hops instead: resumable, and it shows progress. The first hop needs --rsync-path because of the 0700 directories.

rsync -aP --rsync-path="sudo rsync" monoco:/var/lib/trillium/ ~/trilium-copy/
rsync -aP ~/trilium-copy/ emily:~/podman-data/trilium/

Rolling back means rolling back on monoco: ssh monoco 'docker start trillium' is meaningful only while monoco's tree is still the one you copied. Once emily has served and written to document.db, the two have diverged — restore monoco's copy rather than starting it in place.

Verification

systemctl --user status trilium
journalctl --user -u trilium -f
podman logs trilium

# What the playbook checks for you:
podman healthcheck run trilium          # the unit's own HealthCmd=, not the image's

# The unit's mount is real and not the container's writable layer:
podman inspect trilium --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'

# The health the unit reports on its own timer, not just once:
podman inspect trilium --format '{{.State.Health.Status}}'

# The document came across: a migrated document.db is a few megabytes, not the
# kilobytes of one the app creates on an empty install.
ssh emily 'ls -ln ~/podman-data/trilium/document.db'
ssh emily 'command -v sqlite3 >/dev/null && sqlite3 ~/podman-data/trilium/document.db "pragma integrity_check"'

# The site, end to end.
curl -sSI https://notes.tanshu.com

Then open https://notes.tanshu.com in a browser, log in, and open a note. A curl cannot prove the migrated document is readable; only the app can.

Also exercise a write — edit a note — and confirm on the host that what the app wrote is owned by tanshu:

ssh emily 'ls -ln ~/podman-data/trilium/document.db* ~/podman-data/trilium/log/'

An edit touches document.db-wal and appends to today's log; both must be owned by tanshu. That is the check that catches a wrong keep-id; a read-only smoke test will not. The play reports the same thing from the other direction, by looking for a trilium-*.log touched since the unit started.

Then reboot the host and confirm the unit returns unattended — without loginctl enable-linger tanshu it will not, and the failure is silent.

Notes and gotchas

• Do not deploy the old image. The retired zadam/trilium reads /root/trilium-data and predates TriliumNext's schema migrations; pointed at this document.db it is at best a failure to start. The unit pins docker.io/triliumnext/notes:latest.

• The unit's health check is its own. Unlike the v0.95.0 on monoco, the v0.106.0 image ships no HEALTHCHECK, so podman healthcheck run would fail with "no defined healthcheck"; the unit probes the endpoint the app advertises in healthcheck-url instead. is-active alone cannot tell a serving server from a crash-looping one under Restart=always, which is why the play runs the healthcheck rather than trusting it.

• keep-id runs the container as the mapped uid, and this image needs root. The full explanation is in Settings; the short version is that without User=0 the image's own su node cannot work and the unit crash-loops. The three failures this playbook has actually produced, and what each means:

  In the journal	Cause
  Password: then su: Authentication failure	the container process is not root, so the image's privilege drop fails — User=0 is missing
  no defined healthcheck from podman healthcheck run	the image ships none and the unit's HealthCmd= is missing
  DB not initialized, please visit setup page	the app is serving a new, empty document — the copy from monoco has not run (or was wiped)

• The mount check is a mount check, not a file check. Audiobookshelf can assert its database appeared on the host, because the app creates it. Here the migration puts document.db on the host before the app runs, so its presence proves nothing — the play asserts against podman inspect instead, and checks the source as well as the destination, because a destination-only check is satisfied by an anonymous volume mounted where the bind mount should be. That is the one failure that is otherwise silent: the app runs perfectly and loses the document on the next recreate.

• latest floats, and it moved here. monoco runs v0.95.0 (built 2025-06-15, revision b9ae2265) and the pull on emily is v0.106.0 (built 2026-09-25, revision c0fdceba) — so this cutover upgrades the app as well as the runtime, and the first start migrates document.db forward. That is one-way for the copy on emily; monoco's own tree is untouched and remains the rollback. Pin a version tag or a digest in vars/default.yml if the migration should change only the runtime and leave the upgrade to a separate, deliberate step (podman image inspect docker.io/triliumnext/notes:latest --format '{{.Digest}}'). Take Trilium's own backup, or a stop-and-copy, before any upgrade.

• trustedReverseProxy=false in config.ini. Trilium does not trust X-Forwarded-For, so its own logs see the proxy rather than the client. That is the existing behaviour, carried across unchanged; set it to true (or to the Caddy network's CIDR) in config.ini if you want real client addresses there. It is app state, not something this playbook sets.

• The copy needs sudo on monoco because log/, tmp/ and backup/ are 0700, and backup/ is the one of the three that is state. tanshu cannot read them, and a copy that silently skips them looks like it worked.

• Verify before deploying, not after. The digest comparison in step 5 stops being meaningful once the app has opened document.db, because it may migrate it.

• No TZ. The Docker-era container set no environment at all, so the app runs on UTC — that is where the log filenames' dates come from. Trilium renders dates in the browser, so this is cosmetic, but it is a deliberate carry-over rather than an omission.

• Registering restarts all of Caddy. Writing the network unit and the drop-in changes Caddy's generated unit, and Quadlet has no live equivalent of docker network connect, so Caddy is recreated — a brief restart of every hosted site, not just this one. See caddy/readme.md, "Trade-off worth knowing".

• If the Caddyfile does not exist yet, this playbook creates it containing only the site block. That is a valid Caddyfile; the Caddy playbook's seeded global-options block is then not added, because its seed only fires on a missing or zero-length file. Deploy Caddy first on a fresh host.

• DNS. notes.tanshu.com must resolve to emily before Caddy can obtain a certificate for it. The certificates and ACME account key from monoco's /var/lib/caddy/data are not part of this migration — Caddy's own playbook and readme cover that. Until the name moves, emily's Caddy retries the ACME order and logs the failures; that is expected while monoco still holds the name, and the site keeps being served from monoco.

• Root's crontab cannot podman exec into another user's rootless containers, so anything scheduled belongs in a systemd user timer next to the Quadlet units. Trilium's own backup/ mechanism runs inside the app (noBackup=false in config.ini), so it keeps working without cron.

• Never back up a hot document.db. Same WAL problem as the migration: stop the container, or let the app's own backup do it.

• Do not restart the container on monoco. Its tree is a stale snapshot of a document emily has moved past.

Not done here

• Pinning the image by digest or version tag, or wiring up AutoUpdate=registry with podman auto-update.
• A backup script or systemd user timer for document.db — the app's own Settings → Backup into backup/ is the mechanism, and the 12 MB it has produced so far comes across with the copy.
• Setting TZ, or turning on trustedReverseProxy, both of which are runtime settings inside config.ini rather than Ansible variables.
• Renaming trillium to trilium on monoco, or updating the docs that call this service trillium: docs/podman-migration.md (§4.1, §4.2, §4.3, §5.5) and docs/podman-compat-audit.md (§3.2, §4).