The second run of this play stopped at "Wait for the Nextcloud unit to become
active" with:
{"attempts": 2, "cmd": ["systemctl", "--user", "is-active",
"nextcloud.service"], "rc": 3, "stdout": "failed"}
The wait itself did the right thing: `failed` is not `activating`, so it stopped
immediately instead of retrying for five minutes. But it sat outside the start
block, so the rescue that pulls status, journalctl and podman logs never ran, and
the failure carried no reason at all.
Move the wait inside "Start Nextcloud" so one rescue covers both, and widen the
diagnostics for the failure that is most likely after an interrupted run:
- `systemctl --user cat` -- the generated unit, which settles what systemd thinks
it is waiting for when a unit sits in `activating`.
- `podman ps -a --filter name=...` -- a container left behind under this name
makes `podman run` refuse to start, `Restart=always` retries that into systemd's
start limit, and the unit ends up `failed` while `podman logs` shows only the
previous container and explains nothing.
The final message now names that case and the commands that clear it, and points
an `activating`-for-the-whole-budget unit at the generated unit and journalctl
instead.
Running this play against an empty ~/podman-data/nextcloud does not fail. It
creates the webroot, writes the environment file, starts the image, and the
entrypoint initialises a fresh, uninstalled instance there -- its own code
rsynced in, data/.ocdata created, and no config.php, because there is no
NEXTCLOUD_ADMIN_USER to install with.
Copying monoco's tree on top of that then merges two versions of the code, which
is the drift the rest of this role warns about, and nothing in the play's output
says so. The role now stats config/config.php before the container starts and
warns when it is missing, naming both cases: a copy that never landed, and a play
that ran before the copy.
It warns rather than fails because installing a new instance through the web UI is
a legitimate use of this playbook and has no config.php until the installer
finishes. readme.md gains the recovery step next to the deploy step, including
why the rm -rf needs no `podman unshare`.
The first real run of this play stopped at "Verify Nextcloud is running":
fatal: [emily]: FAILED! => {"cmd": ["systemctl", "--user", "is-active",
"nextcloud.service"], "rc": 3, "stdout": "activating"}
Nothing was wrong with the container. `systemctl start` returns while the unit
is still `activating` -- the start job completes before conmon has signalled
readiness -- and `is-active` exits 3 for that state, so checking once reports a
healthy start as a failure. Every other app role here checks is-active only
after a wait that already tolerates startup (postgres waits on pg_isready,
valkey on valkey-cli ping, bitwarden on its healthcheck); Nextcloud is the one
where the check comes first, because its readiness gates are the expensive ones.
The wait retries only through the transition. A unit that is `failed` or
`inactive` is not going to become `active` by waiting, so that fails immediately
with the state in the message rather than five minutes later. What genuinely
needs patience -- the entrypoint rsyncing code and running `occ upgrade` before
Apache answers -- is the readiness gate that follows, which already has the
richer rescue.
Replaces the 2022 nginx + docker_container playbook with a Quadlet systemd user
unit on emily, and moves state from /var/lib/trillium to ~/podman-data/trilium.
There is no database to attach: the state is one SQLite file (document.db, with
its WAL sidecars), config.ini, session_secret.txt and Trilium's own backup/
directory, so unlike bitwarden/ and tandoor/ this play has no postgres role and
nothing to restore from the shared cluster.
The old playbook was wrong about what monoco actually runs, in three ways that
each break the deployment on their own. monoco has been on TriliumNext since
June 2025 (`triliumnext/notes:latest`, v0.95.0), not the retired zadam/trilium
the playbook pulls, and its document.db carries the schema migrations that came
with the change; the data directory is /home/node/trilium-data, not
/root/trilium-data; and the container sits on trillium_net behind Caddy, with
nothing published on the host. All three were read off the live host.
UserNS=keep-id:uid=1000,gid=1000, because the entrypoint starts as container-root
and drops to the image's node user, and uid 1000 is also what owns the whole
Docker-era tree -- so the copied data arrives already in the right shape and
needs no chown pass. The unit defines no HealthCmd=: unlike audiobookshelf, this
image ships its own HEALTHCHECK, and the play runs it with podman healthcheck
run.
The mount assertion is a mount check and not a file check, which is where this
play has to differ from audiobookshelf's. Asserting that document.db appeared on
the host works there because the app creates it; here the migration puts it on
the host before the app has ever run, so the file proves nothing. It asserts
against podman inspect instead, and checks Source as well as Destination -- a
Destination-only check would be satisfied by an anonymous volume mounted where
the bind mount should be, which is exactly the silent case. Bind mount,
anonymous volume, no mount and a payload with no Mounts key were each run
against the expression before this commit; only the first passes.
The site is unchanged; the service name is not. The live container is
`trillium`, with an extra l, and that exists only in the hand-made docker run
that created it. Everything on emily is `trilium`, following the app and this
directory, and the only place the misspelling survives is the copy source path
/var/lib/trillium that the readme documents.
The readme carries the monoco -> emily copy command by command: why the source
needs sudo (log/, tmp/ and backup/ are 0700 and backup/ is state), why the copy
cannot be hot (document.db is live SQLite in WAL mode), and why the digest
comparison has to happen before the first start, since Trilium can migrate the
document as it opens it. Of the 106 MB, 91 MB is regenerable logs, and the
no-sudo variant that skips them says out loud what it drops.
The integration goes through the shared include rather than a local copy --
caddy/roles/register for the network and the Caddy drop-in -- as in the other
service playbooks, and it runs in pre_tasks because Quadlet fails unit
generation, not merely startup, when a referenced .network file does not exist
yet. The dead files/nginx.conf.j2 goes with the nginx front end: Caddy handles
the WebSocket upgrade the live editor needs and imposes no request-body limit,
so there is nothing to carry over.
Three documents written before the container migration, all referenced from the
service readmes, so they belong in the tree:
- podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's
architecture, privileges and namespaces; the four things that actually break
(anonymous volumes, per-container uids, tooling that assumes Docker, image
naming); and the per-container uid matrix that the UserNS=keep-id lines in the
Quadlet units come from.
- podman-migration.md -- the plan: target architecture, the ownership/chown
pass, the Quadlet unit shape, and the low-port and lingering prerequisites.
- backup-plan.md -- per-service irreplaceable state, the consistency problems
(live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1
architecture.
caddy, postgres, tandoor and bitwarden each link into these from their readme,
so leaving them untracked left those links dead.
The DO Community placeholder servers are gone. The file now lists the three
hosts the playbooks address by name -- monoco, clair and emily -- under a `web`
group, with the Python 3 interpreter setting kept as an all-hosts var.
No ansible_host is given for any of them: the control node's ~/.ssh/config
already maps each alias to an address, the same convention
prepare-server/inventory documents. Like that file, this one is only read when
it is named explicitly (`-i hosts`) -- Ansible does not discover an inventory by
name.
The comment claimed every attached app "sees a cold cache". That is not what
happens. RDB persistence is on by default, the recreate is a graceful stop, and
valkey-server writes a final dump.rdb on SIGTERM that the new container loads
back off the bind mount -- so keys survive, which is the part the readme already
gets right and this comment did not.
What does not survive is availability for the seconds the container is down:
Nextcloud keeps its PHP sessions and memcache.locking there, and Yamtrack uses it
as its Celery broker, so those seconds are seconds in which logins and queued
tasks are unreachable. Only a hard kill or a power loss costs changes since the
last save point. That is the warning a future reader needs, and it is what the
nextcloud playbook's own handler comment now points at.
Replaces the docker_container playbook, which bind-mounted /var/lib/nextcloud at
/var/www/html on monoco and ran cron.php from root's crontab. Nextcloud now runs
on emily behind the shared Caddy, with its database in the shared Postgres and
its cache in the shared Valkey, each attached the way every other service
attaches to them.
/var/www/html is the unusual part: it holds the code as well as the state --
version.php, core/, lib/ and apps/ alongside config/config.php, data/ and
custom_apps/ -- and the entrypoint's version check reads version.php from the
mount. That is why the migration copies the whole tree rather than only the three
irreplaceable directories (docs/backup-plan.md 2.1 lists the rest as regenerable,
and they are, but only from an image that is the same version or newer). It is
also what makes an image bump an in-place upgrade: a newer image rsyncs its code
in and runs `occ upgrade` on first start, and an older one refuses to start at
all, so the version on both hosts is worth checking before the copy.
UserNS=keep-id:uid=33,gid=33. 33 is www-data, the uid the server actually writes
as -- `docker exec nextcloud id -u` reports 0, because the entrypoint drops
privileges after starting (docs/podman-compat-audit.md 3.2). 33 is also what the
data is owned by on monoco, so the copy needs no chown pass: extracting as the
connection user is already the ownership the container expects. One file is
created by container-root and never chowned, the entrypoint's
nextcloud-init-sync.lock, so the role pre-creates it -- the entrypoint truncates
that file rather than recreating it, so its owner then never changes, and the
webroot ends up with no subuid-owned file in it.
The Valkey attach is not optional. config.php carries memcache.locking pointed at
Redis, written by the image from REDIS_HOST, and the same variable rewrites PHP's
session-handler config on every start -- so an unresolvable `redis` does not just
make things slower, it fails uploads at write time. Hence the `redis` alias
rather than the container name, and hence roles/valkey flushing handlers before
the app starts, the same as roles/postgres.
cron.php moves from root's crontab to a systemd user timer, because root cannot
`podman exec` into another user's rootless containers and there is no container
user to own a crontab (docs/podman-migration.md 8b). Unlike the Quadlet units
this pair is plain and is genuinely enabled, which is what brings it back after a
reboot; journalctl --user -u nextcloud-cron shows every run, which the crontab
never did.
Readiness is three gates, because no one of them is sufficient: is-active (which
Restart=always satisfies even while crash-looping), `occ status` reporting
installed, and status.php served over loopback by Apache rather than by the PHP
CLI. The second is followed by a wait for maintenance mode to clear rather than
an assert: maintenance mode is on for the whole of a first-start `occ upgrade`,
so asserting it off would fail a healthy deploy halfway through. The case that
never clears is a config.php copied with `maintenance => true`, which
docs/backup-plan.md 3.4 tells an operator to set before a consistent dump, and
which is invisible from outside because status.php is exempt from maintenance
mode -- so the play names it and prints the command that clears it.
Register and attach ordering is the same as the other service playbooks: register
runs in pre_tasks, because the unit references nextcloud.network and Quadlet
fails unit generation when that file does not exist, and both attaches flush
handlers before the app starts, which is also what pulls Caddy onto the network.
The Caddy snippet is unchanged, HSTS header included; only caddyfile_path moves,
from /var/lib/caddy/conf/Caddyfile to ~/podman-data/caddy/conf/Caddyfile. The
Docker-era roles/network is deleted rather than ported: it existed to attach an
already-created container to a network with the `redis` alias, and
podman_network has no connected:/appends: equivalent
(docs/podman-compat-audit.md 3.3). Nothing is published host-side; Caddy reaches
nextcloud:80 over nextcloud_net.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
valkey/playbook.yml has to have been run first, or the first Restart Valkey
handler fails against a unit that does not exist yet.
Replaces the docker_container playbook, which published 127.0.0.1:6379 and kept
its /data in an anonymous Docker volume -- the one the audit flagged because no
backup script can see it (docs/podman-compat-audit.md 3.1). Valkey becomes the
base Quadlet unit, the same shape as Postgres: it knows nothing about its
clients, and a service attaches itself by dropping a file into
valkey.container.d/ via valkey/roles/attach. Nothing on the host needs 6379, so
there is no PublishPort at all rather than a narrower one.
The name stays `valkey` for the directory, container, unit and Podman network,
and only the alias clients connect on is `redis`: Nextcloud's config.php
(written by the image from REDIS_HOST=redis), outline and overleaf reach it as
`redis`, while paperless-ngx and yamtrack use the container name. Renaming would
touch every REDIS_HOST for no gain.
The uid/gid mapping is the detail worth spelling out, because the audit only
records the uid. The Alpine image's `valkey` account is 999:1000, not 999:999 --
Alpine already has a gid 999, and the Dockerfile says so before creating the
group with the next id. 999 is the uid the anonymous volume was owned by, so the
uid matches what was already on disk, but mapping only the uid would leave every
file valkey-server writes, dump.rdb included, group-owned by a subgid. Both ids
map onto the connection user, so ~/podman-data/valkey stays tanshu:tanshu.
Persistence stays on deliberately, and the mount with it. The contents are
disposable -- nothing is copied off monoco, and a cold start is survivable -- but
this is not purely a cache: Nextcloud keeps its PHP sessions and
memcache.locking in it, and Yamtrack uses it as its Celery broker. With RDB on, a
rebuild is a graceful stop that writes a final snapshot on SIGTERM that the next
container loads, so a redeploy does not log every Nextcloud user out or discard
queued tasks. Tmpfs=/data with `save ""` would remove the mount and the whole
keep-id mapping with it, and is the right shape for a pure cache, but it would
make every redeploy exactly that.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
Two ordering details are load-bearing:
- The attach include asserts that <name>.network exists before writing a drop-in
that names it. Quadlet rejects the whole valkey unit, with an error that never
mentions the drop-in, when a referenced .network file is missing.
- A service that attaches flushes handlers before starting the app. The restart
that joins Valkey to the new network is a handler, so by default it runs at the
end of the play, after the app has already come up and failed to resolve
`redis`. nextcloud/roles/valkey is the working example.
The play waits on `valkey-cli ping` inside the container rather than trusting
systemctl is-active. Restart=always reports active even while the container
crash-loops, which is exactly what a wrong keep-id looks like.
Replaces the docker_container role on monoco with a Quadlet systemd user unit on
emily, and moves state from /var/lib/bitwarden to ~/podman-data/bitwarden.
Postgres is authoritative for the vault and already moved with the cluster
restore, so only the file data comes across -- and the stale db.sqlite3 that has
sat beside it for years is deliberately excluded from that copy. The role warns
if it finds the file in the data directory rather than failing, because with
DATABASE_URL set it is inert but looks exactly like a restore source.
The image runs as container-root and writes /data as root, so
UserNS=keep-id:uid=0,gid=0 maps that onto the connecting user and leaves nothing
owned by a subuid.
Both integrations go through the shared includes rather than local copies --
caddy/roles/register for the network and the Caddy drop-in, postgres/roles/attach
for the database -- so this app writes the same drop-in bytes as every other
service and a new service never means editing Caddy or Postgres. That is also
what lets the Docker-era roles/network go: it existed to attach an
already-created container to a network (docker_network's connected:/appends:),
which podman_network has no equivalent for. The dead files/nginx.conf.j2 goes
with it -- Caddy handles the WebSocket upgrade and imposes no request-body limit,
so its 525M client_max_body_size needs no replacement.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play, after the app has already started and failed to resolve the
database.
- The play waits on the image's own health check (podman healthcheck run) rather
than on systemctl is-active. Vaultwarden runs its Diesel migrations before
Rocket answers, and Restart=always means even a crash-looping container is
reported active.
DATABASE_URL and DOMAIN move into a 0600 environment file read through
EnvironmentFile=, since the Quadlet unit is world-readable and DATABASE_URL
carries the password. The credentials are unchanged, so the restored database
connects with no reset. readme.md carries the monoco -> emily file copy command
by command.
Replaces the docker_container role with a Quadlet systemd user unit, and moves
state from /var/lib/tandoor to ~/podman-data/tandoor. Tandoor's entrypoint
starts as root and drops privileges, but root is what ends up owning the data,
so keep-id maps container root onto the connecting user and everything stays
owned by tanshu rather than a subuid.
The play does not run the rootless-Podman prerequisites: they are the shared
task file at caddy/roles/podman, and a host that has run caddy/playbook.yml has
them. Both integrations go through the shared includes rather than local copies
-- caddy/roles/register for the network and the Caddy drop-in, postgres/roles/
attach for the database -- so this app writes the same drop-in bytes as every
other service and a new service never means editing Caddy or Postgres.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play -- after the app has already started and failed to resolve `db`.
- The role waits for a real HTTP response from inside the container instead of
trusting systemctl is-active. The entrypoint runs Django migrations before
gunicorn answers, and until it does the image's own nginx returns 502, which
Caddy relays verbatim -- complete with `server: nginx`, which reads as a stray
reverse proxy that is not actually there.
Also drops the dead nginx.conf.j2 and the network role, both superseded by the
shared includes, and moves .env into the role's templates/ as tandoor.env.j2,
rendered at mode 0600 and read through EnvironmentFile=.
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept
its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same
shape as Caddy: it knows nothing about its clients, and a service attaches itself
by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing
here changes when a service is added or removed.
Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both
the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its
default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker),
and the 18 entrypoint refuses to start when it finds a cluster in the old layout.
So postgres_major is the single knob: it picks the image tag and the PGDATA path
together, and the mount is now the parent directory. That also settles the
migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster
comes across as a pg_dumpall.
Three things that bit us and are worth keeping:
- PGDATA and its version directory are created by the role, not left to the
container. The entrypoint runs as container-root, which under keep-id is a
subuid on the host, and it only chowns PGDATA and its contents -- so
~/podman-data/postgres/18 would have been stranded subuid-owned, and the host
user could not even rm -rf it without podman unshare.
- NetworkAlias= applies to every network the container joins, which is what a
database hub wants, but it is one fixed value in a shared drop-in rather than
per-service: two services writing their own would make Quadlet pass
--network-alias twice. It is not in postgres.container itself because on a
fresh host that unit has no Network= at all.
- The credentials live in a 0600 file read with EnvironmentFile=, not in
Environment= lines: the unit is world-readable in ~/.config.
The play no longer uses become, and does not apply the rootless-Podman
prerequisites -- those are the shared task file at caddy/roles/podman, and the
three variables they need are duplicated at the top of vars/default.yml.
migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer
and restore commands, the checks that prove the restore is complete (per-database
table counts against the source), and the verified result.
The dead nginx/php-fpm templates in postgres/files/ are removed.
Replaces the docker_container role with Quadlet systemd user units under
~/.config/containers/systemd, and moves state from /var/lib/caddy to
~/podman-data/caddy.
Caddy is now the base unit and no longer lists its upstream networks. Each
service registers itself by writing its own <name>.network unit plus a drop-in
into caddy.container.d/, so adding or removing a service never means editing the
Caddy playbook. Drop-ins are used rather than blockinfile because this playbook
re-templates caddy.container on every run, which would wipe appended blocks.
The play runs as the connecting user instead of using become: the rootless units
belong to it, so the connection's PAM session is what makes systemctl --user
work without setting XDG_RUNTIME_DIR. podman_user and the data directory are
derived from facts, so the play fails loudly if that resolves to root.
Two things that bit us and are worth keeping:
- The privileged-port sysctl is applied inline with sysctl -w, not via a
handler. Handlers only run at the end of the play, by which point Caddy has
already failed to bind 80/443.
- The Caddyfile is seeded with a minimal valid config. Caddy exits 1 with a
bare "EOF" on an empty config file, so touching it into existence made the
service crash-loop.
Two plays: bootstrap as the provider account (ubuntu, over bootstrap.pem), then
everything else as the user that first play just created. The split is forced
rather than stylistic -- the second play logs in as that user, so its
authorized_keys entry and its NOPASSWD sudoers entry both have to exist before
it starts. That is why the key tasks live in the 'user' role and 'wheel' runs
in the first play: move either into play 2 and it fails with an auth error or
"Missing sudo password".
Both accounts and both keys are variables. The inventory sits beside the
playbook and is read via ansible.cfg, which Ansible picks up from the current
working directory rather than the playbook's, so it has to be run from this
directory. bootstrap.pem is gitignored -- it is the provider key for the host.
It lives beside the app it deploys now, with podman building the image and an
inventory carrying one entry per Site (which is what the local 'hosts: monoco'
edit was reaching for). Nothing in this repo referenced it.
The app's health check ran on the target host against the container name, which
only resolves inside the Docker network, so the wait could never succeed. Poll
the image's HEALTHCHECK instead and fail with a readable message. Also pin
site.env to the app user when it exists, and say how to install the provisioning
script on the server.
Deploys the menu app for one Site: pulls the tag, creates the data root, runs
the container against a config baked into the image, waits for health, verifies
the data directory, and points Caddy at the app with a marker it can update in
place. The parallel run uses a temporary hostname; cutover sets the retire
flags.
Updated the web nginx.conf and the source is not given in the readme.
Restricted to gondor.
Webserver image and container names are now picked from the vars file.
Restricted it to gondor host.
Moved back to postgres on the same host. This does away with the etc_hosts link, but kept the variable and playbook entry as a template if and when needed.
tanshu