Replaces the docker_container deployment on monoco with a Quadlet systemd user
unit on emily, and moves state from /var/lib/gitea to ~/podman-data/gitea. Gitea
is SQLite -- the live app.ini on monoco says DB_TYPE = sqlite3 with
PATH = /data/gitea/gitea.db -- so there is no postgres role and the database
travels inside the data tree.
git-over-SSH is the part that could not simply be moved. Port 22 stays the host's
sshd, so no client changes its remote, but the second hop does: the Docker-era
design sshed into an sshd *inside* the container, and under keep-id that sshd runs
as a subuid which cannot read the authorized_keys Gitea writes as tanshu. It is
replaced by a `podman exec`, reached through a shim sshd runs as a new `git` login
user, a one-command sudoers rule, and an AuthorizedKeysCommand that hands sshd the
key file without the ownership check StrictModes would otherwise apply.
Two things stopped it starting, and both only showed up when it was run:
keep-id runs the container *process* as the uid it maps, not just the file
ownership, so the whole container came up as `git` while the image's start-up is
written to begin as root. /usr/bin/entrypoint rewrites /etc/passwd,
/etc/s6/gitea/setup prepares /data before the run script `su-exec git`s the
server, and the image's own CMD is s6-svscan, which has to create
.s6-svscan/lock inside root-owned /etc/s6. The container died on the last two --
"sed: can't create temp file '/etc/passwdXXXXXX': Permission denied" and
"s6-svscan: fatal: unable to open .s6-svscan/lock: Permission denied" -- and
Restart=always turned that into a crash loop, which `is-active` reports as
`activating` rather than as a failure. User=0 restores root for the start-up and
leaves the mapping alone: the server still runs as git through su-exec, which is
what satisfies mustNotRunAsRoot. (nextcloud.container carries the same pair for
the same reason.)
The `ssh/` directory under /data is only ever a mount point -- the in-image
sshd's host keys are mounted over it from ~/podman-data/gitea-sshd -- so it is
hidden inside the container and nothing ever chowns it. When Podman has to create
it, it lands in the data tree owned by the subuid, and the role's own "nothing
here belongs to another user" check fails on it, correctly. Creating it in the
role first is what keeps the tree wholly tanshu's.
Start, is-active and the health check are now one block whose rescue dumps
status, journal and podman logs, because they fail the same way: `systemctl
start` reports success as soon as systemd holds the unit, so a container that
dies a second later surfaces only as `activating` -- which is exactly how the two
failures above were reported, with the reason in the journal and nowhere the play
looked.
Verified on emily: the play runs to ok=27/changed=7/failed=0, the health check
and the stray-owner assertion both pass, and the ssh second hop still returns the
container's Gitea 28.1.0 through the wrapper. The role's comments and the readme
were corrected where they explained the old failures as EPERM from the root
phases: container root holds the usual capabilities inside its user namespace, so
those phases succeed and leave subuid-owned files behind, which is the real reason
the tree is created up front. The readme's list of "expected" chmod EPERM lines is
gone, because the log is clean. The data directory was emptied of the fresh
app.ini afterwards and the unit left stopped, ready for the copy from monoco.
The first run of this playbook against emily failed, and all three reasons are
worth recording because none of them is visible from the files alone.
keep-id runs the container *process* as the uid it maps, so
UserNS=keep-id:uid=1000,gid=1000 started the container as `node` rather than root
-- and the image's start-docker.sh is written to start as root, ending with
`exec su -c "node ./main.mjs" node`. `su` from a non-root uid asks for a password
the node account does not have, so the container died with "su: Authentication
failure" and Restart=always turned that into a crash loop. User=0 forces the
process back to root and changes nothing about the mapping: node is still tanshu,
so the app is still what writes the data directory and it stays tanshu's.
nextcloud.container carries the same pair for the same reason.
The health check had to be defined rather than inherited. `latest` floated between
the two pulls -- monoco runs v0.95.0 (built 2025-06-15) and emily pulled v0.106.0
(built 2026-09-25) -- and v0.106.0 ships no HEALTHCHECK where v0.95.0 had
docker_healthcheck.cjs, so `podman healthcheck run` failed with "no defined
healthcheck". The unit now defines HealthCmd= against /api/health-check, the
endpoint the app advertises itself in the healthcheck-url file it writes into the
data directory, and the image has curl for it (but not wget).
The verify and health-check steps are folded in with the start, because they fail
the same way and want the same evidence. `systemctl start` reports success as soon
as systemd holds the unit, so a container that dies a second later surfaced only
as "activating" from `is-active`; the reason -- that "su: Authentication failure"
-- was in the journal and nowhere the play looked. The rescue now dumps status,
journal and podman logs for any of the three, which is how the su failure was
found in the first place.
The version float becomes a documented risk rather than a hypothetical: the
cutover upgrades the app as well as the runtime, and the first start migrates
document.db forward. That is one-way for the copy on emily, so the readme compares
digests before the first start, keeps monoco's own tree as the rollback, and notes
that pinning a tag here would decouple the two changes.
Verified on emily rather than by inspection: the rendered unit was run first as a
throwaway Quadlet unit against a scratch volume, then the playbook itself ran to
ok=22/changed=6/failed=0 with the mount assertion passing, `healthcheck run`
returning 0, podman reporting healthy, and the app's files landing in
~/podman-data/trilium owned by tanshu. The data directory was emptied afterwards
and the unit left stopped, ready for the copy from monoco, which has not run yet.
The first deploy failed its wait. The app was up -- Rocket had launched on port
80 and systemd reported the unit active -- but `podman healthcheck run` kept
returning non-zero. Podman's healthcheck plumbing is unreliable for
Quadlet-managed rootless containers: the check is registered against the *user*
systemd manager, and when that fails the check never runs, leaving `podman
healthcheck run` to report that the container has no healthcheck at all even
though the image declares one (upstream issues #26066, #25034, #22804).
Exec the image's own /healthcheck.sh instead: the identical probe (curl
http://localhost:80/alive) with no plumbing in between, and still nothing the
play has to invent. The cost is that the path is hardcoded, which the image's
Dockerfile pins, so a moved script fails as a plain 127 rather than hanging.
The wait is now a block with a rescue that prints the probe's own stdout/stderr,
the container's health status and the last 20 log lines. The bare "non-zero
return code" the plain task reported is exactly the message that hides the
reason.
The readme's verification commands and a new gotcha record why the probe is
exec'd, so nobody "fixes" it back to `podman healthcheck run`.
The second real run failed with:
(13)Permission denied: AH00072: make_sock: could not bind to address [::]:80
no listening sockets available, shutting down
`keep-id:uid=33` does not only build the mapping, it also runs the container
process as the mapped uid -- so the entrypoint and apache2 were running as
www-data, and www-data has no capabilities. Port 80 inside the container's own
network namespace therefore belongs to root. The caddy unit is the control: it
binds 80 and 443 with keep-id:uid=0, and container-root in a userns does have
NET_BIND_SERVICE.
Tested on emily before writing this, because the last two fixes here were reasoned
from the wrong model:
podman run --userns keep-id:uid=33,gid=33 IMG id -> uid=33(www-data)
podman run --userns keep-id:uid=33,gid=33 --user 0 IMG id -> uid=0(root)
So User=0 restores the process root, and it does not touch the mapping: www-data
is still the connection user, so the webroot stays theirs, the entrypoint's
--chown lands on them, and occ keeps running as www-data through the entrypoint's
own su. That is the configuration the image is written for -- it expects root to
bind the port and to chown the webroot -- which is the real argument for this
option over the two that also tested clean: --sysctl
net.ipv4.ip_unprivileged_port_start=80 and --cap-add NET_BIND_SERVICE. Both would
have fixed the symptom while leaving the entrypoint permanently unprivileged,
which is exactly what produced this failure and the redis-session.ini one before
it.
The readme's debugging section now lists all three failures this playbook has
produced and what each means, and the rescue message points there instead of
repeating a stale list.
The failure that prompted this is the environment file being absent:
Error: parsing file ".../nextcloud.env": ... no such file or directory
nextcloud.service: Main process exited, code=exited, status=125/n/a
The play creates that file before it installs or starts anything, so this cannot
come from a play run: it is a start outside one, after ~/podman-data/nextcloud was
removed. The unit is WantedBy=default.target, so it retries at boot and fails the
same way until the play is run again. The readme now says so, because the fix is
not obvious from a 125.
The rescue message was also stale: it sent readers to look for "Permission denied"
under /usr/local/etc/php/conf.d, which was the PHP session file this playbook no
longer creates or mounts, and it had lost the reset-failed hint when the message
was shortened. It now names the failure that actually happens (a removed webroot),
keeps the database/cache/image pointers that are still real, and gives the command
for a unit left failed by an earlier attempt.
The role created ~/podman-data/nextcloud/redis-session.ini and the unit bind-mounted
it over the image path, purely because the entrypoint cannot write there itself.
Removed: the task, the mount, and -- because it is the only thing that triggers the
write -- REDIS_HOST from the environment file. The entrypoint now skips
configure_redis_session entirely and starts.
The Redis attach stays. The valkey role still puts nextcloud_net on the cache
container with the alias `redis`, so an instance whose config.php names that host
reaches it. Writing that configuration is config.php's business, and the readme
gives the three `occ config:system:set` lines for anyone who wants it, including
that this playbook will not do it for them.
The only thing lost is PHP session storage, which falls back to the container's own
filesystem -- a redeploy logs users out. That is the trade for not shipping a file
whose sole purpose is to work around a directory the image owns as root.
The env template and the valkey role comment both say why REDIS_HOST is absent, so
that adding it back is recognisable as the regression it is.
The timer was justified partly on XDG_RUNTIME_DIR: a user crontab "would need it
faked into every line". That is wrong here. The caddy playbook enables lingering,
so /run/user/<uid> exists whether or not anyone is logged in, and that is where
podman keeps the state `exec` reads. Nothing needs to be set.
The one real constraint is that the job cannot go in root's crontab: root cannot
`podman exec` into another user's rootless containers. This user's crontab is
fine, and the play runs as that user, so `ansible.builtin.cron` with no `user:`
writes exactly there.
That replaces four tasks and two template files with one task and no files, and it
is what the service had before the Quadlet conversion. The cost is that cron sends
its output to local mail instead of the journal; the readme says so and names the
timer as the alternative.
While here, the XDG_RUNTIME_DIR asides come out of the comments that carried them:
the play runs over SSH, so `systemctl --user` works because systemd's user manager
is reachable over that connection, and that is all a reader needs to know.
The role had grown machinery that the problem does not require. Removed:
- The init-sync lock pre-create. It guarded against one file in the webroot being
owned by a subuid, which is cosmetic: the file is 0644, so it is readable and
removable by host-side tools either way.
- The image ID read/pull pair. No other service playbook pulls an image, and the
update path is two commands the readme now spells out.
- redis-session.ini.j2 and its three variables. The mount is load-bearing -- the
entrypoint cannot write into the image's conf.d and dies trying -- but the
template is not: the entrypoint writes that file itself on every start. One
touched empty file and one Volume line replace the template and the variables.
- The config.php stat, the wizard note, the maintenance-mode wait, the occ status
report, the getent checks, the timer report. The installed/not-installed
branching existed only because the readiness gates were written for an installed
instance; with a gate that fits both, none of the branching is needed.
- Comments that narrated this play's earlier runs rather than the system.
What is left is four tasks before the start, a start-plus-gate block whose rescue
collects the journal, and four tasks for the background-job timer: 149 lines and
about 130 of them code, against 313 before. The readme loses its migration-era
bulk for the same reason and is 297 lines, in line with the other services'.
The one gate is now Nextcloud's own status.php inside the container. It answers
only once the entrypoint has finished -- including any `occ upgrade`, which runs
before Apache starts, so an update in progress is waited out rather than missed --
it reports `installed` either way, so a fresh webroot and an installed instance go
through the same check, and it carries `maintenance` as well, which is the one
state a probe would otherwise call healthy while every page returned 503.
The playbook is meant to be usable on its own, on any host, the way every other
one here is: the first run creates the container, and every run after that
converges it. It had picked up a migration shape instead -- a readme that assumed
a webroot copied in from monoco, a warning that nagged when config.php was
missing, and, worse, readiness gates that only passed for an instance that was
already installed.
That last one was a real bug for the intended use. On a host with no webroot, the
image initialises the code, writes no config.php because it has no admin
credentials to install with, and serves the setup wizard -- so `occ status` never
reports installed: true, and the play would have spent its whole retry budget and
then failed on a first run that was going exactly as designed.
What replaces it:
- The readiness gate is Apache answering status.php inside the container, which
is Nextcloud's own endpoint and reports `installed` either way. It is the one
thing `is-active` cannot tell us, and it passes for a wizard that has not been
completed.
- config.php's presence then decides the report: wait for maintenance mode to
clear and print `occ status` for an instance, or say what is left for a human to
do for a fresh one -- and succeed. The two optional credentials in
vars/default.yml install unattended instead, in which case the same endpoints
come back installed and the wizard note never appears.
- The app creates its own database. Postgres is a shared unit with one default
database and nothing else in this repo knows this app's database name, so a
fresh cluster previously reached the installer and failed on "database
nextcloud does not exist". It is a lookup and a `createdb`, a no-op afterwards,
and it fails with the instruction to run postgres/playbook.yml if the cluster
is not there rather than with a bare psql error.
- Updates. An update in a Quadlet deployment is "pull, then recreate": the unit
names a tag, `podman run` reuses the local image, and nothing fetches a newer
one. The role pulls on every run and compares the image ID either side, so an
unchanged tag is not a change and a running site is not restarted for nothing.
- Handlers are flushed before verification, so a queued recreate -- new image,
changed unit, changed environment -- happens before the checks rather than after
them, and the checks describe what will keep running.
Removed: the entire monoco migration section and its copy commands, the
config.php warning that existed only to serve it, the host-specific notes, and the
narrative comments about this play's first runs. References to the audit and
migration planning docs go with them; the ownership reasoning is stated inline,
since it does not depend on where an instance came from. `hosts:` is now the only
host-specific thing in the directory.
The readme is rewritten around the two paths it now has -- "First run" and
"Updates" -- with the deploy order, the session-file trap and the debugging
commands kept.
The real first run of this play did not fail on a transition. The journal shows
the entrypoint dying on every attempt:
nextcloud[302063]: => Configuring PHP session handler...
nextcloud[302063]: /entrypoint.sh: 121: cannot create
/usr/local/etc/php/conf.d/redis-session.ini: Permission denied
With REDIS_HOST set, the entrypoint generates PHP's session-handler config on
every start, and the target directory belongs to the image's root: the php base
image does `mkdir -p "$PHP_INI_DIR/conf.d"` with no chown. The uid this container
runs as has no write bit on it, the entrypoint exits 2 before Apache is reached,
Restart=always retries that, and systemd's start limit turns it into a `failed`
unit. Nothing in that shape says "permissions": the symptom is a unit state and a
message that only exists in journalctl.
The container now mounts a host-owned file over that exact path. A bind-mounted
file replaces the directory entry, so the directory's ownership stops mattering
and the write lands on podman_user. The role templates the five lines the
entrypoint would have written, so a refused write is no longer fatal either --
PHP still gets Redis sessions instead of a crash loop.
This also corrects a reading of the previous run that this play got wrong and
documented wrongly: `activating` is not evidence of a healthy start. A
crash-looping container spends its time `activating` between restarts, so the
earlier "reports a healthy start as a failure" comment and commit message were
describing a start that was in fact failing. The wait is still right -- it stops
promptly at a terminal state and hands over to a rescue that has the logs -- but
its comment now says what it actually buys.
Apache's own runtime directories need none of this, and the php base image is why:
it chowns /var/run/apache2, /var/lock/apache2 and /var/log/apache2 to www-data and
chmods them 1777, so only the session file was ever going to bite. The readme
gains a section on it, the diagnostic commands the play now collects, and a note
that the uid mapping is not to be derived from a `docker exec id -u` of 0.
The second run of this play stopped at "Wait for the Nextcloud unit to become
active" with:
{"attempts": 2, "cmd": ["systemctl", "--user", "is-active",
"nextcloud.service"], "rc": 3, "stdout": "failed"}
The wait itself did the right thing: `failed` is not `activating`, so it stopped
immediately instead of retrying for five minutes. But it sat outside the start
block, so the rescue that pulls status, journalctl and podman logs never ran, and
the failure carried no reason at all.
Move the wait inside "Start Nextcloud" so one rescue covers both, and widen the
diagnostics for the failure that is most likely after an interrupted run:
- `systemctl --user cat` -- the generated unit, which settles what systemd thinks
it is waiting for when a unit sits in `activating`.
- `podman ps -a --filter name=...` -- a container left behind under this name
makes `podman run` refuse to start, `Restart=always` retries that into systemd's
start limit, and the unit ends up `failed` while `podman logs` shows only the
previous container and explains nothing.
The final message now names that case and the commands that clear it, and points
an `activating`-for-the-whole-budget unit at the generated unit and journalctl
instead.
Running this play against an empty ~/podman-data/nextcloud does not fail. It
creates the webroot, writes the environment file, starts the image, and the
entrypoint initialises a fresh, uninstalled instance there -- its own code
rsynced in, data/.ocdata created, and no config.php, because there is no
NEXTCLOUD_ADMIN_USER to install with.
Copying monoco's tree on top of that then merges two versions of the code, which
is the drift the rest of this role warns about, and nothing in the play's output
says so. The role now stats config/config.php before the container starts and
warns when it is missing, naming both cases: a copy that never landed, and a play
that ran before the copy.
It warns rather than fails because installing a new instance through the web UI is
a legitimate use of this playbook and has no config.php until the installer
finishes. readme.md gains the recovery step next to the deploy step, including
why the rm -rf needs no `podman unshare`.
The first real run of this play stopped at "Verify Nextcloud is running":
fatal: [emily]: FAILED! => {"cmd": ["systemctl", "--user", "is-active",
"nextcloud.service"], "rc": 3, "stdout": "activating"}
Nothing was wrong with the container. `systemctl start` returns while the unit
is still `activating` -- the start job completes before conmon has signalled
readiness -- and `is-active` exits 3 for that state, so checking once reports a
healthy start as a failure. Every other app role here checks is-active only
after a wait that already tolerates startup (postgres waits on pg_isready,
valkey on valkey-cli ping, bitwarden on its healthcheck); Nextcloud is the one
where the check comes first, because its readiness gates are the expensive ones.
The wait retries only through the transition. A unit that is `failed` or
`inactive` is not going to become `active` by waiting, so that fails immediately
with the state in the message rather than five minutes later. What genuinely
needs patience -- the entrypoint rsyncing code and running `occ upgrade` before
Apache answers -- is the readiness gate that follows, which already has the
richer rescue.
Replaces the 2022 nginx + docker_container playbook with a Quadlet systemd user
unit on emily, and moves state from /var/lib/trillium to ~/podman-data/trilium.
There is no database to attach: the state is one SQLite file (document.db, with
its WAL sidecars), config.ini, session_secret.txt and Trilium's own backup/
directory, so unlike bitwarden/ and tandoor/ this play has no postgres role and
nothing to restore from the shared cluster.
The old playbook was wrong about what monoco actually runs, in three ways that
each break the deployment on their own. monoco has been on TriliumNext since
June 2025 (`triliumnext/notes:latest`, v0.95.0), not the retired zadam/trilium
the playbook pulls, and its document.db carries the schema migrations that came
with the change; the data directory is /home/node/trilium-data, not
/root/trilium-data; and the container sits on trillium_net behind Caddy, with
nothing published on the host. All three were read off the live host.
UserNS=keep-id:uid=1000,gid=1000, because the entrypoint starts as container-root
and drops to the image's node user, and uid 1000 is also what owns the whole
Docker-era tree -- so the copied data arrives already in the right shape and
needs no chown pass. The unit defines no HealthCmd=: unlike audiobookshelf, this
image ships its own HEALTHCHECK, and the play runs it with podman healthcheck
run.
The mount assertion is a mount check and not a file check, which is where this
play has to differ from audiobookshelf's. Asserting that document.db appeared on
the host works there because the app creates it; here the migration puts it on
the host before the app has ever run, so the file proves nothing. It asserts
against podman inspect instead, and checks Source as well as Destination -- a
Destination-only check would be satisfied by an anonymous volume mounted where
the bind mount should be, which is exactly the silent case. Bind mount,
anonymous volume, no mount and a payload with no Mounts key were each run
against the expression before this commit; only the first passes.
The site is unchanged; the service name is not. The live container is
`trillium`, with an extra l, and that exists only in the hand-made docker run
that created it. Everything on emily is `trilium`, following the app and this
directory, and the only place the misspelling survives is the copy source path
/var/lib/trillium that the readme documents.
The readme carries the monoco -> emily copy command by command: why the source
needs sudo (log/, tmp/ and backup/ are 0700 and backup/ is state), why the copy
cannot be hot (document.db is live SQLite in WAL mode), and why the digest
comparison has to happen before the first start, since Trilium can migrate the
document as it opens it. Of the 106 MB, 91 MB is regenerable logs, and the
no-sudo variant that skips them says out loud what it drops.
The integration goes through the shared include rather than a local copy --
caddy/roles/register for the network and the Caddy drop-in -- as in the other
service playbooks, and it runs in pre_tasks because Quadlet fails unit
generation, not merely startup, when a referenced .network file does not exist
yet. The dead files/nginx.conf.j2 goes with the nginx front end: Caddy handles
the WebSocket upgrade the live editor needs and imposes no request-body limit,
so there is nothing to carry over.
Three documents written before the container migration, all referenced from the
service readmes, so they belong in the tree:
- podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's
architecture, privileges and namespaces; the four things that actually break
(anonymous volumes, per-container uids, tooling that assumes Docker, image
naming); and the per-container uid matrix that the UserNS=keep-id lines in the
Quadlet units come from.
- podman-migration.md -- the plan: target architecture, the ownership/chown
pass, the Quadlet unit shape, and the low-port and lingering prerequisites.
- backup-plan.md -- per-service irreplaceable state, the consistency problems
(live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1
architecture.
caddy, postgres, tandoor and bitwarden each link into these from their readme,
so leaving them untracked left those links dead.
The DO Community placeholder servers are gone. The file now lists the three
hosts the playbooks address by name -- monoco, clair and emily -- under a `web`
group, with the Python 3 interpreter setting kept as an all-hosts var.
No ansible_host is given for any of them: the control node's ~/.ssh/config
already maps each alias to an address, the same convention
prepare-server/inventory documents. Like that file, this one is only read when
it is named explicitly (`-i hosts`) -- Ansible does not discover an inventory by
name.
The comment claimed every attached app "sees a cold cache". That is not what
happens. RDB persistence is on by default, the recreate is a graceful stop, and
valkey-server writes a final dump.rdb on SIGTERM that the new container loads
back off the bind mount -- so keys survive, which is the part the readme already
gets right and this comment did not.
What does not survive is availability for the seconds the container is down:
Nextcloud keeps its PHP sessions and memcache.locking there, and Yamtrack uses it
as its Celery broker, so those seconds are seconds in which logins and queued
tasks are unreachable. Only a hard kill or a power loss costs changes since the
last save point. That is the warning a future reader needs, and it is what the
nextcloud playbook's own handler comment now points at.
Replaces the docker_container playbook, which bind-mounted /var/lib/nextcloud at
/var/www/html on monoco and ran cron.php from root's crontab. Nextcloud now runs
on emily behind the shared Caddy, with its database in the shared Postgres and
its cache in the shared Valkey, each attached the way every other service
attaches to them.
/var/www/html is the unusual part: it holds the code as well as the state --
version.php, core/, lib/ and apps/ alongside config/config.php, data/ and
custom_apps/ -- and the entrypoint's version check reads version.php from the
mount. That is why the migration copies the whole tree rather than only the three
irreplaceable directories (docs/backup-plan.md 2.1 lists the rest as regenerable,
and they are, but only from an image that is the same version or newer). It is
also what makes an image bump an in-place upgrade: a newer image rsyncs its code
in and runs `occ upgrade` on first start, and an older one refuses to start at
all, so the version on both hosts is worth checking before the copy.
UserNS=keep-id:uid=33,gid=33. 33 is www-data, the uid the server actually writes
as -- `docker exec nextcloud id -u` reports 0, because the entrypoint drops
privileges after starting (docs/podman-compat-audit.md 3.2). 33 is also what the
data is owned by on monoco, so the copy needs no chown pass: extracting as the
connection user is already the ownership the container expects. One file is
created by container-root and never chowned, the entrypoint's
nextcloud-init-sync.lock, so the role pre-creates it -- the entrypoint truncates
that file rather than recreating it, so its owner then never changes, and the
webroot ends up with no subuid-owned file in it.
The Valkey attach is not optional. config.php carries memcache.locking pointed at
Redis, written by the image from REDIS_HOST, and the same variable rewrites PHP's
session-handler config on every start -- so an unresolvable `redis` does not just
make things slower, it fails uploads at write time. Hence the `redis` alias
rather than the container name, and hence roles/valkey flushing handlers before
the app starts, the same as roles/postgres.
cron.php moves from root's crontab to a systemd user timer, because root cannot
`podman exec` into another user's rootless containers and there is no container
user to own a crontab (docs/podman-migration.md 8b). Unlike the Quadlet units
this pair is plain and is genuinely enabled, which is what brings it back after a
reboot; journalctl --user -u nextcloud-cron shows every run, which the crontab
never did.
Readiness is three gates, because no one of them is sufficient: is-active (which
Restart=always satisfies even while crash-looping), `occ status` reporting
installed, and status.php served over loopback by Apache rather than by the PHP
CLI. The second is followed by a wait for maintenance mode to clear rather than
an assert: maintenance mode is on for the whole of a first-start `occ upgrade`,
so asserting it off would fail a healthy deploy halfway through. The case that
never clears is a config.php copied with `maintenance => true`, which
docs/backup-plan.md 3.4 tells an operator to set before a consistent dump, and
which is invisible from outside because status.php is exempt from maintenance
mode -- so the play names it and prints the command that clears it.
Register and attach ordering is the same as the other service playbooks: register
runs in pre_tasks, because the unit references nextcloud.network and Quadlet
fails unit generation when that file does not exist, and both attaches flush
handlers before the app starts, which is also what pulls Caddy onto the network.
The Caddy snippet is unchanged, HSTS header included; only caddyfile_path moves,
from /var/lib/caddy/conf/Caddyfile to ~/podman-data/caddy/conf/Caddyfile. The
Docker-era roles/network is deleted rather than ported: it existed to attach an
already-created container to a network with the `redis` alias, and
podman_network has no connected:/appends: equivalent
(docs/podman-compat-audit.md 3.3). Nothing is published host-side; Caddy reaches
nextcloud:80 over nextcloud_net.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
valkey/playbook.yml has to have been run first, or the first Restart Valkey
handler fails against a unit that does not exist yet.
Replaces the docker_container playbook, which published 127.0.0.1:6379 and kept
its /data in an anonymous Docker volume -- the one the audit flagged because no
backup script can see it (docs/podman-compat-audit.md 3.1). Valkey becomes the
base Quadlet unit, the same shape as Postgres: it knows nothing about its
clients, and a service attaches itself by dropping a file into
valkey.container.d/ via valkey/roles/attach. Nothing on the host needs 6379, so
there is no PublishPort at all rather than a narrower one.
The name stays `valkey` for the directory, container, unit and Podman network,
and only the alias clients connect on is `redis`: Nextcloud's config.php
(written by the image from REDIS_HOST=redis), outline and overleaf reach it as
`redis`, while paperless-ngx and yamtrack use the container name. Renaming would
touch every REDIS_HOST for no gain.
The uid/gid mapping is the detail worth spelling out, because the audit only
records the uid. The Alpine image's `valkey` account is 999:1000, not 999:999 --
Alpine already has a gid 999, and the Dockerfile says so before creating the
group with the next id. 999 is the uid the anonymous volume was owned by, so the
uid matches what was already on disk, but mapping only the uid would leave every
file valkey-server writes, dump.rdb included, group-owned by a subgid. Both ids
map onto the connection user, so ~/podman-data/valkey stays tanshu:tanshu.
Persistence stays on deliberately, and the mount with it. The contents are
disposable -- nothing is copied off monoco, and a cold start is survivable -- but
this is not purely a cache: Nextcloud keeps its PHP sessions and
memcache.locking in it, and Yamtrack uses it as its Celery broker. With RDB on, a
rebuild is a graceful stop that writes a final snapshot on SIGTERM that the next
container loads, so a redeploy does not log every Nextcloud user out or discard
queued tasks. Tmpfs=/data with `save ""` would remove the mount and the whole
keep-id mapping with it, and is the right shape for a pure cache, but it would
make every redeploy exactly that.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
Two ordering details are load-bearing:
- The attach include asserts that <name>.network exists before writing a drop-in
that names it. Quadlet rejects the whole valkey unit, with an error that never
mentions the drop-in, when a referenced .network file is missing.
- A service that attaches flushes handlers before starting the app. The restart
that joins Valkey to the new network is a handler, so by default it runs at the
end of the play, after the app has already come up and failed to resolve
`redis`. nextcloud/roles/valkey is the working example.
The play waits on `valkey-cli ping` inside the container rather than trusting
systemctl is-active. Restart=always reports active even while the container
crash-loops, which is exactly what a wrong keep-id looks like.
Replaces the docker_container role on monoco with a Quadlet systemd user unit on
emily, and moves state from /var/lib/bitwarden to ~/podman-data/bitwarden.
Postgres is authoritative for the vault and already moved with the cluster
restore, so only the file data comes across -- and the stale db.sqlite3 that has
sat beside it for years is deliberately excluded from that copy. The role warns
if it finds the file in the data directory rather than failing, because with
DATABASE_URL set it is inert but looks exactly like a restore source.
The image runs as container-root and writes /data as root, so
UserNS=keep-id:uid=0,gid=0 maps that onto the connecting user and leaves nothing
owned by a subuid.
Both integrations go through the shared includes rather than local copies --
caddy/roles/register for the network and the Caddy drop-in, postgres/roles/attach
for the database -- so this app writes the same drop-in bytes as every other
service and a new service never means editing Caddy or Postgres. That is also
what lets the Docker-era roles/network go: it existed to attach an
already-created container to a network (docker_network's connected:/appends:),
which podman_network has no equivalent for. The dead files/nginx.conf.j2 goes
with it -- Caddy handles the WebSocket upgrade and imposes no request-body limit,
so its 525M client_max_body_size needs no replacement.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play, after the app has already started and failed to resolve the
database.
- The play waits on the image's own health check (podman healthcheck run) rather
than on systemctl is-active. Vaultwarden runs its Diesel migrations before
Rocket answers, and Restart=always means even a crash-looping container is
reported active.
DATABASE_URL and DOMAIN move into a 0600 environment file read through
EnvironmentFile=, since the Quadlet unit is world-readable and DATABASE_URL
carries the password. The credentials are unchanged, so the restored database
connects with no reset. readme.md carries the monoco -> emily file copy command
by command.
Replaces the docker_container role with a Quadlet systemd user unit, and moves
state from /var/lib/tandoor to ~/podman-data/tandoor. Tandoor's entrypoint
starts as root and drops privileges, but root is what ends up owning the data,
so keep-id maps container root onto the connecting user and everything stays
owned by tanshu rather than a subuid.
The play does not run the rootless-Podman prerequisites: they are the shared
task file at caddy/roles/podman, and a host that has run caddy/playbook.yml has
them. Both integrations go through the shared includes rather than local copies
-- caddy/roles/register for the network and the Caddy drop-in, postgres/roles/
attach for the database -- so this app writes the same drop-in bytes as every
other service and a new service never means editing Caddy or Postgres.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play -- after the app has already started and failed to resolve `db`.
- The role waits for a real HTTP response from inside the container instead of
trusting systemctl is-active. The entrypoint runs Django migrations before
gunicorn answers, and until it does the image's own nginx returns 502, which
Caddy relays verbatim -- complete with `server: nginx`, which reads as a stray
reverse proxy that is not actually there.
Also drops the dead nginx.conf.j2 and the network role, both superseded by the
shared includes, and moves .env into the role's templates/ as tandoor.env.j2,
rendered at mode 0600 and read through EnvironmentFile=.
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept
its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same
shape as Caddy: it knows nothing about its clients, and a service attaches itself
by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing
here changes when a service is added or removed.
Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both
the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its
default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker),
and the 18 entrypoint refuses to start when it finds a cluster in the old layout.
So postgres_major is the single knob: it picks the image tag and the PGDATA path
together, and the mount is now the parent directory. That also settles the
migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster
comes across as a pg_dumpall.
Three things that bit us and are worth keeping:
- PGDATA and its version directory are created by the role, not left to the
container. The entrypoint runs as container-root, which under keep-id is a
subuid on the host, and it only chowns PGDATA and its contents -- so
~/podman-data/postgres/18 would have been stranded subuid-owned, and the host
user could not even rm -rf it without podman unshare.
- NetworkAlias= applies to every network the container joins, which is what a
database hub wants, but it is one fixed value in a shared drop-in rather than
per-service: two services writing their own would make Quadlet pass
--network-alias twice. It is not in postgres.container itself because on a
fresh host that unit has no Network= at all.
- The credentials live in a 0600 file read with EnvironmentFile=, not in
Environment= lines: the unit is world-readable in ~/.config.
The play no longer uses become, and does not apply the rootless-Podman
prerequisites -- those are the shared task file at caddy/roles/podman, and the
three variables they need are duplicated at the top of vars/default.yml.
migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer
and restore commands, the checks that prove the restore is complete (per-database
table counts against the source), and the verified result.
The dead nginx/php-fpm templates in postgres/files/ are removed.
Replaces the docker_container role with Quadlet systemd user units under
~/.config/containers/systemd, and moves state from /var/lib/caddy to
~/podman-data/caddy.
Caddy is now the base unit and no longer lists its upstream networks. Each
service registers itself by writing its own <name>.network unit plus a drop-in
into caddy.container.d/, so adding or removing a service never means editing the
Caddy playbook. Drop-ins are used rather than blockinfile because this playbook
re-templates caddy.container on every run, which would wipe appended blocks.
The play runs as the connecting user instead of using become: the rootless units
belong to it, so the connection's PAM session is what makes systemctl --user
work without setting XDG_RUNTIME_DIR. podman_user and the data directory are
derived from facts, so the play fails loudly if that resolves to root.
Two things that bit us and are worth keeping:
- The privileged-port sysctl is applied inline with sysctl -w, not via a
handler. Handlers only run at the end of the play, by which point Caddy has
already failed to bind 80/443.
- The Caddyfile is seeded with a minimal valid config. Caddy exits 1 with a
bare "EOF" on an empty config file, so touching it into existence made the
service crash-loop.
Two plays: bootstrap as the provider account (ubuntu, over bootstrap.pem), then
everything else as the user that first play just created. The split is forced
rather than stylistic -- the second play logs in as that user, so its
authorized_keys entry and its NOPASSWD sudoers entry both have to exist before
it starts. That is why the key tasks live in the 'user' role and 'wheel' runs
in the first play: move either into play 2 and it fails with an auth error or
"Missing sudo password".
Both accounts and both keys are variables. The inventory sits beside the
playbook and is read via ansible.cfg, which Ansible picks up from the current
working directory rather than the playbook's, so it has to be run from this
directory. bootstrap.pem is gitignored -- it is the provider key for the host.
It lives beside the app it deploys now, with podman building the image and an
inventory carrying one entry per Site (which is what the local 'hosts: monoco'
edit was reaching for). Nothing in this repo referenced it.
The app's health check ran on the target host against the container name, which
only resolves inside the Docker network, so the wait could never succeed. Poll
the image's HEALTHCHECK instead and fail with a readable message. Also pin
site.env to the app user when it exists, and say how to install the provisioning
script on the server.
Deploys the menu app for one Site: pulls the tag, creates the data root, runs
the container against a config baked into the image, waits for health, verifies
the data directory, and points Caddy at the app with a marker it can update in
place. The parallel run uses a temporary hostname; cutover sets the retire
flags.
Updated the web nginx.conf and the source is not given in the readme.
Restricted to gondor.
Webserver image and container names are now picked from the vars file.
tanshu