The comment claimed every attached app "sees a cold cache". That is not what
happens. RDB persistence is on by default, the recreate is a graceful stop, and
valkey-server writes a final dump.rdb on SIGTERM that the new container loads
back off the bind mount -- so keys survive, which is the part the readme already
gets right and this comment did not.
What does not survive is availability for the seconds the container is down:
Nextcloud keeps its PHP sessions and memcache.locking there, and Yamtrack uses it
as its Celery broker, so those seconds are seconds in which logins and queued
tasks are unreachable. Only a hard kill or a power loss costs changes since the
last save point. That is the warning a future reader needs, and it is what the
nextcloud playbook's own handler comment now points at.
Replaces the docker_container playbook, which bind-mounted /var/lib/nextcloud at
/var/www/html on monoco and ran cron.php from root's crontab. Nextcloud now runs
on emily behind the shared Caddy, with its database in the shared Postgres and
its cache in the shared Valkey, each attached the way every other service
attaches to them.
/var/www/html is the unusual part: it holds the code as well as the state --
version.php, core/, lib/ and apps/ alongside config/config.php, data/ and
custom_apps/ -- and the entrypoint's version check reads version.php from the
mount. That is why the migration copies the whole tree rather than only the three
irreplaceable directories (docs/backup-plan.md 2.1 lists the rest as regenerable,
and they are, but only from an image that is the same version or newer). It is
also what makes an image bump an in-place upgrade: a newer image rsyncs its code
in and runs `occ upgrade` on first start, and an older one refuses to start at
all, so the version on both hosts is worth checking before the copy.
UserNS=keep-id:uid=33,gid=33. 33 is www-data, the uid the server actually writes
as -- `docker exec nextcloud id -u` reports 0, because the entrypoint drops
privileges after starting (docs/podman-compat-audit.md 3.2). 33 is also what the
data is owned by on monoco, so the copy needs no chown pass: extracting as the
connection user is already the ownership the container expects. One file is
created by container-root and never chowned, the entrypoint's
nextcloud-init-sync.lock, so the role pre-creates it -- the entrypoint truncates
that file rather than recreating it, so its owner then never changes, and the
webroot ends up with no subuid-owned file in it.
The Valkey attach is not optional. config.php carries memcache.locking pointed at
Redis, written by the image from REDIS_HOST, and the same variable rewrites PHP's
session-handler config on every start -- so an unresolvable `redis` does not just
make things slower, it fails uploads at write time. Hence the `redis` alias
rather than the container name, and hence roles/valkey flushing handlers before
the app starts, the same as roles/postgres.
cron.php moves from root's crontab to a systemd user timer, because root cannot
`podman exec` into another user's rootless containers and there is no container
user to own a crontab (docs/podman-migration.md 8b). Unlike the Quadlet units
this pair is plain and is genuinely enabled, which is what brings it back after a
reboot; journalctl --user -u nextcloud-cron shows every run, which the crontab
never did.
Readiness is three gates, because no one of them is sufficient: is-active (which
Restart=always satisfies even while crash-looping), `occ status` reporting
installed, and status.php served over loopback by Apache rather than by the PHP
CLI. The second is followed by a wait for maintenance mode to clear rather than
an assert: maintenance mode is on for the whole of a first-start `occ upgrade`,
so asserting it off would fail a healthy deploy halfway through. The case that
never clears is a config.php copied with `maintenance => true`, which
docs/backup-plan.md 3.4 tells an operator to set before a consistent dump, and
which is invisible from outside because status.php is exempt from maintenance
mode -- so the play names it and prints the command that clears it.
Register and attach ordering is the same as the other service playbooks: register
runs in pre_tasks, because the unit references nextcloud.network and Quadlet
fails unit generation when that file does not exist, and both attaches flush
handlers before the app starts, which is also what pulls Caddy onto the network.
The Caddy snippet is unchanged, HSTS header included; only caddyfile_path moves,
from /var/lib/caddy/conf/Caddyfile to ~/podman-data/caddy/conf/Caddyfile. The
Docker-era roles/network is deleted rather than ported: it existed to attach an
already-created container to a network with the `redis` alias, and
podman_network has no connected:/appends: equivalent
(docs/podman-compat-audit.md 3.3). Nothing is published host-side; Caddy reaches
nextcloud:80 over nextcloud_net.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
valkey/playbook.yml has to have been run first, or the first Restart Valkey
handler fails against a unit that does not exist yet.
Replaces the docker_container playbook, which published 127.0.0.1:6379 and kept
its /data in an anonymous Docker volume -- the one the audit flagged because no
backup script can see it (docs/podman-compat-audit.md 3.1). Valkey becomes the
base Quadlet unit, the same shape as Postgres: it knows nothing about its
clients, and a service attaches itself by dropping a file into
valkey.container.d/ via valkey/roles/attach. Nothing on the host needs 6379, so
there is no PublishPort at all rather than a narrower one.
The name stays `valkey` for the directory, container, unit and Podman network,
and only the alias clients connect on is `redis`: Nextcloud's config.php
(written by the image from REDIS_HOST=redis), outline and overleaf reach it as
`redis`, while paperless-ngx and yamtrack use the container name. Renaming would
touch every REDIS_HOST for no gain.
The uid/gid mapping is the detail worth spelling out, because the audit only
records the uid. The Alpine image's `valkey` account is 999:1000, not 999:999 --
Alpine already has a gid 999, and the Dockerfile says so before creating the
group with the next id. 999 is the uid the anonymous volume was owned by, so the
uid matches what was already on disk, but mapping only the uid would leave every
file valkey-server writes, dump.rdb included, group-owned by a subgid. Both ids
map onto the connection user, so ~/podman-data/valkey stays tanshu:tanshu.
Persistence stays on deliberately, and the mount with it. The contents are
disposable -- nothing is copied off monoco, and a cold start is survivable -- but
this is not purely a cache: Nextcloud keeps its PHP sessions and
memcache.locking in it, and Yamtrack uses it as its Celery broker. With RDB on, a
rebuild is a graceful stop that writes a final snapshot on SIGTERM that the next
container loads, so a redeploy does not log every Nextcloud user out or discard
queued tasks. Tmpfs=/data with `save ""` would remove the mount and the whole
keep-id mapping with it, and is the right shape for a pure cache, but it would
make every redeploy exactly that.
The rootless-Podman prerequisites are not applied here: they are the shared task
file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them.
Two ordering details are load-bearing:
- The attach include asserts that <name>.network exists before writing a drop-in
that names it. Quadlet rejects the whole valkey unit, with an error that never
mentions the drop-in, when a referenced .network file is missing.
- A service that attaches flushes handlers before starting the app. The restart
that joins Valkey to the new network is a handler, so by default it runs at the
end of the play, after the app has already come up and failed to resolve
`redis`. nextcloud/roles/valkey is the working example.
The play waits on `valkey-cli ping` inside the container rather than trusting
systemctl is-active. Restart=always reports active even while the container
crash-loops, which is exactly what a wrong keep-id looks like.
Replaces the docker_container role on monoco with a Quadlet systemd user unit on
emily, and moves state from /var/lib/bitwarden to ~/podman-data/bitwarden.
Postgres is authoritative for the vault and already moved with the cluster
restore, so only the file data comes across -- and the stale db.sqlite3 that has
sat beside it for years is deliberately excluded from that copy. The role warns
if it finds the file in the data directory rather than failing, because with
DATABASE_URL set it is inert but looks exactly like a restore source.
The image runs as container-root and writes /data as root, so
UserNS=keep-id:uid=0,gid=0 maps that onto the connecting user and leaves nothing
owned by a subuid.
Both integrations go through the shared includes rather than local copies --
caddy/roles/register for the network and the Caddy drop-in, postgres/roles/attach
for the database -- so this app writes the same drop-in bytes as every other
service and a new service never means editing Caddy or Postgres. That is also
what lets the Docker-era roles/network go: it existed to attach an
already-created container to a network (docker_network's connected:/appends:),
which podman_network has no equivalent for. The dead files/nginx.conf.j2 goes
with it -- Caddy handles the WebSocket upgrade and imposes no request-body limit,
so its 525M client_max_body_size needs no replacement.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play, after the app has already started and failed to resolve the
database.
- The play waits on the image's own health check (podman healthcheck run) rather
than on systemctl is-active. Vaultwarden runs its Diesel migrations before
Rocket answers, and Restart=always means even a crash-looping container is
reported active.
DATABASE_URL and DOMAIN move into a 0600 environment file read through
EnvironmentFile=, since the Quadlet unit is world-readable and DATABASE_URL
carries the password. The credentials are unchanged, so the restored database
connects with no reset. readme.md carries the monoco -> emily file copy command
by command.
Replaces the docker_container role with a Quadlet systemd user unit, and moves
state from /var/lib/tandoor to ~/podman-data/tandoor. Tandoor's entrypoint
starts as root and drops privileges, but root is what ends up owning the data,
so keep-id maps container root onto the connecting user and everything stays
owned by tanshu rather than a subuid.
The play does not run the rootless-Podman prerequisites: they are the shared
task file at caddy/roles/podman, and a host that has run caddy/playbook.yml has
them. Both integrations go through the shared includes rather than local copies
-- caddy/roles/register for the network and the Caddy drop-in, postgres/roles/
attach for the database -- so this app writes the same drop-in bytes as every
other service and a new service never means editing Caddy or Postgres.
Three ordering details are load-bearing:
- The register include runs in pre_tasks, before the unit is started. Quadlet
fails unit *generation*, not merely startup, when a referenced .network file
does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
joins Postgres to this network is a handler, so by default it runs at the end
of the play -- after the app has already started and failed to resolve `db`.
- The role waits for a real HTTP response from inside the container instead of
trusting systemctl is-active. The entrypoint runs Django migrations before
gunicorn answers, and until it does the image's own nginx returns 502, which
Caddy relays verbatim -- complete with `server: nginx`, which reads as a stray
reverse proxy that is not actually there.
Also drops the dead nginx.conf.j2 and the network role, both superseded by the
shared includes, and moves .env into the role's templates/ as tandoor.env.j2,
rendered at mode 0600 and read through EnvironmentFile=.
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept
its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same
shape as Caddy: it knows nothing about its clients, and a service attaches itself
by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing
here changes when a service is added or removed.
Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both
the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its
default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker),
and the 18 entrypoint refuses to start when it finds a cluster in the old layout.
So postgres_major is the single knob: it picks the image tag and the PGDATA path
together, and the mount is now the parent directory. That also settles the
migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster
comes across as a pg_dumpall.
Three things that bit us and are worth keeping:
- PGDATA and its version directory are created by the role, not left to the
container. The entrypoint runs as container-root, which under keep-id is a
subuid on the host, and it only chowns PGDATA and its contents -- so
~/podman-data/postgres/18 would have been stranded subuid-owned, and the host
user could not even rm -rf it without podman unshare.
- NetworkAlias= applies to every network the container joins, which is what a
database hub wants, but it is one fixed value in a shared drop-in rather than
per-service: two services writing their own would make Quadlet pass
--network-alias twice. It is not in postgres.container itself because on a
fresh host that unit has no Network= at all.
- The credentials live in a 0600 file read with EnvironmentFile=, not in
Environment= lines: the unit is world-readable in ~/.config.
The play no longer uses become, and does not apply the rootless-Podman
prerequisites -- those are the shared task file at caddy/roles/podman, and the
three variables they need are duplicated at the top of vars/default.yml.
migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer
and restore commands, the checks that prove the restore is complete (per-database
table counts against the source), and the verified result.
The dead nginx/php-fpm templates in postgres/files/ are removed.
Replaces the docker_container role with Quadlet systemd user units under
~/.config/containers/systemd, and moves state from /var/lib/caddy to
~/podman-data/caddy.
Caddy is now the base unit and no longer lists its upstream networks. Each
service registers itself by writing its own <name>.network unit plus a drop-in
into caddy.container.d/, so adding or removing a service never means editing the
Caddy playbook. Drop-ins are used rather than blockinfile because this playbook
re-templates caddy.container on every run, which would wipe appended blocks.
The play runs as the connecting user instead of using become: the rootless units
belong to it, so the connection's PAM session is what makes systemctl --user
work without setting XDG_RUNTIME_DIR. podman_user and the data directory are
derived from facts, so the play fails loudly if that resolves to root.
Two things that bit us and are worth keeping:
- The privileged-port sysctl is applied inline with sysctl -w, not via a
handler. Handlers only run at the end of the play, by which point Caddy has
already failed to bind 80/443.
- The Caddyfile is seeded with a minimal valid config. Caddy exits 1 with a
bare "EOF" on an empty config file, so touching it into existence made the
service crash-loop.
Two plays: bootstrap as the provider account (ubuntu, over bootstrap.pem), then
everything else as the user that first play just created. The split is forced
rather than stylistic -- the second play logs in as that user, so its
authorized_keys entry and its NOPASSWD sudoers entry both have to exist before
it starts. That is why the key tasks live in the 'user' role and 'wheel' runs
in the first play: move either into play 2 and it fails with an auth error or
"Missing sudo password".
Both accounts and both keys are variables. The inventory sits beside the
playbook and is read via ansible.cfg, which Ansible picks up from the current
working directory rather than the playbook's, so it has to be run from this
directory. bootstrap.pem is gitignored -- it is the provider key for the host.
It lives beside the app it deploys now, with podman building the image and an
inventory carrying one entry per Site (which is what the local 'hosts: monoco'
edit was reaching for). Nothing in this repo referenced it.
The app's health check ran on the target host against the container name, which
only resolves inside the Docker network, so the wait could never succeed. Poll
the image's HEALTHCHECK instead and fail with a readable message. Also pin
site.env to the app user when it exists, and say how to install the provisioning
script on the server.
Deploys the menu app for one Site: pulls the tag, creates the data root, runs
the container against a config baked into the image, waits for health, verifies
the data directory, and points Caddy at the app with a marker it can update in
place. The parallel run uses a temporary hostname; cutover sets the retire
flags.
Updated the web nginx.conf and the source is not given in the readme.
Restricted to gondor.
Webserver image and container names are now picked from the vars file.
Restricted it to gondor host.
Moved back to postgres on the same host. This does away with the etc_hosts link, but kept the variable and playbook entry as a template if and when needed.
tanshu