playbooks/tandoor/readme.md

Tandoor (rootless Podman + Quadlet)

Runs Tandoor Recipes on emily as a rootless Podman container managed by a systemd user unit (Quadlet), behind the shared rootless Caddy, with its database in the shared rootless Postgres. Replaces the previous docker_container deployment on the old host.

The app's own uid is 0 — Tandoor's entrypoint starts as root and drops privileges, but root is what owns the data directories (see docs/podman-compat-audit.md §3.2). The unit therefore maps container root to the connecting user with UserNS=keep-id:uid=0,gid=0, so ~/podman-data/tandoor stays owned by tanshu and no subuid-owned files appear to break backups.

Ordering

Both shared services are already migrated and running on emily, so this playbook is the last step:

1. caddy/playbook.yml — the reverse proxy, which owns the Caddyfile this site appends to, supplies the caddy.service handlers, and is the one play that applies the rootless-Podman prerequisites.
2. postgres/playbook.yml — deployed, and its cluster brought across from monoco. The tandoor database is already restored (16 MB, table counts verified identical); see postgres/migrate-from-monoco.md.
3. This playbook.

The only thing left to move is the app's file data — see Migrating the file data. The database is done.

Tandoor's environment file sets POSTGRES_HOST=db. Until this playbook has attached tandoor_net to Postgres, nothing resolves as db on that network, so running this playbook is what makes the app reachable — there is no separate step.

Layout

tandoor/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/       # attaches this app's network to Postgres (+ its handler)
    ├── tandoor/        # data dirs, .env, the Quadlet unit, start + verify
    └── caddy/          # appends this site's block to the shared Caddyfile

Quadlet units and state:

On the host	What
~/.config/containers/systemd/tandoor.container	the generated unit (Ansible-templated)
~/.config/containers/systemd/tandoor.network	the app network, written by caddy/roles/register
~/.config/containers/systemd/caddy.container.d/50-tandoor.conf	attaches Caddy to that network
~/.config/containers/systemd/postgres.container.d/50-tandoor.conf	attaches Postgres to it
~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf	the shared db alias
~/podman-data/tandoor/	mediafiles/, staticfiles/, externalfiles/, .env

How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls the shared include in caddy/roles/register, which writes the network unit and the drop-in that pulls it into Caddy:

- name: Register Tandoor's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"

It runs in pre_tasks, before the app unit is started, because tandoor.container references tandoor.network: Quadlet fails unit generation — so systemctl --user start reports "unit not found" — when a referenced .network file does not exist yet. The Postgres attach in the next section depends on it for the same reason.

network_name is tandoor, so the file is tandoor.network and the Podman network is tandoor_net (NetworkName= in the network unit pins it, keeping the Docker-era name).

How it attaches to Postgres

Same idea at the other end: Postgres knows nothing about its clients, and this playbook uses the shared include in postgres/roles/attach rather than writing those drop-ins itself. That writes two files, both mergeable into postgres.container by Quadlet:

File	Content
postgres.container.d/50-tandoor.conf	Network=tandoor.network
postgres.container.d/10-network-aliases.conf	NetworkAlias=db

db is the alias Tandoor's .env actually asks for; postgres also resolves, from ContainerName=. The alias lives in one shared file with a fixed value because NetworkAlias= applies to every network a container joins — every app in this repo reaches Postgres as db, and a per-service alias file would make Quadlet pass --network-alias once per service.

roles/postgres wraps that include for two reasons:

• it defines the Restart Postgres handler the include notifies; and
• it flushes handlers immediately afterwards. On a first attach both drop-ins are new, so Postgres has to be recreated to join tandoor_net — and that restart is a handler, which by default runs at the end of the play, after the tandoor role has already started the app. The app would come up before db resolves, fail its own is-active check, and fail the play on a machine where nothing is wrong. Flushing applies the drop-in first; on later runs nothing is pending and it is a no-op.

Note the cost stated plainly in postgres/readme.md: a new drop-in recreates Postgres, which is a few seconds of downtime for every attached app, not just this one. Quadlet has no live equivalent of podman network connect.

Settings

Variable	Value / purpose
podman_user	Derived from the connection (ansible_facts['user_id']), not set by hand
podman_data_dir	~/podman-data
quadlet_dir	~/.config/containers/systemd
host	recipes.tanshu.com — the site block Caddy serves
container_name	tandoor, and the Quadlet unit / container name
image	docker.io/vabene1111/recipes:latest — fully qualified on purpose
network_name	tandoor → tandoor.network / tandoor_net
container_port	80, the port inside the container that Caddy proxies to
host_directory	~/podman-data/tandoor
postgres_container / postgres_unit	The shared database unit this app attaches to
caddy_name / caddyfile_path	The shared Caddy unit and its Caddyfile

The three podman_* / quadlet_dir vars are duplicated from caddy/vars/default.yml, as in every service playbook. That is a deliberate trade: a play cannot include the Caddy vars file wholesale, because it also sets host_directory (to the Caddy data directory) and every service defines that for itself. Keep them in sync. Caddy's fourth, podman_unprivileged_port_start, belongs to the podman tasks this play does not run — Tandoor publishes nothing host-side.

Like Caddy, podman_user is derived from the connection and the play carries no become. Because the podman tasks are not run here, there is no guard against connecting as root; the prerequisites, including that check, live in caddy/playbook.yml.

The app's secrets (SECRET_KEY) and database coordinates live in roles/tandoor/templates/tandoor.env.j2, rendered to ~/podman-data/tandoor/.env at mode 0600 and read by systemd via EnvironmentFile=, not by Ansible. The password is the fleet's shared 123456, which matches pg_password in postgres/vars/default.yml — a migrated app connects unchanged. Both are committed in plaintext as before; flagged as a follow-up, not changed here.

Running

ansible-playbook -u tanshu tandoor/playbook.yml

The play targets emily directly, so the inventory must contain it — the one in prepare-server/ does, and its ansible.cfg names it.

This play applies only Tandoor's own units and data. Its pre_tasks run Caddy's register include and nothing else. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them. To apply them on their own:

- name: Rootless Podman prerequisites
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"

systemctl --user enable tandoor is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Migrating the file data

The database already moved; this is the app's own files, and the playbook does not move them because they are on the old host while the playbook runs on the new one. Run this from the machine that holds the monoco and emily SSH aliases, as in postgres/migrate-from-monoco.md.

mediafiles/ is the only irreplaceable part, and on this host it is 900K; the 98 MB is staticfiles/, which is collectstatic output. Copy both anyway — the bind mount shadows whatever static files the image ships, so if the entrypoint does not re-run collectstatic, an empty volume is a site with no CSS, and copying stale files is the strictly safer failure mode.

# 1. Look before you copy: size, and whether it is readable without sudo.
ssh monoco 'du -sh /var/lib/tandoor/*; ls -ln /var/lib/tandoor'
ssh monoco 'find /var/lib/tandoor -type f ! -readable'   # empty = no sudo needed

# 2. Stop the old container so nothing writes while you copy. Leave it stopped:
#    its database is on emily now, and two live copies would diverge.
ssh monoco 'docker stop tandoor'

# 3. Create the destination (its parent exists from the Postgres move).
ssh emily 'mkdir -p ~/podman-data/tandoor'

# 4. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command. `.env` is excluded — Ansible templates it,
#    at mode 0600, before the unit ever starts.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
set -o pipefail
ssh monoco 'tar --exclude=./.env -C /var/lib/tandoor -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/tandoor -xf -'

#    If step 1 listed unreadable files, read them as root instead:
# ssh monoco 'sudo tar --exclude=./.env -C /var/lib/tandoor -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/tandoor -xf -'

# 5. Verify by content, and over the *whole* tree. Checking mediafiles alone
#    would pass on a copy that dropped almost everything: it is under 1 MB here
#    while staticfiles is the 98 MB.
ssh monoco 'cd /var/lib/tandoor && find . -type f ! -name .env -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
ssh emily  'cd ~/podman-data/tandoor && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/tandoor ! -user tanshu'    # empty = correct

# 7. Deploy.
cd prepare-server && ansible-playbook -u tanshu ../tandoor/playbook.yml

Ownership needs no chown pass: the app runs as uid 0 in-container, which keep-id maps to tanshu, so files extracted as tanshu are already right. That is also why --no-same-owner is explicit on the receiving side — the source is root-owned, and restoring that ownership would be wrong here.

If the link is slow or drops, do it in two hops instead: resumable, and it shows progress.

rsync -aP --exclude=.env monoco:/var/lib/tandoor/ ~/tandoor-copy/
rsync -aP ~/tandoor-copy/ emily:~/podman-data/tandoor/

Rolling back means rolling back both halves: ssh monoco 'docker start tandoor' is only meaningful while monoco's own Postgres 17 cluster is still intact.

Verification

systemctl --user status tandoor
journalctl --user -u tandoor -f
podman logs tandoor

# What the playbook checks for you:
podman exec tandoor getent hosts db          # the alias the .env asks for
podman exec tandoor getent hosts postgres

# The database really is the migrated one:
podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d tandoor -c '\dt' | head

# The site, end to end:
curl -sSI https://recipes.tanshu.com

Also exercise a write through the app (upload a recipe image) and confirm on the host that the new file under ~/podman-data/tandoor/mediafiles is owned by tanshu. That is the check that catches a wrong keep-id; a read-only smoke test will not.

Then reboot the host and confirm the unit returns unattended — without loginctl enable-linger tanshu it will not, and the failure is silent.

Notes and gotchas

• A 502 whose page says nginx is Tandoor's own front end, not a stray host nginx. The image runs nginx in front of gunicorn (80 → 8080, which is why podman ps shows both ports), and Caddy relays upstream response headers verbatim — so server: nginx reaches the browser from inside the container. During the startup window, while Django is still running migrations, that nginx answers 502 and Caddy passes it straight through. The role therefore waits for a real HTTP response from inside the container (Wait for Tandoor to serve HTTP) rather than trusting is-active; a 502 after a green run means the app, not the proxy.
• Client IPs. Caddy's access log may show a gateway address rather than the real client, depending on the rootless port forwarder. See caddy/readme.md, "Client IPs", before relying on it.
• Attaching restarts Postgres for everyone. See the note above; it is a few seconds of downtime for every attached app.
• Registering restarts all of Caddy. Same mechanism at the proxy end — a brief restart of every hosted site, not just this one.
• If the Caddyfile does not exist yet, this playbook creates it containing only the site block. That is a valid Caddyfile; the Caddy playbook's seeded global-options block is then not added, because its seed only fires on a missing or zero-length file. Deploy Caddy first on a fresh host.
• The backup scripts still assume Docker. ~/bin/tandoor-backup.sh uses --link, which Podman does not have; it needs the podman exec form from postgres/readme.md, "Backups", before the old host is decommissioned.
• DNS. recipes.tanshu.com must resolve to emily before Caddy can obtain a certificate for it.
• Do not restart the container on monoco. It would write to monoco's old cluster, which is being kept intact purely as a rollback.

Not done here

• Rotating the secrets that are committed in plaintext.
• Tandoor needs no cache: its env sets no REDIS_URL, so valkey is not attached to this network.