playbooks/tandoor/readme.md

# Tandoor (rootless Podman + Quadlet)

Runs [Tandoor Recipes](https://tandoor.dev/) on `emily` as a **rootless Podman**
container managed by a **systemd user unit** (Quadlet), behind the shared
[rootless Caddy](../caddy/readme.md), with its database in the shared
[rootless Postgres](../postgres/readme.md). Replaces the previous
`docker_container` deployment on the old host.

The app's own uid is **0** — Tandoor's entrypoint starts as root and drops
privileges, but root is what owns the data directories (see
[docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). The unit
therefore maps container root to the connecting user with
`UserNS=keep-id:uid=0,gid=0`, so `~/podman-data/tandoor` stays owned by
`tanshu` and no subuid-owned files appear to break backups.

## Ordering

Both shared services are already migrated and running on `emily`, so this
playbook is the last step:

1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns
   the Caddyfile this site appends to, supplies the `caddy.service` handlers, and
   is the one play that applies the rootless-Podman prerequisites.
2. [postgres/playbook.yml](../postgres/playbook.yml) — deployed, and its cluster
   brought across from monoco. **The `tandoor` database is already restored**
   (16 MB, table counts verified identical); see
   [postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md).
3. This playbook.

The only thing left to move is the app's **file** data — see
[Migrating the file data](#migrating-the-file-data). The database is done.

Tandoor's environment file sets `POSTGRES_HOST=db`. Until this playbook has
attached `tandoor_net` to Postgres, nothing resolves as `db` on that network, so
running this playbook is what makes the app reachable — there is no separate
step.

## Layout

```
tandoor/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/       # attaches this app's network to Postgres (+ its handler)
    ├── tandoor/        # data dirs, .env, the Quadlet unit, start + verify
    └── caddy/          # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/tandoor.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/tandoor.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-tandoor.conf` | attaches Caddy to that network |
| `~/.config/containers/systemd/postgres.container.d/50-tandoor.conf` | attaches Postgres to it |
| `~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf` | the shared `db` alias |
| `~/podman-data/tandoor/` | `mediafiles/`, `staticfiles/`, `externalfiles/`, `.env` |

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook
calls the shared include in [caddy/roles/register](../caddy/roles/register/tasks/main.yaml),
which writes the network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Tandoor's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`tandoor.container` references `tandoor.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet. The Postgres attach in the next
section depends on it for the same reason.

`network_name` is `tandoor`, so the *file* is `tandoor.network` and the *Podman
network* is `tandoor_net` (`NetworkName=` in the network unit pins it, keeping
the Docker-era name).

## How it attaches to Postgres

Same idea at the other end: Postgres knows nothing about its clients, and this
playbook uses the shared include in
[postgres/roles/attach](../postgres/roles/attach/tasks/main.yaml) rather than
writing those drop-ins itself. That writes two files, both mergeable into
`postgres.container` by Quadlet:

| File | Content |
|---|---|
| `postgres.container.d/50-tandoor.conf` | `Network=tandoor.network` |
| `postgres.container.d/10-network-aliases.conf` | `NetworkAlias=db` |

`db` is the alias Tandoor's `.env` actually asks for; `postgres` also resolves,
from `ContainerName=`. The alias lives in one shared file with a fixed value
because `NetworkAlias=` applies to *every* network a container joins — every app
in this repo reaches Postgres as `db`, and a per-service alias file would make
Quadlet pass `--network-alias` once per service.

[roles/postgres](roles/postgres/tasks/main.yaml) wraps that include for two
reasons:

- it defines the `Restart Postgres` handler the include notifies; and
- it **flushes handlers** immediately afterwards. On a first attach both drop-ins
  are new, so Postgres has to be *recreated* to join `tandoor_net` — and that
  restart is a handler, which by default runs at the end of the play, after the
  `tandoor` role has already started the app. The app would come up before `db`
  resolves, fail its own `is-active` check, and fail the play on a machine where
  nothing is wrong. Flushing applies the drop-in first; on later runs nothing is
  pending and it is a no-op.

Note the cost stated plainly in [postgres/readme.md](../postgres/readme.md): a
new drop-in recreates Postgres, which is a few seconds of downtime for **every**
attached app, not just this one. Quadlet has no live equivalent of
`podman network connect`.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `recipes.tanshu.com` — the site block Caddy serves |
| `container_name` | `tandoor`, and the Quadlet unit / container name |
| `image` | `docker.io/vabene1111/recipes:latest` — fully qualified on purpose |
| `network_name` | `tandoor` → `tandoor.network` / `tandoor_net` |
| `container_port` | `80`, the port inside the container that Caddy proxies to |
| `host_directory` | `~/podman-data/tandoor` |
| `postgres_container` / `postgres_unit` | The shared database unit this app attaches to |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run — Tandoor publishes nothing host-side.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

The app's secrets (`SECRET_KEY`) and database coordinates live in
`roles/tandoor/templates/tandoor.env.j2`, rendered to
`~/podman-data/tandoor/.env` at mode `0600` and read by systemd via
`EnvironmentFile=`, not by Ansible. The password is the fleet's shared `123456`,
which matches `pg_password` in [postgres/vars/default.yml](../postgres/vars/default.yml)
— a migrated app connects unchanged. Both are committed in plaintext as before;
flagged as a follow-up, not changed here.

## Running

```command
ansible-playbook -u tanshu tandoor/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

This play applies **only** Tandoor's own units and data. Its `pre_tasks` run
Caddy's `register` include and nothing else. The rootless-Podman prerequisites —
installing podman, `loginctl enable-linger`, the low-port sysctl and the Quadlet
directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.
To apply them on their own:

```yaml
- name: Rootless Podman prerequisites
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
```

`systemctl --user enable tandoor` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

## Migrating the file data

The database already moved; this is the app's own files, and the playbook does
not move them because they are on the old host while the playbook runs on the
new one. Run this from the machine that holds the `monoco` and `emily` SSH
aliases, as in [postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md).

`mediafiles/` is the only irreplaceable part, and on this host it is **900K**;
the 98 MB is `staticfiles/`, which is `collectstatic` output. Copy both anyway —
the bind mount shadows whatever static files the image ships, so if the
entrypoint does not re-run `collectstatic`, an empty volume is a site with no
CSS, and copying stale files is the strictly safer failure mode.

```bash
# 1. Look before you copy: size, and whether it is readable without sudo.
ssh monoco 'du -sh /var/lib/tandoor/*; ls -ln /var/lib/tandoor'
ssh monoco 'find /var/lib/tandoor -type f ! -readable'   # empty = no sudo needed

# 2. Stop the old container so nothing writes while you copy. Leave it stopped:
#    its database is on emily now, and two live copies would diverge.
ssh monoco 'docker stop tandoor'

# 3. Create the destination (its parent exists from the Postgres move).
ssh emily 'mkdir -p ~/podman-data/tandoor'

# 4. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command. `.env` is excluded — Ansible templates it,
#    at mode 0600, before the unit ever starts.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
set -o pipefail
ssh monoco 'tar --exclude=./.env -C /var/lib/tandoor -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/tandoor -xf -'

#    If step 1 listed unreadable files, read them as root instead:
# ssh monoco 'sudo tar --exclude=./.env -C /var/lib/tandoor -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/tandoor -xf -'

# 5. Verify by content, and over the *whole* tree. Checking mediafiles alone
#    would pass on a copy that dropped almost everything: it is under 1 MB here
#    while staticfiles is the 98 MB.
ssh monoco 'cd /var/lib/tandoor && find . -type f ! -name .env -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
ssh emily  'cd ~/podman-data/tandoor && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/tandoor ! -user tanshu'    # empty = correct

# 7. Deploy.
cd prepare-server && ansible-playbook -u tanshu ../tandoor/playbook.yml
```

Ownership needs no `chown` pass: the app runs as uid 0 in-container, which
`keep-id` maps to `tanshu`, so files extracted as `tanshu` are already right.
That is also why `--no-same-owner` is explicit on the receiving side — the
source is root-owned, and restoring that ownership would be wrong here.

If the link is slow or drops, do it in two hops instead: resumable, and it shows
progress.

```bash
rsync -aP --exclude=.env monoco:/var/lib/tandoor/ ~/tandoor-copy/
rsync -aP ~/tandoor-copy/ emily:~/podman-data/tandoor/
```

Rolling back means rolling back both halves: `ssh monoco 'docker start tandoor'`
is only meaningful while monoco's own Postgres 17 cluster is still intact.

## Verification

```bash
systemctl --user status tandoor
journalctl --user -u tandoor -f
podman logs tandoor

# What the playbook checks for you:
podman exec tandoor getent hosts db          # the alias the .env asks for
podman exec tandoor getent hosts postgres

# The database really is the migrated one:
podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d tandoor -c '\dt' | head

# The site, end to end:
curl -sSI https://recipes.tanshu.com
```

Also exercise a **write** through the app (upload a recipe image) and confirm on
the host that the new file under `~/podman-data/tandoor/mediafiles` is owned by
`tanshu`. That is the check that catches a wrong `keep-id`; a read-only smoke
test will not.

Then reboot the host and confirm the unit returns unattended — without
`loginctl enable-linger tanshu` it will not, and the failure is silent.

## Notes and gotchas

- **A 502 whose page says `nginx` is Tandoor's own front end, not a stray host
  nginx.** The image runs nginx in front of gunicorn (80 → 8080, which is why
  `podman ps` shows both ports), and Caddy relays upstream response headers
  verbatim — so `server: nginx` reaches the browser from inside the container.
  During the startup window, while Django is still running migrations, that
  nginx answers 502 and Caddy passes it straight through. The role therefore
  waits for a real HTTP response from inside the container
  (`Wait for Tandoor to serve HTTP`) rather than trusting `is-active`; a 502
  after a *green* run means the app, not the proxy.
- **Client IPs.** Caddy's access log may show a gateway address rather than the
  real client, depending on the rootless port forwarder. See
  [caddy/readme.md](../caddy/readme.md), "Client IPs", before relying on it.
- **Attaching restarts Postgres for everyone.** See the note above; it is a few
  seconds of downtime for every attached app.
- **Registering restarts all of Caddy.** Same mechanism at the proxy end — a
  brief restart of every hosted site, not just this one.
- **If the Caddyfile does not exist yet**, this playbook creates it containing
  only the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a
  missing or zero-length file. Deploy Caddy first on a fresh host.
- **The backup scripts still assume Docker.** `~/bin/tandoor-backup.sh` uses
  `--link`, which Podman does not have; it needs the `podman exec` form from
  [postgres/readme.md](../postgres/readme.md), "Backups", before the old host is
  decommissioned.
- **DNS.** `recipes.tanshu.com` must resolve to `emily` before Caddy can obtain
  a certificate for it.
- **Do not restart the container on monoco.** It would write to monoco's old
  cluster, which is being kept intact purely as a rollback.

## Not done here

- Rotating the secrets that are committed in plaintext.
- Tandoor needs no cache: its env sets no `REDIS_URL`, so `valkey` is not
  attached to this network.