Run the Sites as rootless Podman Quadlets, and build with podman

The fleet on emily is rootless Podman with Quadlet units under ~/.config/containers/systemd and state in ~/podman-data, all behind one shared Caddy unit. This brings the menus in line instead of leaving them the last Docker deployment. The build moves with them: podman reads a Containerfile as readily as a Dockerfile, so the name now says which toolchain is real. The app is one .container and one .network file. Two details carry it. UserNS=keep-id maps the image's own node account onto the user that owns the units, so data/ and site.env stay owned by tanshu: plain tar backups, no subuids. provision-site-env.sh writes site.env with the same mapping rather than --user, which under rootless Podman would have produced a subuid-owned file the app could not read. And the Site registers its own network with Caddy by writing a drop-in beside caddy.container, so Caddy never lists its upstreams and never changes when a Site is added. The unit keeps the Docker-era container name for the scripts and muscle memory that look it up, while the Caddyfile proxies to a dotless NetworkAlias, because a container name with dots in it is a liability in DNS. The unit keeps podman's Pull=missing, so a restart or a reboot never depends on the registry being reachable, while the play pulls explicitly at deploy time. That is also where a missing `podman login` for the private registry now fails, with the instruction, rather than as a unit that will not start. copy-site-data.sh moves a Site's data across, and is not a plain rsync: /var/lib/menu.* on monoco belongs to uid 1000 while the ssh user there is 1001, so the read needs sudo; monoco cannot ssh to emily, so the workstation relays the tar; and the extraction runs as tanshu on emily, which is what makes the files land owned by the user keep-id maps the app onto. It proves the copy by comparing a SHA-256 of the whole tree against the source and checking every file's owner. Not yet done: the copies themselves and the DNS move, which need a human.
2026-10-16 09:54:52 +05:30
1 parent c412181159
commit e68f8ab00d
@@ -8,8 +8,8 @@ public/fonts/OFL.txt
 docs
 test
 compose.yaml
-Dockerfile
-.dockerignore
+Containerfile
+.containerignore
 README.md
 # design prototype, never committed to main and never shipped
 public/prototype
@@ -1,3 +1,6 @@
+# Built with podman. `podman build` reads a Containerfile as readily as a
+# Dockerfile, and the name says which toolchain the pipeline actually uses.
+#
 # The app runs TypeScript directly on Node 24, so there is no build step and no
 # compiler in the image. Node 24 on Alpine is musl, which sharp ships prebuilt
 # binaries for.
@@ -25,6 +28,10 @@ USER node

 EXPOSE 3000

+# podman builds an OCI image by default, which drops this instruction, so the
+# Quadlet unit declares the same check again as HealthCmd= (see
+# ansible/roles/menu-app/templates/menu-app.container.j2). Keeping it here means
+# the image still healthchecks itself when run directly.
 HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \
  CMD node -e "fetch('http://127.0.0.1:'+(process.env.PORT||3000)+'/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

@@ -10,8 +10,9 @@ Built: Site config, the Menu Set store, the image pipeline, the customer-facing
 renderer (landing page, Section pages, archived sets, media route) with the
 approved design, the admin (sign in, create a set, upload and reorder Pages,
 preview, publish, restore, delete), the `verify`/`repair` CLI, the container and
-the Ansible playbook. Not yet done: the parallel-run deployment on `monoco`,
-which needs a human present.
+the Ansible deployment as rootless Podman Quadlet units on `emily`, behind the
+shared Caddy unit. Not yet done: copying each Site's data across from `monoco`
+and moving DNS, which needs a human present.

 ## Deploying

@@ -23,8 +24,10 @@ SKIP_DEPLOY=1 ./deploy.sh 2026-09-22   # build and push only
 The image is built with **podman** as a multi-arch manifest and pushed to
 `registry.tanshu.com/menu-admin` (`Makefile`, mirrored from the other apps on
 this machine). The Ansible deployment lives in `ansible/` in this repo, with one
-inventory entry per Site; see `ansible/readme.md` for how a deploy runs and how to
-roll one back.
+inventory entry per Site. It writes a rootless Podman **Quadlet** unit per Site
+and appends the Site's block to the shared Caddyfile, exactly as the sibling
+services in `ansible-playbooks` do; see `ansible/readme.md` for how a deploy runs
+and how to roll one back.

 ## Layout

@@ -47,11 +50,14 @@ test/                     node:test suites
 docs/plan.md              The agreed plan
 docs/operating.md         Publishing, fixing and rolling back, for whoever operates it
 docs/adr/                 Decisions worth remembering
-deploy.sh                 Tests, build, push
+Containerfile             The image, built by podman
+compose.yaml              Local dev: podman compose
+deploy.sh                 Tests, build, push, deploy
+ansible/                  The deployment: one rootless Quadlet unit per Site
 ```

-The Ansible playbook lives in `ansible-playbooks/menu-admin/`, beside the other
-playbooks, because that is where it is run from.
+The deployment is in `ansible/` here, beside the app it deploys, and is run from
+that directory by `deploy.sh`.

 ## Running it locally

@@ -1,17 +1,24 @@
-# The Ansible inventory for the menus. Every Site runs on the same box, so each
-# entry names it and points at the vars file that describes it.
+# The Ansible inventory for the menus. Every Site runs on the same box — emily,
+# the rootless Podman host — so each entry names it and points at the vars file
+# that describes it.
 #
 #   cd ansible
 #   ansible-playbook --inventory=hosts playbook.yml              # every Site
 #   ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains
 #
-# ansible_host is written as the resolvable name rather than an ssh alias so the
-# inventory works on any machine, whatever its ~/.ssh/config says.
+# This play carries no become: it writes rootless Quadlet units, which belong to
+# whoever Ansible connects as, so it must connect as that user.
+#
+# ansible_host is written as the resolvable name rather than an ssh alias, so the
+# inventory works on any machine, whatever its ~/.ssh/config says. emily has no
+# DNS name yet, so that name is its address today; when DNS — or monoco's elastic
+# address — moves to it at cutover, this is the one line that changes.

 [menus]
-hopsngrains ansible_host=monoco.tanshu.com var_file=vars/hopsngrains.yml
-greatbear ansible_host=monoco.tanshu.com var_file=vars/greatbear.yml
-hngmohali ansible_host=monoco.tanshu.com var_file=vars/hngmohali.yml
+hopsngrains ansible_host=35.154.14.59 var_file=vars/hopsngrains.yml
+greatbear ansible_host=35.154.14.59 var_file=vars/greatbear.yml
+hngmohali ansible_host=35.154.14.59 var_file=vars/hngmohali.yml

 [all:vars]
+ansible_user=tanshu
 ansible_python_interpreter=/usr/bin/python3
@@ -1,15 +1,26 @@
 ---
-# Deploys the menu app for one or more Sites. It lives here, beside the app it
-# deploys, and is run from this directory:
+# Deploys the menu app for one or more Sites as rootless Podman Quadlet units on
+# emily, behind the shared Caddy unit that ansible-playbooks/caddy owns. It lives
+# here, beside the app it deploys, and is run from this directory:
 #
-#   make build-production TAG=<tag>        (from the repo root)
-#   ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains
+#   ./deploy.sh <tag>                                            (repo root)
+#   ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=<tag>
 #
-# One run deploys the Site: it starts the container and points the Site's web
-# address at it. Nothing older is kept serving, so there is no staged rollout —
-# deploying an earlier tag is the rollback.
- hosts: menus
-  become: true
+# Connects as the container user and runs everything as that user: the Quadlet
+# units belong to it, so the connection's PAM session is what makes
+# `systemctl --user` work without any environment juggling.
+#
+# gather_facts is required: podman_user and the home directory come from the
+# user_id / user_dir facts.
+#
+# The rootless-Podman prerequisites (installing podman, enabling lingering, the
+# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied
+# here. They are the shared task file at
+# ansible-playbooks/caddy/roles/podman/tasks/main.yaml, and a host that has
+# already run ansible-playbooks/caddy/playbook.yml has them.
+- name: Deploy the menu app as rootless Podman Quadlets
+  hosts: menus
+  gather_facts: true
  vars_files:
    - vars/default.yml

@@ -19,7 +30,16 @@
        file: '{{ var_file }}'
      when: var_file is defined

+    # This must happen before the app unit is started, not after: the generated
+    # unit references {{ network_name }}.network, and Quadlet fails *unit
+    # generation* — so `systemctl --user start` reports "unit not found" — when a
+    # referenced .network file does not exist yet. The same include writes the
+    # Caddy-side drop-in that attaches Caddy to this network.
+    - name: Register this Site's network with Caddy
+      ansible.builtin.include_tasks: '{{ playbook_dir }}/roles/register/tasks/main.yaml'
+
  roles:
-    - network
+    # The app: the data directory tree, the Quadlet unit, start + verify.
    - menu-app
+    # Appends this Site's block to the shared Caddyfile and reloads Caddy.
    - caddy
@@ -1,26 +1,90 @@
 # The menu deployment

 Lives here, beside the app it deploys. One run deploys a Site: it starts the
-app's container and points the Site's web address at it. There is no older
-system to keep serving, so there is no staged rollout and nothing to retire —
-deploying an earlier tag is the rollback.
+app's container as a **rootless Podman Quadlet unit** on `emily` and points the
+Site's web address at it through the shared **Caddy** unit that
+`ansible-playbooks/caddy` owns. There is no older system to keep serving, so
+there is no staged rollout and nothing to retire — deploying an earlier tag is
+the rollback.

-**The app's data root is the directory the old static menus were served from**
-(`/var/lib/menu.<site>`), on purpose, so the app inherits the location instead of
-a second one. The app only reads and writes `data/` and `site.env` inside it. If
-you later want the old menu files gone, delete those files — not the directory,
-which now holds the live content.
+This is the same shape as the sibling services in `ansible-playbooks` (gitea,
+nextcloud, tandoor, bitwarden, valkey, postgres, trillium): rootless Podman, one
+Quadlet `.container` / `.network` per service in `~/.config/containers/systemd/`,
+state under `~/podman-data/`, and Caddy reaching each app by name on a Podman
+network. Nothing here talks to Docker.
+
+## The pieces
+
+```
+ansible/
+├── hosts                                  # one entry per Site, all on emily
+├── playbook.yml                           # connects as tanshu, no become
+├── vars/{default,hopsngrains,greatbear,hngmohali}.yml
+└── roles/
+    ├── register/   # writes menu-<site>.network and the Caddy drop-in
+    ├── menu-app/   # the Quadlet unit, start + health + verify
+    └── caddy/      # appends the site block, reloads Caddy
+```
+
+Two things are worth internalising, because they are the whole design:
+
+**Caddy knows nothing about its upstreams.** A service registers _itself_ by
+writing two files into the Quadlet directory: its own `<name>.network` unit and a
+drop-in beside `caddy.container` that pulls that network into Caddy. This play
+does that in `pre_tasks`, before the app unit starts, because Quadlet fails unit
+generation when a unit references a `.network` file that does not exist yet.
+
+**The container's uid is mapped onto `tanshu`.** `UserNS=keep-id:uid=1000,gid=1000`
+maps the image's own `node` account onto the user that owns the units, so every
+file the app writes is owned by `tanshu` on the host: plain `tar`/`rsync`
+backups, no subuids, no `podman unshare`. `scripts/provision-site-env.sh` uses
+the same mapping for the same reason.

 ## What lives where

-| Thing                                  | Where                                                                  |
-| -------------------------------------- | ---------------------------------------------------------------------- |
-| Name, tagline, icon, categories, menus | on the server under `/var/lib/menu.<site>/data` — the admin edits them |
-| Theme colours                          | `config/<site>.site.json` in the repo, baked into the image            |
-| Admin password hash, session secret    | `/var/lib/menu.<site>/site.env` (untracked, mode 600)                  |
-| Which image tag runs                   | `-e tag=<tag>`                                                         |
-| Which Site is deployed                 | the inventory entry in `hosts`                                         |
-| The web address it serves              | `host:` in `vars/<site>.yml`                                           |
+| Thing                                  | Where                                                                       |
+| -------------------------------------- | --------------------------------------------------------------------------- |
+| Name, tagline, icon, categories, menus | on the server under `~/podman-data/menu-<site>/data` — the admin edits them |
+| Admin password hash, session secret    | `~/podman-data/menu-<site>/site.env` (untracked, mode 600)                  |
+| Theme colours                          | `config/<site>.site.json` in the repo, baked into the image                 |
+| The unit that runs                     | `~/.config/containers/systemd/menu-<site>.container`                        |
+| The network and Caddy drop-in          | `menu-<site>.network`, `caddy.container.d/50-menu-<site>.conf`              |
+| Which image tag runs                   | `-e tag=<tag>`                                                              |
+| Which Site is deployed                 | the inventory entry in `hosts`                                              |
+| The web address it serves              | `host:` in `vars/<site>.yml`                                                |
+
+`<site>` is the inventory name (`hopsngrains`, `greatbear`, `hngmohali`). The
+container itself keeps the Docker-era name `menu-app.<domain>`, because
+`provision-site-env.sh` and muscle memory look it up that way; the Caddyfile
+proxies to the dotless `menu-<site>` alias, because a container name with dots in
+it is a liability in DNS.
+
+## What the host needs
+
+`emily` is already set up — it runs Caddy, Postgres, Tandoor and Valkey the same
+way. If you are standing up a fresh host, run `ansible-playbooks/caddy` first: it
+installs podman, enables lingering, lowers the privileged-port floor so rootless
+Caddy can bind 80/443, creates the Quadlet directory, and starts the Caddy unit.
+Those prerequisites deliberately are **not** duplicated here, so a host that has
+not run that playbook will fail at the first `systemctl --user` — see
+`ansible-playbooks/caddy/readme.md`.
+
+The image is in the private registry, so log in once as `tanshu` on `emily`,
+before the first deploy:
+
+```sh
+podman login registry.tanshu.com
+```
+
+The play pulls the image itself, so a missing login fails there with that
+instruction rather than as a container that will not start. The unit keeps
+podman's default `Pull=missing` on purpose, so a restart or a reboot never
+depends on the registry being reachable.
+
+`ssh` on some machines complains about
+`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is
+a local file-permission bug whose fix is `sudo chown -R root:root
+/etc/ssh/ssh_config.d`, not something this playbook works around.

 ## Deploying

@@ -33,8 +97,8 @@ SKIP_DEPLOY=1 ./deploy.sh 2026-09-22   # build and push only

 `deploy.sh` runs `make build-production TAG=<tag>`, which builds a multi-arch
 podman manifest (`linux/amd64,linux/arm64`) and pushes `latest` and the tag to
-`registry.tanshu.com/menu-admin`. Then it runs this playbook against the Site
-named by `SITE` (default `hopsngrains`).
+`registry.tanshu.com/menu-admin`. Then it runs this playbook against every Site,
+or the one named by `SITE`.

 By hand:

@@ -45,16 +109,89 @@ ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=2026-
 ```

 Every Site runs on the same box, so `hosts` has one entry per Site pointing at
-the same `ansible_host` with its own `var_file`. `ansible_host` is written as the
-resolvable name rather than an ssh alias, so the inventory works whatever a
-machine's `~/.ssh/config` says. (`ssh` on some machines complains about
-`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is
-a local file-permission bug whose fix is `sudo chown -R root:root
-/etc/ssh/ssh_config.d`, not something this playbook works around.)
+the same `ansible_host` with its own `var_file`. The play carries no `become`: the
+Quadlet units belong to `ansible_user` (`tanshu`), and running as that user is
+what gives us a real PAM session, which is what makes `systemctl --user` work.
+Connecting as anyone else would write units into the wrong home.

 Adding a Site is one line in `hosts`, a copy of `vars/hopsngrains.yml`, and a
-`config/<site>.site.json`. The image is shared, so the config has to be in it:
-a new Site needs a new tag.
+`config/<site>.site.json`. The image is shared, so the config has to be in it: a
+new Site needs a new tag.
+
+## Migrating a Site's data from `monoco`
+
+The menus still serve from Docker containers on `monoco`. A Site moves across
+with its data, before it is ever pointed at from DNS. Run this from a machine
+that can ssh to both hosts — the `monoco` and `emily` aliases this repo's own
+tooling uses:
+
+```sh
+scripts/copy-site-data.sh hopsngrains
+scripts/copy-site-data.sh greatbear
+scripts/copy-site-data.sh hngmohali
+```
+
+For one Site it stops the old container, streams `/var/lib/menu.<domain>` across,
+starts the old container again so `monoco` keeps serving until DNS moves, and
+then proves the copy: the SHA-256 of the whole tree is compared against the
+source, and every file is checked to belong to `tanshu`. `--live` skips the stop;
+`MONOCO=`, `EMILY=`, `SRC_ROOT=` and `DEST_ROOT=` override the defaults.
+
+Two things make the copy look different from a plain `rsync`:
+
+- **`sudo` on monoco.** `/var/lib/menu.<domain>` belongs to uid 1000 (the
+  `ubuntu` account) and the ssh user there is 1001, so it is not readable without
+  it. monoco's `tanshu` has passwordless sudo.
+- **The workstation relays it.** monoco cannot ssh to emily, so the tar streams
+  through the machine running the script; nothing large is written locally.
+  Extracting as `tanshu` on emily is what makes the files land owned by the user
+  `keep-id` maps the app onto — as root they would arrive owned by uid 1000,
+  which on emily is nobody.
+
+By hand, that is:
+
+```sh
+SITE=hopsngrains
+HOST=menu.hopsngrains.com
+
+ssh monoco "docker stop menu-app.$HOST"
+ssh -n monoco "sudo -n tar -C /var/lib/$HOST -cf - ." \
+  | ssh emily "mkdir -p ~/podman-data/menu-$SITE && tar -C ~/podman-data/menu-$SITE -xf -"
+ssh monoco "docker start menu-app.$HOST"
+
+# Same checks the script runs.
+ssh -n monoco "sudo -n sh -c 'cd /var/lib/$HOST && find . -type f -exec sha256sum {} +' | sort | sha256sum"
+ssh -n emily  "cd ~/podman-data/menu-$SITE && find . -type f -exec sha256sum {} + | sort | sha256sum"
+ssh -n emily  "find ~/podman-data/menu-$SITE ! -user tanshu"
+```
+
+The old layout put everything under `/var/lib/menu.<domain>/`, with the content
+one level down in `data/`; the new one is `~/podman-data/menu-<site>/` with the
+same `data/` inside it and `site.env` beside it. Nothing inside changes.
+`site.env` carries over so the admin keeps working; without it the app starts
+with the admin disabled and the playbook says so rather than failing.
+
+Then deploy that Site and check it on a `Host:` override before touching DNS.
+`monoco` keeps serving until the cutover; only then stop its container — and
+leave it in place, not removed, so `docker start` is the rollback.
+
+## The admin password
+
+The app starts with the admin disabled while `~/podman-data/menu-<site>/site.env`
+is missing, and the playbook says so rather than failing. Create it once per Site;
+run it again with `--force` to reset the password (a fresh session secret signs
+everyone out). There is no minimum password length.
+
+```sh
+scp scripts/provision-site-env.sh emily:/tmp/
+ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains
+systemctl --user restart menu-hopsngrains.service
+```
+
+The `-t` matters: the script prompts for the password, and without a terminal on
+the remote side the prompt cannot be read, so it would sit there looking frozen.
+With no image argument it borrows the one the running container uses. It has to
+run as `tanshu`, the user whose rootless Podman owns the container.

 ## Checking it landed

@@ -63,25 +200,8 @@ npm run check-site -- https://menu.hopsngrains.com
 ```

 Read-only, exits non-zero on failure. Then from a phone: the landing page, a
-Section page, the page counter, tap-to-zoom, and signing into `/admin` to
-publish something.
-
-## The admin password
-
-The app starts with the admin disabled while `/var/lib/menu.<site>/site.env`
-is missing, and the playbook says so rather than failing. Create it once per
-Site; run it again with `--force` to reset the password (a fresh session secret
-signs everyone out). There is no minimum password length.
-
-```sh
-scp scripts/provision-site-env.sh monoco:/tmp/
-ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com
-ssh monoco docker restart menu-app.hopsngrains.com
-```
-
-The `-t` matters: the script prompts for the password, and without a terminal on
-the remote side the prompt cannot be read, so it would sit there looking frozen.
-With no image argument it borrows the one the running container uses.
+Section page, the page counter, tap-to-zoom, and signing into `/admin` to publish
+something.

 ## Rolling back

@@ -91,4 +211,7 @@ Deploy the previous tag:
 ./deploy.sh <previous-tag>
 ```

-`docker logs menu-app.<site>` is where a failed container says why.
+`podman logs menu-app.<site>` and `journalctl --user -u menu-<site>` are where a
+container that will not start says why. To go all the way back to the Docker
+deployment, start the container that was left on `monoco` and point Caddy at it
+again.
@@ -1,6 +1,31 @@
 ---
- name: Reload Caddy
-  # Caddy validates before it applies, so a mistake here leaves the running
-  # config untouched and fails the play loudly.
-  command: 'docker exec -w /etc/caddy {{ caddy_container }} caddy reload'
-  listen: Reload Caddy
+# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
+# Quadlet unit there. These handlers run as the connection user, so
+# `systemctl --user` and `podman exec` work without any XDG_RUNTIME_DIR juggling.
+
+# Notified by this role when the Caddyfile changes. A reload keeps existing
+# connections; a restart would briefly drop every hosted site.
+#
+# Note the config path: /etc/caddy/Caddyfile is the mount inside the container,
+# not {{ caddyfile_path }}, which is the host side of it.
+- name: Reload Caddy configuration
+  ansible.builtin.command: >-
+    podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
+  changed_when: true
+  listen: 'Reload Caddy'
+
+# Notified by roles/register when this Site's network unit, or the drop-in that
+# attaches Caddy to it, is written. Quadlet has no live equivalent of
+# `docker network connect`, so the container has to be recreated — a brief
+# restart of every hosted site, not just this one. See the Caddy playbook's
+# readme, "Trade-off worth knowing".
+#
+# Defined after the reload above so that when both are notified the cheap config
+# reload happens first.
+- name: Restart Caddy
+  ansible.builtin.systemd_service:
+    name: '{{ caddy_name }}.service'
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Caddy
@@ -1,14 +1,29 @@
 ---
+# Append this Site's block to the shared Caddyfile.
+#
+# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
+# Quadlet unit there, so this role does not manage the container — it only points
+# it at the app. The Caddyfile is a bind mount into that container, so Caddy sees
+# a written change as soon as the file lands; the handler only asks Caddy to
+# re-read it. That is a reload, not a restart, so the other hosted sites are
+# untouched.
+#
+# blockinfile is idempotent on its own via the marker, so there is no need to
+# slurp the file and compare first — which also matters because slurp *fails*
+# when the Caddyfile does not exist yet, on a host where the Caddy playbook has
+# not run.
+
 - name: Check the Caddyfile exists
-  stat:
+  ansible.builtin.stat:
    path: '{{ caddyfile_path }}'
  register: menu_caddyfile

 - name: Fail when Caddy is not where we expect it
-  fail:
+  ansible.builtin.fail:
    msg: >-
-      {{ caddyfile_path }} is missing. Deploy the caddy playbook, which owns the
-      Caddy container and its config, before this one.
+      {{ caddyfile_path }} is missing. Deploy the Caddy playbook
+      (ansible-playbooks/caddy/playbook.yml), which owns the Caddy unit and its
+      config, before this one.
  when: not menu_caddyfile.stat.exists

 # The Site's web address may still be claimed by an older snippet: the retired
@@ -16,7 +31,7 @@
 # Caddy refuse the whole config, and a refused reload leaves the old proxy in
 # place, which is a 502 once the old container is gone.
 - name: Drop any older snippet claiming this web address
-  blockinfile:
+  ansible.builtin.blockinfile:
    path: '{{ caddyfile_path }}'
    marker: '# {mark} Ansible managed Caddy snippet for {{ host }}'
    state: absent
@@ -25,9 +40,12 @@
 # blockinfile replaces whatever sits between its markers, so redeploying updates
 # the snippet in place rather than appending a second one.
 - name: Point the Site's web address at the app
-  blockinfile:
+  ansible.builtin.blockinfile:
    path: '{{ caddyfile_path }}'
    marker: '# {mark} ANSIBLE MANAGED: menu app ({{ site }})'
    block: '{{ caddy_snippet }}'
-    create: no
+    create: false
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
  notify: Reload Caddy
@@ -0,0 +1,19 @@
+---
+# All handlers run as the connection user ({{ podman_user }}), so systemctl
+# --user works without setting XDG_RUNTIME_DIR.
+
+- name: Reload systemd user units
+  ansible.builtin.command: systemctl --user daemon-reload
+  changed_when: true
+  listen: Reload systemd user units
+
+# A changed unit file changes the generated unit, so the app is recreated. All of
+# its state is on the bind mount, so a recreate loses nothing — but it is a few
+# seconds of downtime for the one Site.
+- name: Restart the menu app
+  ansible.builtin.systemd_service:
+    name: '{{ unit_name }}'
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart the menu app
@@ -1,134 +1,178 @@
 ---
- name: Pull the app image
-  docker_image:
-    name: '{{ image_name }}'
-    tag: '{{ image_tag }}'
-    source: pull
-    force_source: yes
+# The menu app as a rootless Podman Quadlet unit.
+#
+# Everything here runs as {{ podman_user }} — the connection user. No become, no
+# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
+# user manager is reachable through the standard environment.

+# The pull is explicit, rather than left to the Quadlet unit, for two reasons.
+# The image lives in a private registry, so a failure here is where a missing
+# `podman login` should show up — not two tasks later as "unit not found". And
+# the unit keeps podman's default `Pull=missing`, so it never reaches for the
+# registry on a restart or after a reboot: that is what lets a box come back up
+# when the registry is down. Pulling here is what keeps a redeployed tag current.
+- name: Pull the app image
+  ansible.builtin.command: 'podman pull {{ image_name }}:{{ image_tag }}'
+  register: menu_pull
+  changed_when: false
+  failed_when: false
+
+- name: Fail when the image cannot be pulled
+  ansible.builtin.fail:
+    msg: >-
+      podman could not pull {{ image_name }}:{{ image_tag }}.
+      {{ menu_pull.stderr | default('') | trim }}
+      The registry is private: run `podman login registry.tanshu.com` once as
+      {{ podman_user }} on this host.
+  when: menu_pull.rc != 0
+
+# The data directory is created here rather than left to the image. Under
+# keep-id the image's own root phase maps to a subuid, so anything it tries to
+# create under the mount fails with EPERM; the directory has to exist already,
+# owned by {{ podman_user }}.
 - name: Ensure the app root exists
-  file:
+  ansible.builtin.file:
    path: '{{ app_root }}'
    state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
    mode: '0750'

+# MENU_ROOT is the app root, and the app keeps its content one level down in
+# data/ (originals/, drafts/, published/, site.json, sections.json).
 - name: Ensure the data directory exists
-  file:
+  ansible.builtin.file:
    path: '{{ app_root }}/data'
    state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
    mode: '0750'

 - name: Look for the untracked site.env
-  stat:
+  ansible.builtin.stat:
    path: '{{ app_root }}/site.env'
  register: menu_site_env

 - name: Warn when the admin has no password yet
-  debug:
+  ansible.builtin.debug:
    msg: >-
      {{ app_root }}/site.env is missing, so the admin is disabled: the public
-      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on
-      this host, then re-run this playbook.
+      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on this
+      host, then re-run this playbook.
  when: not menu_site_env.stat.exists

- name: Run the menu app
-  docker_container:
-    name: '{{ docker_container }}'
-    image: '{{ image_name }}:{{ image_tag }}'
-    state: started
-    restart_policy: unless-stopped
-    user: '{{ app_uid }}:{{ app_gid }}'
-    # Force a rebuild of the container with -e recreate_container=true: that is
-    # what applies a changed healthcheck or env to an existing container, since
-    # the module otherwise leaves a running one alone.
-    recreate: '{{ recreate_container | default(false) | bool }}'
-    # Declared here rather than relied on from the image: podman builds an OCI
-    # image by default and drops Docker's HEALTHCHECK instruction, so the
-    # Dockerfile's own healthcheck never reaches the container. node is in the
-    # image and the exec form needs no shell quoting.
-    healthcheck:
-      test:
-        - CMD
-        - node
-        - -e
-        - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-      interval: 30s
-      timeout: 5s
-      start_period: 30s
-      retries: 3
-    env:
-      MENU_ROOT: /data
-      MENU_SITE_CONFIG: '/app/config/{{ site }}.site.json'
-      SECURE_COOKIES: 'true'
-      LOG_LEVEL: info
-      PORT: '3000'
-    volumes:
-      - '{{ app_root }}:/data'
-    networks:
-      - name: '{{ docker_network }}'
+- name: Install the menu app Quadlet unit
+  ansible.builtin.template:
+    src: menu-app.container.j2
+    dest: '{{ quadlet_dir }}/{{ network_name }}.container'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+  notify:
+    - Reload systemd user units
+    - Restart the menu app

- name: Keep site.env readable only by the app user
-  file:
-    path: '{{ app_root }}/site.env'
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
-    mode: '0600'
-  when: menu_site_env.stat.exists
+- name: Start the menu app
+  block:
+    - name: Start the menu app
+      ansible.builtin.systemd_service:
+        name: '{{ unit_name }}'
+        scope: user
+        state: started
+        daemon_reload: true
+  rescue:
+    # systemd reports "control process exited with error code" and nothing else,
+    # which is useless for the usual failures (a missing network unit, an
+    # unwritable /data, a bad uid mapping). Pull the real reason out.
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_diagnostics
+      changed_when: false
+      failed_when: false

-# Deliberately not the image's HEALTHCHECK: podman builds an OCI image by
-# default and drops the Docker HEALTHCHECK instruction, so State.Health is absent
-# and a wait on it fails on a perfectly healthy container — which stops the play
-# before Caddy is ever pointed at the app.
- name: Wait for the app to answer inside its own container
-  ansible.builtin.command:
-    argv:
-      - docker
-      - exec
-      - '{{ docker_container }}'
-      - node
-      - -e
-      - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-  register: menu_health
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: 'The menu app did not start — see the diagnostics above.'
+
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+- name: Verify the menu app is running
+  ansible.builtin.command: 'systemctl --user is-active {{ unit_name }}'
+  register: menu_active
  changed_when: false
-  retries: 20
-  delay: 3
+  failed_when: menu_active.stdout != 'active'
+
+# `is-active` says the container is running; it does not say the app can serve.
+# Restart=always means even a crash-looping container is reported active — which
+# is exactly what a fatal integrity problem in the data directory produces. The
+# unit declares the health check, so run that once, synchronously.
+- name: Wait for the app to pass its health check
+  ansible.builtin.command: 'podman healthcheck run {{ container_name }}'
+  register: menu_health
  until: menu_health.rc == 0
+  retries: 24
+  delay: 5
+  changed_when: false
  failed_when: false

- name: Fail when the app never answers
-  fail:
-    msg: >-
-      {{ docker_container }} did not answer /healthz within a minute. Its logs
-      say why: docker logs {{ docker_container }}
+- name: Report a container that never answered
  when: menu_health.rc != 0
+  block:
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_health_diagnostics
+      changed_when: false
+      failed_when: false
+
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_health_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: >-
+          {{ container_name }} did not pass its health check within two minutes.
+          Its logs are above, and: podman logs {{ container_name }}
+
+- name: Keep site.env readable only by the app user
+  ansible.builtin.file:
+    path: '{{ app_root }}/site.env'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0600'
+  when: menu_site_env.stat.exists

 - name: Check the data directory for integrity problems
  # Reads the same directory the container writes, as the same user, so a
  # permissions mistake shows up here rather than at 8pm service.
-  command:
+  ansible.builtin.command:
    argv:
-      - docker
-      - run
-      - --rm
-      - --user
-      - '{{ app_uid }}:{{ app_gid }}'
-      - -e
+      - podman
+      - exec
+      - --env
      - MENU_ROOT=/data
-      - -e
+      - --env
      - 'MENU_SITE_CONFIG=/app/config/{{ site }}.site.json'
-      - --entrypoint
+      - '{{ container_name }}'
      - node
-      - '{{ image_name }}:{{ image_tag }}'
      - src/cli/verify.ts
  register: menu_verify
  changed_when: false
  failed_when: false

 - name: Report integrity problems
-  debug:
+  ansible.builtin.debug:
    msg: '{{ menu_verify.stdout_lines | default([]) }}'
  when: menu_verify.rc != 0
@@ -0,0 +1,60 @@
+# {{ container_name }} — rootless Podman Quadlet unit.
+# Managed by Ansible (menu-admin/roles/menu-app). Do not edit by hand.
+#
+# Joins {{ network_name }}.network, written by roles/register — the same include
+# that attaches Caddy to it. Caddy reaches the app by container name, so nothing
+# is published on the host and the app is not reachable from outside the Podman
+# network at all.
+
+[Unit]
+Description=Menu app for {{ host }}
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName={{ container_name }}
+Image={{ image_name }}:{{ image_tag }}
+
+# keep-id maps the image's own account — `node`, uid/gid {{ app_uid }} — onto
+# {{ podman_user }}, which is what keeps every file the app writes owned by
+# {{ podman_user }} on the host: plain tar/rsync backups, no subuids, no
+# `podman unshare`. The same mapping is why scripts/provision-site-env.sh writes
+# site.env with --userns=keep-id.
+UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
+
+# The alias is deliberately dotless. ContainerName keeps the Docker-era name so
+# operators and scripts find it, but a name with dots in it is a liability in
+# DNS, so {{ network_name }} is what the Caddyfile proxies to. Both resolve from
+# inside the network.
+NetworkAlias={{ network_name }}
+
+Environment=MENU_ROOT=/data
+Environment=MENU_SITE_CONFIG=/app/config/{{ site }}.site.json
+Environment=SECURE_COOKIES=true
+Environment=LOG_LEVEL=info
+Environment=PORT={{ container_port }}
+
+# Everything the app writes: data/ (originals, drafts, published, site.json,
+# sections.json) and site.env. The Site's own config is baked into the image and
+# read from /app.
+Volume={{ app_root }}:/data
+
+Network={{ network_name }}.network
+
+# podman builds an OCI image by default and drops the Containerfile's
+# HEALTHCHECK instruction, so the same check is declared here and the play gates
+# on it with `podman healthcheck run`. node is in the image.
+HealthCmd={{ health_cmd }}
+HealthInterval=30s
+HealthTimeout=5s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+# The app validates its data directory before it serves and refuses to boot on a
+# fatal integrity problem, so a first start over copied data can take a moment.
+TimeoutStartSec=120
+
+[Install]
+WantedBy=default.target
@@ -1,8 +0,0 @@
---
- name: 'Ensure {{ docker_network }} exists and Caddy is on it'
-  docker_network:
-    name: '{{ docker_network }}'
-    state: present
-    connected:
-      - '{{ caddy_container }}'
-    appends: yes
@@ -0,0 +1,51 @@
+---
+# Register this Site's logical network with Caddy.
+#
+# Copied from ansible-playbooks/caddy/roles/register/tasks/main.yaml: this repo
+# carries its own deployment, so it cannot include that file by path. Keep the
+# two in step.
+#
+# Caddy is the *base* unit and knows nothing about its upstreams. A service adds
+# itself by writing two files into the Quadlet directory:
+#
+#   1. its own <name>.network unit, and
+#   2. a drop-in next to caddy.container that pulls that network into Caddy.
+#
+# Caddy itself is owned by ansible-playbooks/caddy/playbook.yml; this play only
+# registers with it. The including play defines a "Restart Caddy" handler.
+
+- name: 'Create the {{ network_name }} network unit'
+  ansible.builtin.copy:
+    dest: '{{ quadlet_dir }}/{{ network_name }}.network'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+    content: |
+      # {{ network_name }} — logical app network.
+      # Managed by Ansible. Do not edit by hand.
+      #
+      # NetworkName pins the Podman network name. Without it Quadlet would
+      # create "systemd-{{ network_name }}".
+      #
+      # isolate blocks traffic between separate Podman networks, so a container
+      # on this network cannot reach one that is only on another. Caddy and any
+      # shared service still reach it, because they are members of it.
+      [Network]
+      NetworkName={{ network_name }}_net
+      Options=isolate=true
+  notify: Restart Caddy
+
+- name: 'Register {{ network_name }} with Caddy'
+  ansible.builtin.copy:
+    dest: '{{ quadlet_dir }}/{{ caddy_name }}.container.d/50-{{ network_name }}.conf'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+    content: |
+      # Added by the {{ network_name }} playbook. Do not edit by hand.
+      #
+      # Quadlet merges this into caddy.container. Deleting this file removes
+      # the network from Caddy on the next daemon-reload.
+      [Container]
+      Network={{ network_name }}.network
+  notify: Restart Caddy
@@ -2,19 +2,58 @@
 # Defaults shared by every Site. Each Site's own file is named by the `var_file`
 # in ansible/hosts, and its slug is its inventory name.

+# ---------------------------------------------------------------------------
+# Rootless Podman / Quadlet
+# ---------------------------------------------------------------------------
+# podman_user is derived from the connection: this play carries no become, so the
+# Quadlet units belong to whoever Ansible connects as. Requires gather_facts
+# (set in playbook.yml).
+#
+# These three duplicate caddy/vars/default.yml in the ansible-playbooks repo,
+# which owns the Caddy unit these Sites sit behind. Keep them in sync.
+podman_user: "{{ ansible_facts['user_id'] }}"
+podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
+quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
+
 # Which Site config inside the image to run, and which web address it serves.
 site: '{{ inventory_hostname }}'

+# ---------------------------------------------------------------------------
+# The app
+# ---------------------------------------------------------------------------
 image_name: registry.tanshu.com/menu-admin
 image_tag: "{{ tag | default('latest') }}"

+# The unit file is menu-<site>.container and the Podman network is
+# menu-<site>_net. The container keeps the Docker-era name, dots and all, because
+# scripts/provision-site-env.sh and anything an operator types look it up that
+# way — but a name with dots is a DNS liability, so the Caddyfile proxies to the
+# dotless NetworkAlias the unit sets beside it.
+network_name: 'menu-{{ site }}'
+unit_name: '{{ network_name }}.service'
+container_name: 'menu-app.{{ host }}'
+
+# The image's own account (`node`), which keep-id maps onto podman_user.
 app_uid: 1000
 app_gid: 1000

-caddy_container: caddy
-caddyfile_path: /var/lib/caddy/conf/Caddyfile
+container_port: 3000
+
+# podman builds an OCI image by default and drops the Containerfile's
+# HEALTHCHECK instruction, so the unit declares the same check and the play gates
+# on it. node is in the image and the sh -c podman wraps it in handles the quotes.
+health_cmd: >-
+  node -e "fetch('http://127.0.0.1:{{ container_port }}/healthz').then(r =>
+  process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
+
+# ---------------------------------------------------------------------------
+# The shared Caddy unit and its Caddyfile, both owned by the caddy playbook in
+# ansible-playbooks (caddy/playbook.yml).
+# ---------------------------------------------------------------------------
+caddy_name: caddy
+caddyfile_path: '{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile'

 caddy_snippet: |
  {{ host }} {
-      reverse_proxy {{ docker_container }}:3000
+      reverse_proxy {{ network_name }}:{{ container_port }}
  }
@@ -1,13 +1,11 @@
 ---
-# The Great Bear. Its name, tagline, icon, categories and menus live on the server under
-# app_root and are edited in the admin; config/greatbear.site.json only seeds the
-# theme and the first category list.
+# The Great Bear. Its name, tagline, icon, categories and menus live on the
+# server under app_root and are edited in the admin; config/greatbear.site.json
+# only seeds the theme and the first category list.
 title: greatbear.in
 host: 'menu.{{ title }}'

-docker_container: 'menu-app.{{ title }}'
-docker_network: '{{ docker_container }}_net'
-
-# Everything the app writes — data/ and the untracked site.env. Deliberately the
-# directory the old static menus were served from.
-app_root: '/var/lib/{{ host }}'
+# Everything the app writes — data/ (the content) and site.env (the admin
+# password hash and session secret). One directory per Site, each of which is
+# also its own Podman network and its own Caddy upstream.
+app_root: '{{ podman_data_dir }}/{{ network_name }}'
@@ -1,13 +1,11 @@
 ---
-# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the server under
-# app_root and are edited in the admin; config/hngmohali.site.json only seeds the
-# theme and the first category list.
+# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the
+# server under app_root and are edited in the admin; config/hngmohali.site.json
+# only seeds the theme and the first category list.
 title: hngmohali.com
 host: 'menu.{{ title }}'

-docker_container: 'menu-app.{{ title }}'
-docker_network: '{{ docker_container }}_net'
-
-# Everything the app writes — data/ and the untracked site.env. Deliberately the
-# directory the old static menus were served from.
-app_root: '/var/lib/{{ host }}'
+# Everything the app writes — data/ (the content) and site.env (the admin
+# password hash and session secret). One directory per Site, each of which is
+# also its own Podman network and its own Caddy upstream.
+app_root: '{{ podman_data_dir }}/{{ network_name }}'
@@ -1,7 +1,7 @@
 ---
-# Hops n Grains. Its name, tagline, icon, categories and menus live on the
-# server under app_root and are edited in the admin; config/hopsngrains.site.json
-# only seeds the theme. This file says where the Site runs and what it answers.
+# Hops n Grains. Its name, tagline, icon, categories and menus live on the server
+# under app_root and are edited in the admin; config/hopsngrains.site.json only
+# seeds the theme. This file says where the Site runs and what it answers.
 #
 # Every value that interpolates a template is quoted: YAML reads a leading `{`
 # as a flow mapping, which is the parse error Ansible reports as "missing quotes
@@ -9,13 +9,7 @@
 title: hopsngrains.com
 host: 'menu.{{ title }}'

-docker_container: 'menu-app.{{ title }}'
-docker_network: '{{ docker_container }}_net'
-
-# Everything the app writes — data/ and the untracked site.env.
-#
-# Deliberately the directory the old static menus were served from, so the app
-# inherits the location rather than a second one. The app only ever reads and
-# writes its own data/ inside it; the old menu files can be emptied out when
-# they are no longer wanted, and the directory itself must stay.
-app_root: '/var/lib/{{ host }}'
+# Everything the app writes — data/ (the content) and site.env (the admin
+# password hash and session secret). One directory per Site, each of which is
+# also its own Podman network and its own Caddy upstream.
+app_root: '{{ podman_data_dir }}/{{ network_name }}'
@@ -1,7 +1,10 @@
-# Local development. Production is the menu-admin playbook in ansible-playbooks;
-# this file exists so the container can be tried out without touching a server.
+# Local development. Production is the playbook in ansible/; this file exists so
+# the container can be tried out without touching a server.
 #
-#   npm run seed && docker compose up --build
+#   npm run seed && podman compose up --build
+#
+# `podman compose` needs a provider on PATH — podman-compose or docker-compose.
+# The image is built from the Containerfile beside this file.
 services:
  menu:
    build: .
@@ -1,7 +1,9 @@
 #!/usr/bin/env bash
 #
 # Builds the image with podman, pushes it to the private registry, then deploys
-# the Site with the playbook that lives in ansible/.
+# the Site with the playbook that lives in ansible/. The playbook runs rootless
+# Podman Quadlet units on emily, behind the shared Caddy unit from
+# ansible-playbooks/caddy.
 #
 #   ./deploy.sh 2026-09-22                  deploy the default Site
 #   SITE=greatbear ./deploy.sh 2026-09-22   deploy another Site
@@ -24,6 +26,11 @@ if ! command -v podman >/dev/null 2>&1; then
  exit 1
 fi

+if [[ -z "${SKIP_DEPLOY:-}" ]] && ! command -v ansible-playbook >/dev/null 2>&1; then
+  echo "ansible-playbook is required to deploy; SKIP_DEPLOY=1 builds and pushes only" >&2
+  exit 1
+fi
+
 if [[ -z "${SKIP_CHECKS:-}" ]]; then
  echo "==> lint, format, types and tests"
  npm run lint
@@ -0,0 +1,31 @@
+# Sites run as rootless Podman Quadlets behind a shared Caddy
+
+Each Site is deployed as a **rootless Podman** container managed by a systemd
+user unit (Quadlet) on `emily`, reached through the one shared Caddy unit rather
+than a long-lived rootful Docker container. The image is built by `podman` too,
+from a `Containerfile`. This matches how every other service on that host is
+deployed, and it takes the deployment off Docker entirely.
+
+The app owns its whole serving path, so it needed the smallest deployment of the
+fleet: one `.container` file, one `.network` file, and a block appended to the
+Caddyfile. Two details carry the design. `UserNS=keep-id:uid=1000,gid=1000` maps
+the image's own `node` account onto the user that owns the units, which is what
+keeps `data/` and `site.env` owned by a real host user — plain `tar` backups, no
+subuids — and `scripts/provision-site-env.sh` uses the same mapping so both sides
+agree on who owns the file. And the Site registers its own network with Caddy by
+writing a drop-in beside `caddy.container`, so Caddy never lists its upstreams
+and never has to be edited when a Site is added or removed.
+
+**Considered Options**: staying on rootful Docker on `monoco`, rejected because
+the fleet is moving and leaving one app behind means two runtimes and two Caddy
+configs to reason about; running rootful Podman, rejected because it keeps the
+same root-owned data and loses the plain-backup property; and naming the Quadlet
+unit after the container (`menu-app.<domain>`), rejected because the dots make
+the unit name awkward and the name fragile in DNS — the unit is `menu-<site>`
+while the container keeps its old name for the scripts that look it up.
+
+**Consequences**: the app is not published on a host port at all — only Caddy,
+on the shared Podman network, can reach it — and deployment, logs and restarts
+move from `docker` to `systemctl --user` / `podman`. The data root moves to
+`~/podman-data/menu-<site>`, so each Site's content has to be copied across from
+`/var/lib/menu.<domain>` once, before DNS points at it.
@@ -2,19 +2,26 @@

 For whoever is on the other end of "the menu is wrong". The design is in
 `plan.md`, the vocabulary in `CONTEXT.md`, and the deployment in
-`ansible-playbooks/menu-admin/readme.md`.
+`../ansible/readme.md`.
+
+Each Site is one rootless Podman **Quadlet** unit on `emily` — a systemd *user*
+service named `menu-<site>` — behind the shared Caddy unit. There is no Docker in
+this path.

 ## Where things are

-| Thing | Where |
-| --- | --- |
-| The app | container `menu-app.<site>` on `monoco`, image `registry.tanshu.com/menu-admin:<tag>` |
-| Menu content | `/var/lib/menu.<site>/data` — `originals/`, `drafts/`, `published/` |
-| Admin | `https://<vhost>/admin` |
-| The password | `/var/lib/menu.<site>/site.env` (mode 600, never in git) |
-| Categories | `data/sections.json`, edited in the admin |
+| Thing           | Where                                                                                     |
+| --------------- | ----------------------------------------------------------------------------------------- |
+| The app         | user unit `menu-<site>.service` on `emily`, container `menu-app.<site>`, image `registry.tanshu.com/menu-admin:<tag>` |
+| Menu content    | `~/podman-data/menu-<site>/data` — `originals/`, `drafts/`, `published/`                   |
+| Admin           | `https://<vhost>/admin`                                                                    |
+| The password    | `~/podman-data/menu-<site>/site.env` (mode 600, never in git)                              |
+| Categories      | `data/sections.json`, edited in the admin                                                  |
 | Brand and theme | `config/<site>.site.json` in this repo, baked into the image (the palette follows the Stitch system in `docs/design/stitch/`) |

+Everything runs as the user `tanshu`, so every command below is run on `emily` as
+that user. `sudo` is never needed, and never works for these units.
+
 ## Everyday tasks

 **Publish a new menu.** Sign in at `/admin`, find the Section, fill in "New Menu
@@ -30,8 +37,8 @@ open the set you want and press "Make current". Its URL does not change.
 which signs out anyone who was signed in. Nothing else changes.

 ```sh
-ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com --force
-ssh monoco docker restart menu-app.hopsngrains.com
+ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains --force
+ssh emily systemctl --user restart menu-hopsngrains
 ```

 There is no minimum length and no password policy — the one rule is that the
@@ -45,8 +52,8 @@ HASH=$(printf '%s' "$PW" | node -e "let s='';process.stdin.setEncoding('utf8');p
 SECRET=$(node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))")
 printf 'ADMIN_PASSWORD_HASH=%s
 SESSION_SECRET=%s
-' "$HASH" "$SECRET" | ssh monoco "docker run --rm -i --user 1000:1000 -v /var/lib/menu.hopsngrains.com:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'"
-ssh monoco docker restart menu-app.hopsngrains.com
+' "$HASH" "$SECRET" | ssh emily "podman run --rm -i --userns=keep-id:uid=1000,gid=1000 -v ~/podman-data/menu-hopsngrains:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'"
+ssh emily systemctl --user restart menu-hopsngrains
 ```

 **Add, rename, reorder or remove a category.** In the admin, open the
@@ -69,14 +76,15 @@ rebuild and push (`./deploy.sh <tag>`) and the playbook with the new tag.
 Original survives, the app can rebuild them:

 ```sh
-docker run --rm --user 1000:1000 -e MENU_ROOT=/data \
+podman run --rm --userns=keep-id:uid=1000,gid=1000 -e MENU_ROOT=/data \
  -e MENU_SITE_CONFIG=/app/config/hopsngrains.site.json \
-  -v /var/lib/menu.hopsngrains.com:/data \
+  -v ~/podman-data/menu-hopsngrains:/data \
  --entrypoint node registry.tanshu.com/menu-admin:<tag> src/cli/repair.ts
 ```

 `src/cli/verify.ts` does the same inventory without changing anything, and is
-what the server refuses to boot on.
+what the server refuses to boot on. Both need the `--userns=keep-id` mapping, or
+they would see the files as a different user and fail to write.

 ## When something looks wrong

@@ -84,15 +92,21 @@ what the server refuses to boot on.
 | --- | --- |
 | Landing page says the menus are being updated | no Section has a Current Menu Set — publish one in the admin |
 | A Section 404s | that Section has no Current set, or the URL is misspelled |
-| Sign-in rejects the right password | `docker logs menu-app.<site>`, then check `site.env` exists and is owned by the app user |
-| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `docker restart` |
+| Sign-in rejects the right password | `podman logs menu-app.<site>`, then check `site.env` exists and is owned by `tanshu` |
+| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `systemctl --user restart menu-<site>` |
 | A photo uploads but looks wrong | the browser shrinks to 3000px before upload; that is expected, and the Original is kept |
-| The container will not start | `docker logs` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust |
+| The container will not start | `systemctl --user status menu-<site>` and `podman logs menu-app.<site>` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust |
+| The unit is "not found" | its `.network` file is missing, so Quadlet refused to generate it: check `~/.config/containers/systemd/` |
 | Caddy reload failed during a deploy | nothing is down: Caddy validates before applying. Read the error in the playbook output |
 | Everything is slow | check the tag is current; a Viewing image is ~1600px, so a Section page should not be megabytes |

-**Logs.** `docker logs -f menu-app.<site>` — JSON lines, one per event, including
-every upload, publish, restore and failed sign-in.
+**Logs.** `podman logs -f menu-app.<site>`, or `journalctl --user -u menu-<site>
+-f` for the unit. The app logs JSON lines, one per event, including every upload,
+publish, restore and failed sign-in.
+
+**Restarting.** `systemctl --user restart menu-<site>`. A *reload* of the Caddy
+config is `podman exec caddy caddy reload --config /etc/caddy/Caddyfile`, which
+does not drop the other sites.

 **Checking from outside.** No sign-in, no writes, exits non-zero on failure, so
 it can gate a cutover:
@@ -104,133 +118,19 @@ npm run check-site -- https://menu.hopsngrains.com
 ## Backups

 The app is the only writer. The irreplaceable part is
-`/var/lib/menu.<site>/data/originals` — the photos exactly as uploaded, from
+`~/podman-data/menu-<site>/data/originals` — the photos exactly as uploaded, from
 which everything else can be regenerated with `repair`. Back that directory up;
 `site.env` is worth keeping too (otherwise you re-provision the password).

-## Rolling back
-
-Deploy the previous tag — `./deploy.sh <previous-tag>` — which rebuilds the Site at
-that version. `docker logs menu-app.<site>` is where a container that will not start
-says why.
-
-
-For whoever is on the other end of "the menu is wrong". The design is in
-`plan.md`, the vocabulary in `CONTEXT.md`, and the deployment in
-`ansible-playbooks/menu-admin/readme.md`.
-
-## Where things are
-
-| Thing | Where |
-| --- | --- |
-| The app | container `menu-app.<site>` on `monoco`, image `registry.tanshu.com/menu-admin:<tag>` |
-| Menu content | `/var/lib/menu.<site>/data` — `originals/`, `drafts/`, `published/` |
-| Admin | `https://<vhost>/admin` |
-| The password | `/var/lib/menu.<site>/site.env` (mode 600, never in git) |
-| Categories | `data/sections.json`, edited in the admin |
-| Brand and theme | `config/<site>.site.json` in this repo, baked into the image (the palette follows the Stitch system in `docs/design/stitch/`) |
-
-## Everyday tasks
-
-**Publish a new menu.** Sign in at `/admin`, find the Section, fill in "New Menu
-Set" (label, optional valid-until date), Create Draft, then drop the photos in.
-Drag to reorder — the order in the editor is the order customers see. Press
-Preview to check it, then Publish. Publishing archives whatever was current.
-
-**Put an older menu back.** Open the Section's "older menus" fold in the admin,
-open the set you want and press "Make current". Its URL does not change.
-
-**Change or reset the admin password.** Run the provision script again with
-`--force`: it overwrites the password hash, and generates a fresh session secret,
-which signs out anyone who was signed in. Nothing else changes.
-
-```sh
-ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com --force
-ssh monoco docker restart menu-app.hopsngrains.com
-```
-
-There is no minimum length and no password policy — the one rule is that the
-password cannot be empty. It can also be set from a machine that has this repo
-checked out, with the prompt on that machine rather than the server:
-
-```sh
-cd ~/programming/menu-admin
-read -r -s -p 'Admin password: ' PW; echo
-HASH=$(printf '%s' "$PW" | node -e "let s='';process.stdin.setEncoding('utf8');process.stdin.on('data',c=>s+=c).on('end',async()=>{const m=await import('./src/auth.ts');process.stdout.write(await m.hashPassword(s))})")
-SECRET=$(node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))")
-printf 'ADMIN_PASSWORD_HASH=%s
-SESSION_SECRET=%s
-' "$HASH" "$SECRET" | ssh monoco "docker run --rm -i --user 1000:1000 -v /var/lib/menu.hopsngrains.com:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'"
-ssh monoco docker restart menu-app.hopsngrains.com
-```
-
-**Add, rename, reorder or remove a category.** In the admin, open the
-**Categories** fold at the bottom of the dashboard. A new category starts empty,
-so customers do not see it until you publish a menu into it. Reordering there is
-the order customers see. Two rules: a category's web address is fixed when it is
-created (rename the name as often as you like — a printed QR code keeps working),
-and a category can only be removed once it holds no menus.
-
-**Change the restaurant's name or tagline, the icon, or the published dates.** In
-the admin, open the **Site identity** fold: the name and tagline customers see,
-a checkbox for whether a menu says when it was published, and an SVG upload for
-the icon (under 256 KB; the upload replaces the bundled mark and "Remove the
-uploaded icon" brings it back). None of this changes your web addresses.
-
-**Change the colours.** Those are still `config/<site>.site.json`, so they need a
-rebuild and push (`./deploy.sh <tag>`) and the playbook with the new tag.
-
-**Fix a missing image.** If a Page's thumbnail or viewing image is lost while its
-Original survives, the app can rebuild them:
-
-```sh
-docker run --rm --user 1000:1000 -e MENU_ROOT=/data \
-  -e MENU_SITE_CONFIG=/app/config/hopsngrains.site.json \
-  -v /var/lib/menu.hopsngrains.com:/data \
-  --entrypoint node registry.tanshu.com/menu-admin:<tag> src/cli/repair.ts
-```
-
-`src/cli/verify.ts` does the same inventory without changing anything, and is
-what the server refuses to boot on.
-
-## When something looks wrong
-
-| Symptom | Look here |
-| --- | --- |
-| Landing page says the menus are being updated | no Section has a Current Menu Set — publish one in the admin |
-| A Section 404s | that Section has no Current set, or the URL is misspelled |
-| Sign-in rejects the right password | `docker logs menu-app.<site>`, then check `site.env` exists and is owned by the app user |
-| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `docker restart` |
-| A photo uploads but looks wrong | the browser shrinks to 3000px before upload; that is expected, and the Original is kept |
-| The container will not start | `docker logs` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust |
-| Caddy reload failed during a deploy | nothing is down: Caddy validates before applying. Read the error in the playbook output |
-| Everything is slow | check the tag is current; a Viewing image is ~1600px, so a Section page should not be megabytes |
-
-**Logs.** `docker logs -f menu-app.<site>` — JSON lines, one per event, including
-every upload, publish, restore and failed sign-in.
-
-**Checking from outside.** No sign-in, no writes, exits non-zero on failure, so
-it can gate a cutover:
-
-```sh
-npm run check-site -- https://menu.hopsngrains.com
-```
-
-## Backups
-
-The app is the only writer. The irreplaceable part is
-`/var/lib/menu.<site>/data/originals` — the photos exactly as uploaded, from
-which everything else can be regenerated with `repair`. Back that directory up;
-`site.env` is worth keeping too (otherwise you re-provision the password).
+Because of `keep-id`, everything there is owned by `tanshu`, so a plain
+`tar`/`rsync` of `~/podman-data/menu-<site>` is complete — no subuid-owned files
+to trip over and no `podman unshare` needed.

 ## Rolling back

-The old nginx Site is kept, not deleted. To go back:
+Deploy the previous tag — `./deploy.sh <previous-tag>` — which rebuilds the Site
+at that version. `podman logs menu-app.<site>` is where a container that will not
+start says why.

-```sh
-docker start nginx-menu.hopsngrains.com
-```
-
-then point Caddy back at it — the cutover section of the playbook readme has the
-exact invocation, including the flag that stops the app's snippet from being
-re-added.
+The Docker-era container was left on `monoco`, stopped and not removed, so the
+last resort is to start it again and point Caddy back at it.
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+#
+# Copies one Site's data from the Docker deployment on monoco to the rootless
+# Podman deployment on emily.
+#
+#   scripts/copy-site-data.sh hopsngrains
+#   scripts/copy-site-data.sh hopsngrains --live   # copy while the app keeps serving
+#
+# Run it from a machine that can ssh to both hosts (the ssh aliases monoco and
+# emily, overridable with MONOCO= and EMILY=). This is the one-off part of the
+# move: the playbook deploys the container, this hands it the content.
+#
+# By default it stops the old container first, so the copy is of quiesced data,
+# and starts it again afterwards — monoco keeps serving until DNS moves. Use
+# --live to skip the stop; the app writes its manifests atomically and its images
+# are immutable once written, so a live copy is usually clean, but an upload
+# landing mid-copy is missed. Nothing on monoco is removed either way.
+#
+# Two details are load-bearing:
+#
+#   * monoco cannot ssh to emily, so this machine is the relay — the tar streams
+#     through it and nothing large is written locally.
+#   * sudo on monoco, because /var/lib/menu.<domain> belongs to uid 1000 (the
+#     ubuntu account) while the ssh user there is 1001. The extraction runs as
+#     tanshu on emily, which is the uid keep-id maps the app onto, so the files
+#     land owned by the right user rather than by root or a subuid.
+set -euo pipefail
+
+usage() {
+  cat <<'USAGE'
+usage: copy-site-data.sh <site> [--live]
+
+  <site>    inventory name from ansible/hosts: hopsngrains, greatbear, hngmohali
+  --live    copy without stopping the old container (not quiesced)
+
+Environment overrides:
+  MONOCO=host     ssh target holding the Docker deployment (default: monoco)
+  EMILY=host      ssh target holding the rootless Podman deployment (default: emily)
+  SRC_ROOT=dir    absolute data root on monoco (default: /var/lib)
+  DEST_ROOT=dir   absolute data root on emily (default: <emily home>/podman-data,
+                  resolved over ssh — ~ and $HOME are not expanded for you)
+USAGE
+}
+
+SITE="${1:-}"
+if [[ -z "$SITE" || "$SITE" == "-h" || "$SITE" == "--help" ]]; then
+  usage
+  [[ -n "$SITE" ]] || exit 1
+  exit 0
+fi
+shift
+
+LIVE=0
+for argument in "$@"; do
+  case "$argument" in
+    --live) LIVE=1 ;;
+    *) echo "unknown argument: $argument" >&2; usage >&2; exit 1 ;;
+  esac
+done
+
+cd "$(dirname "${BASH_SOURCE[0]}")/.."
+
+# The repo is the source of truth for which Sites exist and what they answer.
+VARS="ansible/vars/$SITE.yml"
+if ! grep -qE "^$SITE[[:space:]]" ansible/hosts || [[ ! -f "$VARS" ]]; then
+  echo "'$SITE' is not a Site in ansible/hosts" >&2
+  exit 1
+fi
+TITLE="$(sed -n 's/^title:[[:space:]]*//p' "$VARS" | head -n 1)"
+HOST="menu.$TITLE"
+CONTAINER="menu-app.$HOST"
+
+MONOCO="${MONOCO:-monoco}"
+EMILY="${EMILY:-emily}"
+SRC_ROOT="${SRC_ROOT:-/var/lib}"
+
+for tool in ssh tar; do
+  command -v "$tool" >/dev/null 2>&1 || { echo "$tool is required" >&2; exit 1; }
+done
+
+# Resolved rather than left as a literal ~ or $HOME, so every path below is
+# absolute and can be quoted safely on the remote side.
+if [[ -z "${DEST_ROOT:-}" ]]; then
+  DEST_ROOT="$(ssh -n "$EMILY" 'printf %s "$HOME"')/podman-data"
+fi
+
+SRC="$SRC_ROOT/$HOST"
+DEST="$DEST_ROOT/menu-$SITE"
+
+echo "==> $SITE: $MONOCO:$SRC -> $EMILY:$DEST"
+
+# Start the old container again however this exits, so a failed copy never
+# leaves the live Site down.
+restart_needed=0
+restart_old() {
+  if [[ "$restart_needed" -eq 1 ]]; then
+    echo "==> starting $CONTAINER again on $MONOCO"
+    ssh -n "$MONOCO" docker start "$CONTAINER" >/dev/null || true
+    restart_needed=0
+  fi
+}
+trap restart_old EXIT
+
+# -n on every monoco call: ssh would otherwise eat this script's stdin, and the
+# copy below is a pipeline whose input matters.
+if [[ "$LIVE" -eq 0 ]]; then
+  echo "==> stopping $CONTAINER on $MONOCO"
+  ssh -n "$MONOCO" docker stop "$CONTAINER" >/dev/null
+  restart_needed=1
+fi
+
+echo "==> copying"
+ssh -n "$MONOCO" "sudo -n tar -C '$SRC' -cf - ." \
+  | ssh "$EMILY" "mkdir -p '$DEST' && tar -C '$DEST' -xf -"
+
+restart_old
+
+echo "==> verifying"
+src_sum="$(ssh -n "$MONOCO" "sudo -n sh -c 'cd $SRC && find . -type f -exec sha256sum {} +' | sort | sha256sum" | awk '{print $1}')"
+dst_sum="$(ssh -n "$EMILY" "cd '$DEST' && find . -type f -exec sha256sum {} + | sort | sha256sum" | awk '{print $1}')"
+
+if [[ -z "$src_sum" || "$src_sum" != "$dst_sum" ]]; then
+  echo "the copy does not match the source: $src_sum != $dst_sum" >&2
+  exit 1
+fi
+
+user="$(id -un)"
+strays="$(ssh -n "$EMILY" "find '$DEST' ! -user '$user'" | head -n 5)"
+if [[ -n "$strays" ]]; then
+  echo "the copy holds files $user does not own, so keep-id will not see them:" >&2
+  echo "$strays" >&2
+  exit 1
+fi
+
+echo
+echo "ok  ${dst_sum:0:12}  $(ssh -n "$EMILY" "du -sb '$DEST'" | awk '{print $1}') bytes, all owned by $user"
+echo
+echo "next:"
+echo "  cd ansible && ansible-playbook --inventory=hosts playbook.yml --limit $SITE -e tag=<tag>"
+echo "  then check the Site on a Host: override before moving DNS to $EMILY"
@@ -1,9 +1,10 @@
 #!/usr/bin/env bash
 #
 # Writes the one file that never goes in git: the admin password hash and the
-# session secret. Run this on the server, once per Site.
+# session secret. Run this on the server, once per Site, as the user that owns
+# the rootless Podman units — tanshu:
 #
-#   scripts/provision-site-env.sh /var/lib/menu-app.hopsngrains.com
+#   scripts/provision-site-env.sh ~/podman-data/menu-hopsngrains
 #
 # The hash is produced by the app image itself, so the server needs no Node and
 # the password is never visible in the process list.
@@ -14,8 +15,8 @@ usage() {
 usage: provision-site-env.sh <data-root> [image] [--force]

  <data-root>  directory the app mounts as MENU_ROOT, e.g.
-               /var/lib/menu-app.hopsngrains.com
-  [image]      app image to borrow Node from (default menu-admin:latest)
+               ~/podman-data/menu-hopsngrains
+  [image]      app image to borrow Node from (default: the running app's)
  --force      overwrite an existing site.env
 USAGE
 }
@@ -43,19 +44,17 @@ if [[ -f "$ENV_FILE" && "$FORCE" -ne 1 ]]; then
  exit 1
 fi

-if ! command -v docker >/dev/null 2>&1; then
-  echo "docker is required" >&2
+if ! command -v podman >/dev/null 2>&1; then
+  echo "podman is required" >&2
  exit 1
 fi

-# The image is only needed to borrow node for hashing, so take the one the app
-# is already running rather than guessing a name. The playbook names the
-# container menu-app.<domain> for the web address menu.<domain>.
+# The image is only needed to borrow node for hashing, so take the one an app
+# container is already running rather than guessing a tag. The playbook names
+# every Site's container menu-app.<domain>. All Sites share the image, so which
+# one answers does not matter.
 if [[ -z "$IMAGE" ]]; then
-  for candidate in "menu-app.${DATA_ROOT##*/menu.}" "menu-app.${DATA_ROOT##*/}"; do
-    IMAGE="$(docker inspect --format '{{.Config.Image}}' "$candidate" 2>/dev/null || true)"
-    [[ -n "$IMAGE" ]] && break
-  done
+  IMAGE="$(podman ps --filter 'name=^menu-app\.' --format '{{.Image}}' | head -n 1 || true)"
 fi

 if [[ -z "$IMAGE" ]]; then
@@ -64,7 +63,7 @@ if [[ -z "$IMAGE" ]]; then
  exit 1
 fi

-if ! docker image inspect "$IMAGE" >/dev/null 2>&1; then
+if ! podman image inspect "$IMAGE" >/dev/null 2>&1; then
  echo "image $IMAGE is not present on this host." >&2
  echo "Pass the one the app runs: $0 $DATA_ROOT registry.tanshu.com/menu-admin:<tag>" >&2
  exit 1
@@ -96,11 +95,11 @@ if [[ "$PASSWORD" != "$REPEAT" ]]; then
  exit 1
 fi

-HASH="$(printf '%s' "$PASSWORD" | docker run --rm -i --entrypoint node "$IMAGE" \
+HASH="$(printf '%s' "$PASSWORD" | podman run --rm -i --entrypoint node "$IMAGE" \
  --input-type=module \
  -e "let s='';process.stdin.setEncoding('utf8');process.stdin.on('data',c=>s+=c);process.stdin.on('end',async()=>{const {pathToFileURL}=await import('node:url');const m=await import(pathToFileURL('/app/src/auth.ts').href);process.stdout.write(await m.hashPassword(s))})")"

-SECRET="$(docker run --rm --entrypoint node "$IMAGE" --input-type=module \
+SECRET="$(podman run --rm --entrypoint node "$IMAGE" --input-type=module \
  -e "import {randomBytes} from 'node:crypto';process.stdout.write(randomBytes(32).toString('base64url'))")"

 if [[ -z "$HASH" || -z "$SECRET" ]]; then
@@ -109,10 +108,14 @@ if [[ -z "$HASH" || -z "$SECRET" ]]; then
 fi

 # The file is written by a container running as the app's own uid, not by the
-# SSH user: the data root belongs to the app user (0750), so the SSH user
-# usually cannot write into it, and a file they did write would be unreadable by
-# the app. This way the ownership is right the first time and there is no
-# "re-run the playbook afterwards" step.
+# SSH user, so its ownership is right the first time and there is no "re-run the
+# playbook afterwards" step.
+#
+# Rootless Podman maps container uid 1000 onto a subuid by default, which would
+# produce a file the host user cannot read and a tar backup would drop. keep-id
+# maps it onto this script's user instead — the same mapping the app's Quadlet
+# unit uses (UserNS=keep-id in roles/menu-app), which is what keeps both sides
+# agreeing on who owns site.env.
 if [[ ! -d "$DATA_ROOT" ]]; then
  echo "$DATA_ROOT does not exist; run the menu-admin playbook first" >&2
  exit 1
@@ -120,18 +123,20 @@ fi

 APP_UID="${APP_UID:-1000}"
 APP_GID="${APP_GID:-1000}"
+USERNS="keep-id:uid=$APP_UID,gid=$APP_GID"

-if ! printf 'ADMIN_PASSWORD_HASH=%s\nSESSION_SECRET=%s\n' "$HASH" "$SECRET" | docker run --rm -i \
-  --user "$APP_UID:$APP_GID" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \
+if ! printf 'ADMIN_PASSWORD_HASH=%s\nSESSION_SECRET=%s\n' "$HASH" "$SECRET" | podman run --rm -i \
+  --userns="$USERNS" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \
  -c 'umask 077 && cat > /data/site.env'; then
-  echo "could not write $ENV_FILE as $APP_UID:$APP_GID" >&2
+  echo "could not write $ENV_FILE as the app's uid" >&2
  exit 1
 fi

 unset PASSWORD REPEAT HASH SECRET

-docker run --rm --user "$APP_UID:$APP_GID" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \
+podman run --rm --userns="$USERNS" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \
  -c 'ls -l /data/site.env'

-echo "wrote $ENV_FILE, owned by $APP_UID:$APP_GID"
-echo "restart the app container to pick it up; changing the password later means running this again with --force"
+echo "wrote $ENV_FILE, owned by $(id -un)"
+echo "restart the app to pick it up: systemctl --user restart menu-<site>"
+echo "changing the password later means running this again with --force"