# The menu deployment Lives here, beside the app it deploys. One run deploys a Site: it starts the app's container as a **rootless Podman Quadlet unit** on `emily` and points the Site's web address at it through the shared **Caddy** unit that `ansible-playbooks/caddy` owns. There is no older system to keep serving, so there is no staged rollout and nothing to retire — deploying an earlier tag is the rollback. This is the same shape as the sibling services in `ansible-playbooks` (gitea, nextcloud, tandoor, bitwarden, valkey, postgres, trillium): rootless Podman, one Quadlet `.container` / `.network` per service in `~/.config/containers/systemd/`, state under `~/podman-data/`, and Caddy reaching each app by name on a Podman network. Nothing here talks to Docker. ## The pieces ``` ansible/ ├── hosts # one entry per Site, all on emily ├── playbook.yml # connects as tanshu, no become ├── vars/{default,hopsngrains,greatbear,hngmohali}.yml └── roles/ ├── register/ # writes menu-.network and the Caddy drop-in ├── menu-app/ # the Quadlet unit, start + health + verify └── caddy/ # appends the site block, reloads Caddy ``` Two things are worth internalising, because they are the whole design: **Caddy knows nothing about its upstreams.** A service registers _itself_ by writing two files into the Quadlet directory: its own `.network` unit and a drop-in beside `caddy.container` that pulls that network into Caddy. This play does that in `pre_tasks`, before the app unit starts, because Quadlet fails unit generation when a unit references a `.network` file that does not exist yet. **The container's uid is mapped onto `tanshu`.** `UserNS=keep-id:uid=1000,gid=1000` maps the image's own `node` account onto the user that owns the units, so every file the app writes is owned by `tanshu` on the host: plain `tar`/`rsync` backups, no subuids, no `podman unshare`. `scripts/provision-site-env.sh` uses the same mapping for the same reason. ## What lives where | Thing | Where | | -------------------------------------- | --------------------------------------------------------------------------- | | Name, tagline, icon, categories, menus | on the server under `~/podman-data/menu-/data` — the admin edits them | | Admin password hash, session secret | `~/podman-data/menu-/site.env` (untracked, mode 600) | | Theme colours | `config/.site.json` in the repo, baked into the image | | The unit that runs | `~/.config/containers/systemd/menu-.container` | | The network and Caddy drop-in | `menu-.network`, `caddy.container.d/50-menu-.conf` | | Which image tag runs | `-e tag=` | | Which Site is deployed | the inventory entry in `hosts` | | The web address it serves | `host:` in `vars/.yml` | `` is the inventory name (`hopsngrains`, `greatbear`, `hngmohali`). The container itself keeps the Docker-era name `menu-app.`, because `provision-site-env.sh` and muscle memory look it up that way; the Caddyfile proxies to the dotless `menu-` alias, because a container name with dots in it is a liability in DNS. ## What the host needs `emily` is already set up — it runs Caddy, Postgres, Tandoor and Valkey the same way. If you are standing up a fresh host, run `ansible-playbooks/caddy` first: it installs podman, enables lingering, lowers the privileged-port floor so rootless Caddy can bind 80/443, creates the Quadlet directory, and starts the Caddy unit. Those prerequisites deliberately are **not** duplicated here, so a host that has not run that playbook will fail at the first `systemctl --user` — see `ansible-playbooks/caddy/readme.md`. The image is in the private registry, so log in once as `tanshu` on `emily`, before the first deploy: ```sh podman login registry.tanshu.com ``` The play pulls the image itself, so a missing login fails there with that instruction rather than as a container that will not start. The unit keeps podman's default `Pull=missing` on purpose, so a restart or a reboot never depends on the registry being reachable. `ssh` on some machines complains about `/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is a local file-permission bug whose fix is `sudo chown -R root:root /etc/ssh/ssh_config.d`, not something this playbook works around. ## Deploying From the repo root: ```sh ./deploy.sh 2026-09-22 # checks, podman build+push, then deploys SKIP_DEPLOY=1 ./deploy.sh 2026-09-22 # build and push only ``` `deploy.sh` runs `make build-production TAG=`, which builds a multi-arch podman manifest (`linux/amd64,linux/arm64`) and pushes `latest` and the tag to `registry.tanshu.com/menu-admin`. Then it runs this playbook against every Site, or the one named by `SITE`. By hand: ```sh make build-production TAG=2026-09-22 cd ansible ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=2026-09-22 ``` Every Site runs on the same box, so `hosts` has one entry per Site pointing at the same `ansible_host` with its own `var_file`. The play carries no `become`: the Quadlet units belong to `ansible_user` (`tanshu`), and running as that user is what gives us a real PAM session, which is what makes `systemctl --user` work. Connecting as anyone else would write units into the wrong home. Adding a Site is one line in `hosts`, a copy of `vars/hopsngrains.yml`, and a `config/.site.json`. The image is shared, so the config has to be in it: a new Site needs a new tag. ## Migrating a Site's data from `monoco` The menus still serve from Docker containers on `monoco`. A Site moves across with its data, before it is ever pointed at from DNS. Run this from a machine that can ssh to both hosts — the `monoco` and `emily` aliases this repo's own tooling uses: ```sh scripts/copy-site-data.sh hopsngrains scripts/copy-site-data.sh greatbear scripts/copy-site-data.sh hngmohali ``` For one Site it stops the old container, streams `/var/lib/menu.` across, starts the old container again so `monoco` keeps serving until DNS moves, and then proves the copy: the SHA-256 of the whole tree is compared against the source, and every file is checked to belong to `tanshu`. `--live` skips the stop; `MONOCO=`, `EMILY=`, `SRC_ROOT=` and `DEST_ROOT=` override the defaults. Two things make the copy look different from a plain `rsync`: - **`sudo` on monoco.** `/var/lib/menu.` belongs to uid 1000 (the `ubuntu` account) and the ssh user there is 1001, so it is not readable without it. monoco's `tanshu` has passwordless sudo. - **The workstation relays it.** monoco cannot ssh to emily, so the tar streams through the machine running the script; nothing large is written locally. Extracting as `tanshu` on emily is what makes the files land owned by the user `keep-id` maps the app onto — as root they would arrive owned by uid 1000, which on emily is nobody. By hand, that is: ```sh SITE=hopsngrains HOST=menu.hopsngrains.com ssh monoco "docker stop menu-app.$HOST" ssh -n monoco "sudo -n tar -C /var/lib/$HOST -cf - ." \ | ssh emily "mkdir -p ~/podman-data/menu-$SITE && tar -C ~/podman-data/menu-$SITE -xf -" ssh monoco "docker start menu-app.$HOST" # Same checks the script runs. ssh -n monoco "sudo -n sh -c 'cd /var/lib/$HOST && find . -type f -exec sha256sum {} +' | sort | sha256sum" ssh -n emily "cd ~/podman-data/menu-$SITE && find . -type f -exec sha256sum {} + | sort | sha256sum" ssh -n emily "find ~/podman-data/menu-$SITE ! -user tanshu" ``` The old layout put everything under `/var/lib/menu./`, with the content one level down in `data/`; the new one is `~/podman-data/menu-/` with the same `data/` inside it and `site.env` beside it. Nothing inside changes. `site.env` carries over so the admin keeps working; without it the app starts with the admin disabled and the playbook says so rather than failing. Then deploy that Site and check it on a `Host:` override before touching DNS. `monoco` keeps serving until the cutover; only then stop its container — and leave it in place, not removed, so `docker start` is the rollback. ## The admin password The app starts with the admin disabled while `~/podman-data/menu-/site.env` is missing, and the playbook says so rather than failing. Create it once per Site; run it again with `--force` to reset the password (a fresh session secret signs everyone out). There is no minimum password length. ```sh scp scripts/provision-site-env.sh emily:/tmp/ ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains systemctl --user restart menu-hopsngrains.service ``` The `-t` matters: the script prompts for the password, and without a terminal on the remote side the prompt cannot be read, so it would sit there looking frozen. With no image argument it borrows the one the running container uses. It has to run as `tanshu`, the user whose rootless Podman owns the container. ## Checking it landed ```sh npm run check-site -- https://menu.hopsngrains.com ``` Read-only, exits non-zero on failure. Then from a phone: the landing page, a Section page, the page counter, tap-to-zoom, and signing into `/admin` to publish something. ## Rolling back Deploy the previous tag: ```sh ./deploy.sh ``` `podman logs menu-app.` and `journalctl --user -u menu-` are where a container that will not start says why. To go all the way back to the Docker deployment, start the container that was left on `monoco` and point Caddy at it again.