menu/ansible

The menu deployment

Lives here, beside the app it deploys. One run deploys a Site: it starts the app's container as a rootless Podman Quadlet unit on emily and points the Site's web address at it through the shared Caddy unit that ansible-playbooks/caddy owns. There is no older system to keep serving, so there is no staged rollout and nothing to retire — deploying the image built from an earlier commit is the rollback.

This is the same shape as the sibling services in ansible-playbooks (gitea, nextcloud, tandoor, bitwarden, valkey, postgres, trillium): rootless Podman, one Quadlet .container / .network per service in ~/.config/containers/systemd/, state under ~/podman-data/, and Caddy reaching each app by name on a Podman network. Nothing here talks to Docker.

The pieces

ansible/
├── hosts                                  # one entry per Site, all on emily
├── playbook.yml                           # connects as tanshu, no become
├── vars/{default,hopsngrains,greatbear,hngmohali}.yml
└── roles/
    ├── register/   # writes menu-<site>.network and the Caddy drop-in
    ├── menu-app/   # the Quadlet unit, start + health + verify
    └── caddy/      # appends the site block, reloads Caddy

Two things are worth internalising, because they are the whole design:

Caddy knows nothing about its upstreams. A service registers itself by writing two files into the Quadlet directory: its own <name>.network unit and a drop-in beside caddy.container that pulls that network into Caddy. This play does that in pre_tasks, before the app unit starts, because Quadlet fails unit generation when a unit references a .network file that does not exist yet.

The container's uid is mapped onto tanshu. UserNS=keep-id:uid=1000,gid=1000 maps the image's own node account onto the user that owns the units, so every file the app writes is owned by tanshu on the host: plain tar/rsync backups, no subuids, no podman unshare. scripts/provision-site-env.sh uses the same mapping for the same reason.

A Quadlet HealthCmd must not contain quotes. Quadlet's parser strips a trailing double quote, so a check written as node -e "fetch(…)" reaches podman as an unterminated shell string: every run fails with /bin/sh: syntax error: unterminated quoted string, and the container serves while being permanently unhealthy. That is why the check is wget -q -O /dev/null … in vars/default.yml — quote-free, and the same reason the sibling apps use a bare curl. Nothing about it is visible in podman ps beyond the word unhealthy; podman inspect <container> --format '{{json .State.Health.Log}}' shows it.

What lives where

Thing	Where
Name, tagline, icon, categories, menus	on the server under ~/podman-data/menu-<site>/data — the admin edits them
Admin password hash, session secret	~/podman-data/menu-<site>/site.env (untracked, mode 600)
Theme colours	config/<site>.site.json in the repo, baked into the image
The unit that runs	~/.config/containers/systemd/menu-<site>.container
The network and Caddy drop-in	menu-<site>.network, caddy.container.d/50-menu-<site>.conf
Which image tag runs	-e image_tag=<commit>
Which Site is deployed	the inventory entry in hosts
The web address it serves	host: in vars/<site>.yml

<site> is the inventory name (hopsngrains, greatbear, hngmohali), and it is the single slug for everything: the container, its Podman network and its data directory are all menu-<site>. That is how the sibling hops and greatbear apps are deployed, and it keeps podman logs menu-<site> and systemctl --user status menu-<site> the same two commands across all three.

What the host needs

emily is already set up — it runs Caddy, Postgres, Tandoor and Valkey the same way. If you are standing up a fresh host, run ansible-playbooks/caddy first: it installs podman, enables lingering, lowers the privileged-port floor so rootless Caddy can bind 80/443, creates the Quadlet directory, and starts the Caddy unit. Those prerequisites deliberately are not duplicated here, so a host that has not run that playbook will fail at the first systemctl --user — see ansible-playbooks/caddy/readme.md.

The registry is private, but nothing here needs a manual login: the play logs in from registry_username / registry_password in vars/default.yml, pulls the image, and logs out again, so the credential is never left in the target user's auth file. The unit keeps podman's default Pull=missing on purpose, so a restart or a reboot never depends on the registry being reachable.

ssh on some machines complains about /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf being owned by nobody; that is a local file-permission bug whose fix is sudo chown -R root:root /etc/ssh/ssh_config.d, not something this playbook works around.

Deploying

From the repo root:

./deploy.sh                  # push HEAD, build <commit>, deploy every Site
./deploy.sh --no-push        # skip the push (you already pushed)
./deploy.sh --force          # skip the guards
./deploy.sh --checks         # lint, format, typecheck and test first
SITE=greatbear ./deploy.sh   # one Site instead of every Site

deploy.sh requires a clean tracked tree and HEAD == origin/main, tags the build with the commit it came from, and refuses a dirty tree for a reason: the image is built from git archive HEAD, so what runs on the host is exactly what git knows about. It runs make build-production COMMIT=<commit>, which builds linux/arm64 — the architecture emily runs — and pushes both the commit tag and latest to registry.tanshu.com/menu-admin.

By hand:

git push
make build-production COMMIT=$(git rev-parse --short HEAD)
cd ansible
ansible-playbook -u tanshu --inventory=hosts playbook.yml -e image_tag=$(git rev-parse --short HEAD)

Every Site runs on the same box, so hosts has one entry per Site pointing at the same ansible_host with its own var_file. The play carries no become: the Quadlet units belong to ansible_user (tanshu), and running as that user is what gives us a real PAM session, which is what makes systemctl --user work. Connecting as anyone else would write units into the wrong home.

Adding a Site is one line in hosts, a copy of vars/hopsngrains.yml, and a config/<site>.site.json. The image is shared and carries every Site's config, so a new Site needs a new build — which the commit tag gives it.

Migrating a Site's data from monoco

The menus still serve from Docker containers on monoco. A Site moves across with its data, before it is ever pointed at from DNS. Run this from a machine that can ssh to both hosts — the monoco and emily aliases this repo's own tooling uses:

scripts/copy-site-data.sh hopsngrains
scripts/copy-site-data.sh greatbear
scripts/copy-site-data.sh hngmohali

For one Site it stops the old container, streams /var/lib/menu.<domain> across, starts the old container again so monoco keeps serving until DNS moves, and then proves the copy: the SHA-256 of the whole tree is compared against the source, and every file is checked to belong to tanshu. --live skips the stop; MONOCO=, EMILY=, SRC_ROOT= and DEST_ROOT= override the defaults.

Two things make the copy look different from a plain rsync:

• sudo on monoco. /var/lib/menu.<domain> belongs to uid 1000 (the ubuntu account) and the ssh user there is 1001, so it is not readable without it. monoco's tanshu has passwordless sudo.
• The workstation relays it. monoco cannot ssh to emily, so the tar streams through the machine running the script; nothing large is written locally. Extracting as tanshu on emily is what makes the files land owned by the user keep-id maps the app onto — as root they would arrive owned by uid 1000, which on emily is nobody.

By hand, that is:

SITE=hopsngrains
HOST=menu.hopsngrains.com

ssh monoco "docker stop menu-app.$HOST"
ssh -n monoco "sudo -n tar -C /var/lib/$HOST -cf - ." \
  | ssh emily "mkdir -p ~/podman-data/menu-$SITE && tar -C ~/podman-data/menu-$SITE -xf -"
ssh monoco "docker start menu-app.$HOST"

# Same checks the script runs.
ssh -n monoco "sudo -n sh -c 'cd /var/lib/$HOST && find . -type f -exec sha256sum {} +' | sort | sha256sum"
ssh -n emily  "cd ~/podman-data/menu-$SITE && find . -type f -exec sha256sum {} + | sort | sha256sum"
ssh -n emily  "find ~/podman-data/menu-$SITE ! -user tanshu"

The old layout put everything under /var/lib/menu.<domain>/, with the content one level down in data/; the new one is ~/podman-data/menu-<site>/ with the same data/ inside it and site.env beside it. Nothing inside changes. site.env carries over so the admin keeps working; without it the app starts with the admin disabled and the playbook says so rather than failing.

Then deploy that Site and check it on a Host: override before touching DNS. monoco keeps serving until the cutover; only then stop its container — and leave it in place, not removed, so docker start is the rollback.

The admin password

The app starts with the admin disabled while ~/podman-data/menu-<site>/site.env is missing, and the playbook says so rather than failing. Create it once per Site; run it again with --force to reset the password (a fresh session secret signs everyone out). There is no minimum password length.

scp scripts/provision-site-env.sh emily:/tmp/
ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains
systemctl --user restart menu-hopsngrains.service

The -t matters: the script prompts for the password, and without a terminal on the remote side the prompt cannot be read, so it would sit there looking frozen. With no image argument it borrows the one the running container uses. It has to run as tanshu, the user whose rootless Podman owns the container.

Checking it landed

npm run check-site -- https://menu.hopsngrains.com

Read-only, exits non-zero on failure. Then from a phone: the landing page, a Section page, the page counter, tap-to-zoom, and signing into /admin to publish something.

Rolling back

Deploy the image built from an earlier commit — the commit tag is what makes this a one-liner, since every commit ever pushed to the registry is still there:

cd ansible
ansible-playbook -u tanshu --inventory=hosts playbook.yml -e image_tag=<older-commit>

podman logs menu-<site> and journalctl --user -u menu-<site> are where a container that will not start says why. To go all the way back to the Docker deployment, start the container that was left on monoco and point Caddy at it again.