diff --git a/.dockerignore b/.containerignore similarity index 85% rename from .dockerignore rename to .containerignore index f21dd74..baa251c 100644 --- a/.dockerignore +++ b/.containerignore @@ -8,8 +8,8 @@ public/fonts/OFL.txt docs test compose.yaml -Dockerfile -.dockerignore +Containerfile +.containerignore README.md # design prototype, never committed to main and never shipped public/prototype diff --git a/Dockerfile b/Containerfile similarity index 70% rename from Dockerfile rename to Containerfile index 6779e72..e18af39 100644 --- a/Dockerfile +++ b/Containerfile @@ -1,3 +1,6 @@ +# Built with podman. `podman build` reads a Containerfile as readily as a +# Dockerfile, and the name says which toolchain the pipeline actually uses. +# # The app runs TypeScript directly on Node 24, so there is no build step and no # compiler in the image. Node 24 on Alpine is musl, which sharp ships prebuilt # binaries for. @@ -25,6 +28,10 @@ USER node EXPOSE 3000 +# podman builds an OCI image by default, which drops this instruction, so the +# Quadlet unit declares the same check again as HealthCmd= (see +# ansible/roles/menu-app/templates/menu-app.container.j2). Keeping it here means +# the image still healthchecks itself when run directly. HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \ CMD node -e "fetch('http://127.0.0.1:'+(process.env.PORT||3000)+'/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" diff --git a/README.md b/README.md index dd06f4e..2e3e59d 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,9 @@ Built: Site config, the Menu Set store, the image pipeline, the customer-facing renderer (landing page, Section pages, archived sets, media route) with the approved design, the admin (sign in, create a set, upload and reorder Pages, preview, publish, restore, delete), the `verify`/`repair` CLI, the container and -the Ansible playbook. Not yet done: the parallel-run deployment on `monoco`, -which needs a human present. +the Ansible deployment as rootless Podman Quadlet units on `emily`, behind the +shared Caddy unit. Not yet done: copying each Site's data across from `monoco` +and moving DNS, which needs a human present. ## Deploying @@ -23,8 +24,10 @@ SKIP_DEPLOY=1 ./deploy.sh 2026-09-22 # build and push only The image is built with **podman** as a multi-arch manifest and pushed to `registry.tanshu.com/menu-admin` (`Makefile`, mirrored from the other apps on this machine). The Ansible deployment lives in `ansible/` in this repo, with one -inventory entry per Site; see `ansible/readme.md` for how a deploy runs and how to -roll one back. +inventory entry per Site. It writes a rootless Podman **Quadlet** unit per Site +and appends the Site's block to the shared Caddyfile, exactly as the sibling +services in `ansible-playbooks` do; see `ansible/readme.md` for how a deploy runs +and how to roll one back. ## Layout @@ -47,11 +50,14 @@ test/ node:test suites docs/plan.md The agreed plan docs/operating.md Publishing, fixing and rolling back, for whoever operates it docs/adr/ Decisions worth remembering -deploy.sh Tests, build, push +Containerfile The image, built by podman +compose.yaml Local dev: podman compose +deploy.sh Tests, build, push, deploy +ansible/ The deployment: one rootless Quadlet unit per Site ``` -The Ansible playbook lives in `ansible-playbooks/menu-admin/`, beside the other -playbooks, because that is where it is run from. +The deployment is in `ansible/` here, beside the app it deploys, and is run from +that directory by `deploy.sh`. ## Running it locally diff --git a/ansible/hosts b/ansible/hosts index 59bf344..62bc7d7 100644 --- a/ansible/hosts +++ b/ansible/hosts @@ -1,17 +1,24 @@ -# The Ansible inventory for the menus. Every Site runs on the same box, so each -# entry names it and points at the vars file that describes it. +# The Ansible inventory for the menus. Every Site runs on the same box — emily, +# the rootless Podman host — so each entry names it and points at the vars file +# that describes it. # # cd ansible # ansible-playbook --inventory=hosts playbook.yml # every Site # ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains # -# ansible_host is written as the resolvable name rather than an ssh alias so the -# inventory works on any machine, whatever its ~/.ssh/config says. +# This play carries no become: it writes rootless Quadlet units, which belong to +# whoever Ansible connects as, so it must connect as that user. +# +# ansible_host is written as the resolvable name rather than an ssh alias, so the +# inventory works on any machine, whatever its ~/.ssh/config says. emily has no +# DNS name yet, so that name is its address today; when DNS — or monoco's elastic +# address — moves to it at cutover, this is the one line that changes. [menus] -hopsngrains ansible_host=monoco.tanshu.com var_file=vars/hopsngrains.yml -greatbear ansible_host=monoco.tanshu.com var_file=vars/greatbear.yml -hngmohali ansible_host=monoco.tanshu.com var_file=vars/hngmohali.yml +hopsngrains ansible_host=35.154.14.59 var_file=vars/hopsngrains.yml +greatbear ansible_host=35.154.14.59 var_file=vars/greatbear.yml +hngmohali ansible_host=35.154.14.59 var_file=vars/hngmohali.yml [all:vars] +ansible_user=tanshu ansible_python_interpreter=/usr/bin/python3 diff --git a/ansible/playbook.yml b/ansible/playbook.yml index 91031c7..2ee898e 100644 --- a/ansible/playbook.yml +++ b/ansible/playbook.yml @@ -1,15 +1,26 @@ --- -# Deploys the menu app for one or more Sites. It lives here, beside the app it -# deploys, and is run from this directory: +# Deploys the menu app for one or more Sites as rootless Podman Quadlet units on +# emily, behind the shared Caddy unit that ansible-playbooks/caddy owns. It lives +# here, beside the app it deploys, and is run from this directory: # -# make build-production TAG= (from the repo root) -# ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains +# ./deploy.sh (repo root) +# ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag= # -# One run deploys the Site: it starts the container and points the Site's web -# address at it. Nothing older is kept serving, so there is no staged rollout — -# deploying an earlier tag is the rollback. -- hosts: menus - become: true +# Connects as the container user and runs everything as that user: the Quadlet +# units belong to it, so the connection's PAM session is what makes +# `systemctl --user` work without any environment juggling. +# +# gather_facts is required: podman_user and the home directory come from the +# user_id / user_dir facts. +# +# The rootless-Podman prerequisites (installing podman, enabling lingering, the +# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied +# here. They are the shared task file at +# ansible-playbooks/caddy/roles/podman/tasks/main.yaml, and a host that has +# already run ansible-playbooks/caddy/playbook.yml has them. +- name: Deploy the menu app as rootless Podman Quadlets + hosts: menus + gather_facts: true vars_files: - vars/default.yml @@ -19,7 +30,16 @@ file: '{{ var_file }}' when: var_file is defined + # This must happen before the app unit is started, not after: the generated + # unit references {{ network_name }}.network, and Quadlet fails *unit + # generation* — so `systemctl --user start` reports "unit not found" — when a + # referenced .network file does not exist yet. The same include writes the + # Caddy-side drop-in that attaches Caddy to this network. + - name: Register this Site's network with Caddy + ansible.builtin.include_tasks: '{{ playbook_dir }}/roles/register/tasks/main.yaml' + roles: - - network + # The app: the data directory tree, the Quadlet unit, start + verify. - menu-app + # Appends this Site's block to the shared Caddyfile and reloads Caddy. - caddy diff --git a/ansible/readme.md b/ansible/readme.md index 60179fd..df90600 100644 --- a/ansible/readme.md +++ b/ansible/readme.md @@ -1,26 +1,90 @@ # The menu deployment Lives here, beside the app it deploys. One run deploys a Site: it starts the -app's container and points the Site's web address at it. There is no older -system to keep serving, so there is no staged rollout and nothing to retire — -deploying an earlier tag is the rollback. +app's container as a **rootless Podman Quadlet unit** on `emily` and points the +Site's web address at it through the shared **Caddy** unit that +`ansible-playbooks/caddy` owns. There is no older system to keep serving, so +there is no staged rollout and nothing to retire — deploying an earlier tag is +the rollback. -**The app's data root is the directory the old static menus were served from** -(`/var/lib/menu.`), on purpose, so the app inherits the location instead of -a second one. The app only reads and writes `data/` and `site.env` inside it. If -you later want the old menu files gone, delete those files — not the directory, -which now holds the live content. +This is the same shape as the sibling services in `ansible-playbooks` (gitea, +nextcloud, tandoor, bitwarden, valkey, postgres, trillium): rootless Podman, one +Quadlet `.container` / `.network` per service in `~/.config/containers/systemd/`, +state under `~/podman-data/`, and Caddy reaching each app by name on a Podman +network. Nothing here talks to Docker. + +## The pieces + +``` +ansible/ +├── hosts # one entry per Site, all on emily +├── playbook.yml # connects as tanshu, no become +├── vars/{default,hopsngrains,greatbear,hngmohali}.yml +└── roles/ + ├── register/ # writes menu-.network and the Caddy drop-in + ├── menu-app/ # the Quadlet unit, start + health + verify + └── caddy/ # appends the site block, reloads Caddy +``` + +Two things are worth internalising, because they are the whole design: + +**Caddy knows nothing about its upstreams.** A service registers _itself_ by +writing two files into the Quadlet directory: its own `.network` unit and a +drop-in beside `caddy.container` that pulls that network into Caddy. This play +does that in `pre_tasks`, before the app unit starts, because Quadlet fails unit +generation when a unit references a `.network` file that does not exist yet. + +**The container's uid is mapped onto `tanshu`.** `UserNS=keep-id:uid=1000,gid=1000` +maps the image's own `node` account onto the user that owns the units, so every +file the app writes is owned by `tanshu` on the host: plain `tar`/`rsync` +backups, no subuids, no `podman unshare`. `scripts/provision-site-env.sh` uses +the same mapping for the same reason. ## What lives where -| Thing | Where | -| -------------------------------------- | ---------------------------------------------------------------------- | -| Name, tagline, icon, categories, menus | on the server under `/var/lib/menu./data` — the admin edits them | -| Theme colours | `config/.site.json` in the repo, baked into the image | -| Admin password hash, session secret | `/var/lib/menu./site.env` (untracked, mode 600) | -| Which image tag runs | `-e tag=` | -| Which Site is deployed | the inventory entry in `hosts` | -| The web address it serves | `host:` in `vars/.yml` | +| Thing | Where | +| -------------------------------------- | --------------------------------------------------------------------------- | +| Name, tagline, icon, categories, menus | on the server under `~/podman-data/menu-/data` — the admin edits them | +| Admin password hash, session secret | `~/podman-data/menu-/site.env` (untracked, mode 600) | +| Theme colours | `config/.site.json` in the repo, baked into the image | +| The unit that runs | `~/.config/containers/systemd/menu-.container` | +| The network and Caddy drop-in | `menu-.network`, `caddy.container.d/50-menu-.conf` | +| Which image tag runs | `-e tag=` | +| Which Site is deployed | the inventory entry in `hosts` | +| The web address it serves | `host:` in `vars/.yml` | + +`` is the inventory name (`hopsngrains`, `greatbear`, `hngmohali`). The +container itself keeps the Docker-era name `menu-app.`, because +`provision-site-env.sh` and muscle memory look it up that way; the Caddyfile +proxies to the dotless `menu-` alias, because a container name with dots in +it is a liability in DNS. + +## What the host needs + +`emily` is already set up — it runs Caddy, Postgres, Tandoor and Valkey the same +way. If you are standing up a fresh host, run `ansible-playbooks/caddy` first: it +installs podman, enables lingering, lowers the privileged-port floor so rootless +Caddy can bind 80/443, creates the Quadlet directory, and starts the Caddy unit. +Those prerequisites deliberately are **not** duplicated here, so a host that has +not run that playbook will fail at the first `systemctl --user` — see +`ansible-playbooks/caddy/readme.md`. + +The image is in the private registry, so log in once as `tanshu` on `emily`, +before the first deploy: + +```sh +podman login registry.tanshu.com +``` + +The play pulls the image itself, so a missing login fails there with that +instruction rather than as a container that will not start. The unit keeps +podman's default `Pull=missing` on purpose, so a restart or a reboot never +depends on the registry being reachable. + +`ssh` on some machines complains about +`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is +a local file-permission bug whose fix is `sudo chown -R root:root +/etc/ssh/ssh_config.d`, not something this playbook works around. ## Deploying @@ -33,8 +97,8 @@ SKIP_DEPLOY=1 ./deploy.sh 2026-09-22 # build and push only `deploy.sh` runs `make build-production TAG=`, which builds a multi-arch podman manifest (`linux/amd64,linux/arm64`) and pushes `latest` and the tag to -`registry.tanshu.com/menu-admin`. Then it runs this playbook against the Site -named by `SITE` (default `hopsngrains`). +`registry.tanshu.com/menu-admin`. Then it runs this playbook against every Site, +or the one named by `SITE`. By hand: @@ -45,16 +109,89 @@ ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=2026- ``` Every Site runs on the same box, so `hosts` has one entry per Site pointing at -the same `ansible_host` with its own `var_file`. `ansible_host` is written as the -resolvable name rather than an ssh alias, so the inventory works whatever a -machine's `~/.ssh/config` says. (`ssh` on some machines complains about -`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is -a local file-permission bug whose fix is `sudo chown -R root:root -/etc/ssh/ssh_config.d`, not something this playbook works around.) +the same `ansible_host` with its own `var_file`. The play carries no `become`: the +Quadlet units belong to `ansible_user` (`tanshu`), and running as that user is +what gives us a real PAM session, which is what makes `systemctl --user` work. +Connecting as anyone else would write units into the wrong home. Adding a Site is one line in `hosts`, a copy of `vars/hopsngrains.yml`, and a -`config/.site.json`. The image is shared, so the config has to be in it: -a new Site needs a new tag. +`config/.site.json`. The image is shared, so the config has to be in it: a +new Site needs a new tag. + +## Migrating a Site's data from `monoco` + +The menus still serve from Docker containers on `monoco`. A Site moves across +with its data, before it is ever pointed at from DNS. Run this from a machine +that can ssh to both hosts — the `monoco` and `emily` aliases this repo's own +tooling uses: + +```sh +scripts/copy-site-data.sh hopsngrains +scripts/copy-site-data.sh greatbear +scripts/copy-site-data.sh hngmohali +``` + +For one Site it stops the old container, streams `/var/lib/menu.` across, +starts the old container again so `monoco` keeps serving until DNS moves, and +then proves the copy: the SHA-256 of the whole tree is compared against the +source, and every file is checked to belong to `tanshu`. `--live` skips the stop; +`MONOCO=`, `EMILY=`, `SRC_ROOT=` and `DEST_ROOT=` override the defaults. + +Two things make the copy look different from a plain `rsync`: + +- **`sudo` on monoco.** `/var/lib/menu.` belongs to uid 1000 (the + `ubuntu` account) and the ssh user there is 1001, so it is not readable without + it. monoco's `tanshu` has passwordless sudo. +- **The workstation relays it.** monoco cannot ssh to emily, so the tar streams + through the machine running the script; nothing large is written locally. + Extracting as `tanshu` on emily is what makes the files land owned by the user + `keep-id` maps the app onto — as root they would arrive owned by uid 1000, + which on emily is nobody. + +By hand, that is: + +```sh +SITE=hopsngrains +HOST=menu.hopsngrains.com + +ssh monoco "docker stop menu-app.$HOST" +ssh -n monoco "sudo -n tar -C /var/lib/$HOST -cf - ." \ + | ssh emily "mkdir -p ~/podman-data/menu-$SITE && tar -C ~/podman-data/menu-$SITE -xf -" +ssh monoco "docker start menu-app.$HOST" + +# Same checks the script runs. +ssh -n monoco "sudo -n sh -c 'cd /var/lib/$HOST && find . -type f -exec sha256sum {} +' | sort | sha256sum" +ssh -n emily "cd ~/podman-data/menu-$SITE && find . -type f -exec sha256sum {} + | sort | sha256sum" +ssh -n emily "find ~/podman-data/menu-$SITE ! -user tanshu" +``` + +The old layout put everything under `/var/lib/menu./`, with the content +one level down in `data/`; the new one is `~/podman-data/menu-/` with the +same `data/` inside it and `site.env` beside it. Nothing inside changes. +`site.env` carries over so the admin keeps working; without it the app starts +with the admin disabled and the playbook says so rather than failing. + +Then deploy that Site and check it on a `Host:` override before touching DNS. +`monoco` keeps serving until the cutover; only then stop its container — and +leave it in place, not removed, so `docker start` is the rollback. + +## The admin password + +The app starts with the admin disabled while `~/podman-data/menu-/site.env` +is missing, and the playbook says so rather than failing. Create it once per Site; +run it again with `--force` to reset the password (a fresh session secret signs +everyone out). There is no minimum password length. + +```sh +scp scripts/provision-site-env.sh emily:/tmp/ +ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains +systemctl --user restart menu-hopsngrains.service +``` + +The `-t` matters: the script prompts for the password, and without a terminal on +the remote side the prompt cannot be read, so it would sit there looking frozen. +With no image argument it borrows the one the running container uses. It has to +run as `tanshu`, the user whose rootless Podman owns the container. ## Checking it landed @@ -63,25 +200,8 @@ npm run check-site -- https://menu.hopsngrains.com ``` Read-only, exits non-zero on failure. Then from a phone: the landing page, a -Section page, the page counter, tap-to-zoom, and signing into `/admin` to -publish something. - -## The admin password - -The app starts with the admin disabled while `/var/lib/menu./site.env` -is missing, and the playbook says so rather than failing. Create it once per -Site; run it again with `--force` to reset the password (a fresh session secret -signs everyone out). There is no minimum password length. - -```sh -scp scripts/provision-site-env.sh monoco:/tmp/ -ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com -ssh monoco docker restart menu-app.hopsngrains.com -``` - -The `-t` matters: the script prompts for the password, and without a terminal on -the remote side the prompt cannot be read, so it would sit there looking frozen. -With no image argument it borrows the one the running container uses. +Section page, the page counter, tap-to-zoom, and signing into `/admin` to publish +something. ## Rolling back @@ -91,4 +211,7 @@ Deploy the previous tag: ./deploy.sh ``` -`docker logs menu-app.` is where a failed container says why. +`podman logs menu-app.` and `journalctl --user -u menu-` are where a +container that will not start says why. To go all the way back to the Docker +deployment, start the container that was left on `monoco` and point Caddy at it +again. diff --git a/ansible/roles/caddy/handlers/main.yaml b/ansible/roles/caddy/handlers/main.yaml index bd3fe90..16ffe76 100644 --- a/ansible/roles/caddy/handlers/main.yaml +++ b/ansible/roles/caddy/handlers/main.yaml @@ -1,6 +1,31 @@ --- -- name: Reload Caddy - # Caddy validates before it applies, so a mistake here leaves the running - # config untouched and fails the play loudly. - command: 'docker exec -w /etc/caddy {{ caddy_container }} caddy reload' - listen: Reload Caddy +# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless +# Quadlet unit there. These handlers run as the connection user, so +# `systemctl --user` and `podman exec` work without any XDG_RUNTIME_DIR juggling. + +# Notified by this role when the Caddyfile changes. A reload keeps existing +# connections; a restart would briefly drop every hosted site. +# +# Note the config path: /etc/caddy/Caddyfile is the mount inside the container, +# not {{ caddyfile_path }}, which is the host side of it. +- name: Reload Caddy configuration + ansible.builtin.command: >- + podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile + changed_when: true + listen: 'Reload Caddy' + +# Notified by roles/register when this Site's network unit, or the drop-in that +# attaches Caddy to it, is written. Quadlet has no live equivalent of +# `docker network connect`, so the container has to be recreated — a brief +# restart of every hosted site, not just this one. See the Caddy playbook's +# readme, "Trade-off worth knowing". +# +# Defined after the reload above so that when both are notified the cheap config +# reload happens first. +- name: Restart Caddy + ansible.builtin.systemd_service: + name: '{{ caddy_name }}.service' + scope: user + state: restarted + daemon_reload: true + listen: Restart Caddy diff --git a/ansible/roles/caddy/tasks/main.yaml b/ansible/roles/caddy/tasks/main.yaml index b36ee25..87b0911 100644 --- a/ansible/roles/caddy/tasks/main.yaml +++ b/ansible/roles/caddy/tasks/main.yaml @@ -1,14 +1,29 @@ --- +# Append this Site's block to the shared Caddyfile. +# +# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless +# Quadlet unit there, so this role does not manage the container — it only points +# it at the app. The Caddyfile is a bind mount into that container, so Caddy sees +# a written change as soon as the file lands; the handler only asks Caddy to +# re-read it. That is a reload, not a restart, so the other hosted sites are +# untouched. +# +# blockinfile is idempotent on its own via the marker, so there is no need to +# slurp the file and compare first — which also matters because slurp *fails* +# when the Caddyfile does not exist yet, on a host where the Caddy playbook has +# not run. + - name: Check the Caddyfile exists - stat: + ansible.builtin.stat: path: '{{ caddyfile_path }}' register: menu_caddyfile - name: Fail when Caddy is not where we expect it - fail: + ansible.builtin.fail: msg: >- - {{ caddyfile_path }} is missing. Deploy the caddy playbook, which owns the - Caddy container and its config, before this one. + {{ caddyfile_path }} is missing. Deploy the Caddy playbook + (ansible-playbooks/caddy/playbook.yml), which owns the Caddy unit and its + config, before this one. when: not menu_caddyfile.stat.exists # The Site's web address may still be claimed by an older snippet: the retired @@ -16,7 +31,7 @@ # Caddy refuse the whole config, and a refused reload leaves the old proxy in # place, which is a 502 once the old container is gone. - name: Drop any older snippet claiming this web address - blockinfile: + ansible.builtin.blockinfile: path: '{{ caddyfile_path }}' marker: '# {mark} Ansible managed Caddy snippet for {{ host }}' state: absent @@ -25,9 +40,12 @@ # blockinfile replaces whatever sits between its markers, so redeploying updates # the snippet in place rather than appending a second one. - name: Point the Site's web address at the app - blockinfile: + ansible.builtin.blockinfile: path: '{{ caddyfile_path }}' marker: '# {mark} ANSIBLE MANAGED: menu app ({{ site }})' block: '{{ caddy_snippet }}' - create: no + create: false + owner: '{{ podman_user }}' + group: '{{ podman_user }}' + mode: '0644' notify: Reload Caddy diff --git a/ansible/roles/menu-app/handlers/main.yaml b/ansible/roles/menu-app/handlers/main.yaml new file mode 100644 index 0000000..eac3ab9 --- /dev/null +++ b/ansible/roles/menu-app/handlers/main.yaml @@ -0,0 +1,19 @@ +--- +# All handlers run as the connection user ({{ podman_user }}), so systemctl +# --user works without setting XDG_RUNTIME_DIR. + +- name: Reload systemd user units + ansible.builtin.command: systemctl --user daemon-reload + changed_when: true + listen: Reload systemd user units + +# A changed unit file changes the generated unit, so the app is recreated. All of +# its state is on the bind mount, so a recreate loses nothing — but it is a few +# seconds of downtime for the one Site. +- name: Restart the menu app + ansible.builtin.systemd_service: + name: '{{ unit_name }}' + scope: user + state: restarted + daemon_reload: true + listen: Restart the menu app diff --git a/ansible/roles/menu-app/tasks/main.yaml b/ansible/roles/menu-app/tasks/main.yaml index ad8dd42..e5523cc 100644 --- a/ansible/roles/menu-app/tasks/main.yaml +++ b/ansible/roles/menu-app/tasks/main.yaml @@ -1,134 +1,178 @@ --- -- name: Pull the app image - docker_image: - name: '{{ image_name }}' - tag: '{{ image_tag }}' - source: pull - force_source: yes +# The menu app as a rootless Podman Quadlet unit. +# +# Everything here runs as {{ podman_user }} — the connection user. No become, no +# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's +# user manager is reachable through the standard environment. +# The pull is explicit, rather than left to the Quadlet unit, for two reasons. +# The image lives in a private registry, so a failure here is where a missing +# `podman login` should show up — not two tasks later as "unit not found". And +# the unit keeps podman's default `Pull=missing`, so it never reaches for the +# registry on a restart or after a reboot: that is what lets a box come back up +# when the registry is down. Pulling here is what keeps a redeployed tag current. +- name: Pull the app image + ansible.builtin.command: 'podman pull {{ image_name }}:{{ image_tag }}' + register: menu_pull + changed_when: false + failed_when: false + +- name: Fail when the image cannot be pulled + ansible.builtin.fail: + msg: >- + podman could not pull {{ image_name }}:{{ image_tag }}. + {{ menu_pull.stderr | default('') | trim }} + The registry is private: run `podman login registry.tanshu.com` once as + {{ podman_user }} on this host. + when: menu_pull.rc != 0 + +# The data directory is created here rather than left to the image. Under +# keep-id the image's own root phase maps to a subuid, so anything it tries to +# create under the mount fails with EPERM; the directory has to exist already, +# owned by {{ podman_user }}. - name: Ensure the app root exists - file: + ansible.builtin.file: path: '{{ app_root }}' state: directory - owner: '{{ app_uid }}' - group: '{{ app_gid }}' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' mode: '0750' +# MENU_ROOT is the app root, and the app keeps its content one level down in +# data/ (originals/, drafts/, published/, site.json, sections.json). - name: Ensure the data directory exists - file: + ansible.builtin.file: path: '{{ app_root }}/data' state: directory - owner: '{{ app_uid }}' - group: '{{ app_gid }}' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' mode: '0750' - name: Look for the untracked site.env - stat: + ansible.builtin.stat: path: '{{ app_root }}/site.env' register: menu_site_env - name: Warn when the admin has no password yet - debug: + ansible.builtin.debug: msg: >- {{ app_root }}/site.env is missing, so the admin is disabled: the public - menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on - this host, then re-run this playbook. + menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on this + host, then re-run this playbook. when: not menu_site_env.stat.exists -- name: Run the menu app - docker_container: - name: '{{ docker_container }}' - image: '{{ image_name }}:{{ image_tag }}' - state: started - restart_policy: unless-stopped - user: '{{ app_uid }}:{{ app_gid }}' - # Force a rebuild of the container with -e recreate_container=true: that is - # what applies a changed healthcheck or env to an existing container, since - # the module otherwise leaves a running one alone. - recreate: '{{ recreate_container | default(false) | bool }}' - # Declared here rather than relied on from the image: podman builds an OCI - # image by default and drops Docker's HEALTHCHECK instruction, so the - # Dockerfile's own healthcheck never reaches the container. node is in the - # image and the exec form needs no shell quoting. - healthcheck: - test: - - CMD - - node - - -e - - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))" - interval: 30s - timeout: 5s - start_period: 30s - retries: 3 - env: - MENU_ROOT: /data - MENU_SITE_CONFIG: '/app/config/{{ site }}.site.json' - SECURE_COOKIES: 'true' - LOG_LEVEL: info - PORT: '3000' - volumes: - - '{{ app_root }}:/data' - networks: - - name: '{{ docker_network }}' +- name: Install the menu app Quadlet unit + ansible.builtin.template: + src: menu-app.container.j2 + dest: '{{ quadlet_dir }}/{{ network_name }}.container' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' + mode: '0644' + notify: + - Reload systemd user units + - Restart the menu app -- name: Keep site.env readable only by the app user - file: - path: '{{ app_root }}/site.env' - owner: '{{ app_uid }}' - group: '{{ app_gid }}' - mode: '0600' - when: menu_site_env.stat.exists +- name: Start the menu app + block: + - name: Start the menu app + ansible.builtin.systemd_service: + name: '{{ unit_name }}' + scope: user + state: started + daemon_reload: true + rescue: + # systemd reports "control process exited with error code" and nothing else, + # which is useless for the usual failures (a missing network unit, an + # unwritable /data, a bad uid mapping). Pull the real reason out. + - name: Collect diagnostics + ansible.builtin.command: '{{ item }}' + loop: + - 'systemctl --user status {{ unit_name }} --no-pager' + - 'journalctl --user -u {{ unit_name }} --no-pager -n 40' + - 'podman logs --tail 40 {{ container_name }}' + register: menu_diagnostics + changed_when: false + failed_when: false -# Deliberately not the image's HEALTHCHECK: podman builds an OCI image by -# default and drops the Docker HEALTHCHECK instruction, so State.Health is absent -# and a wait on it fails on a perfectly healthy container — which stops the play -# before Caddy is ever pointed at the app. -- name: Wait for the app to answer inside its own container - ansible.builtin.command: - argv: - - docker - - exec - - '{{ docker_container }}' - - node - - -e - - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))" - register: menu_health + - name: Show diagnostics + ansible.builtin.debug: + msg: '{{ menu_diagnostics.results | map(attribute="stdout") | list }}' + + - name: Fail + ansible.builtin.fail: + msg: 'The menu app did not start — see the diagnostics above.' + +# Quadlet honours [Install] WantedBy= when it generates the unit, so +# `systemctl --user enable` is neither needed nor possible on a generated unit. +- name: Verify the menu app is running + ansible.builtin.command: 'systemctl --user is-active {{ unit_name }}' + register: menu_active changed_when: false - retries: 20 - delay: 3 + failed_when: menu_active.stdout != 'active' + +# `is-active` says the container is running; it does not say the app can serve. +# Restart=always means even a crash-looping container is reported active — which +# is exactly what a fatal integrity problem in the data directory produces. The +# unit declares the health check, so run that once, synchronously. +- name: Wait for the app to pass its health check + ansible.builtin.command: 'podman healthcheck run {{ container_name }}' + register: menu_health until: menu_health.rc == 0 + retries: 24 + delay: 5 + changed_when: false failed_when: false -- name: Fail when the app never answers - fail: - msg: >- - {{ docker_container }} did not answer /healthz within a minute. Its logs - say why: docker logs {{ docker_container }} +- name: Report a container that never answered when: menu_health.rc != 0 + block: + - name: Collect diagnostics + ansible.builtin.command: '{{ item }}' + loop: + - 'systemctl --user status {{ unit_name }} --no-pager' + - 'journalctl --user -u {{ unit_name }} --no-pager -n 40' + - 'podman logs --tail 40 {{ container_name }}' + register: menu_health_diagnostics + changed_when: false + failed_when: false + + - name: Show diagnostics + ansible.builtin.debug: + msg: '{{ menu_health_diagnostics.results | map(attribute="stdout") | list }}' + + - name: Fail + ansible.builtin.fail: + msg: >- + {{ container_name }} did not pass its health check within two minutes. + Its logs are above, and: podman logs {{ container_name }} + +- name: Keep site.env readable only by the app user + ansible.builtin.file: + path: '{{ app_root }}/site.env' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' + mode: '0600' + when: menu_site_env.stat.exists - name: Check the data directory for integrity problems # Reads the same directory the container writes, as the same user, so a # permissions mistake shows up here rather than at 8pm service. - command: + ansible.builtin.command: argv: - - docker - - run - - --rm - - --user - - '{{ app_uid }}:{{ app_gid }}' - - -e + - podman + - exec + - --env - MENU_ROOT=/data - - -e + - --env - 'MENU_SITE_CONFIG=/app/config/{{ site }}.site.json' - - --entrypoint + - '{{ container_name }}' - node - - '{{ image_name }}:{{ image_tag }}' - src/cli/verify.ts register: menu_verify changed_when: false failed_when: false - name: Report integrity problems - debug: + ansible.builtin.debug: msg: '{{ menu_verify.stdout_lines | default([]) }}' when: menu_verify.rc != 0 diff --git a/ansible/roles/menu-app/templates/menu-app.container.j2 b/ansible/roles/menu-app/templates/menu-app.container.j2 new file mode 100644 index 0000000..33cf9ba --- /dev/null +++ b/ansible/roles/menu-app/templates/menu-app.container.j2 @@ -0,0 +1,60 @@ +# {{ container_name }} — rootless Podman Quadlet unit. +# Managed by Ansible (menu-admin/roles/menu-app). Do not edit by hand. +# +# Joins {{ network_name }}.network, written by roles/register — the same include +# that attaches Caddy to it. Caddy reaches the app by container name, so nothing +# is published on the host and the app is not reachable from outside the Podman +# network at all. + +[Unit] +Description=Menu app for {{ host }} +After=network-online.target +Wants=network-online.target + +[Container] +ContainerName={{ container_name }} +Image={{ image_name }}:{{ image_tag }} + +# keep-id maps the image's own account — `node`, uid/gid {{ app_uid }} — onto +# {{ podman_user }}, which is what keeps every file the app writes owned by +# {{ podman_user }} on the host: plain tar/rsync backups, no subuids, no +# `podman unshare`. The same mapping is why scripts/provision-site-env.sh writes +# site.env with --userns=keep-id. +UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }} + +# The alias is deliberately dotless. ContainerName keeps the Docker-era name so +# operators and scripts find it, but a name with dots in it is a liability in +# DNS, so {{ network_name }} is what the Caddyfile proxies to. Both resolve from +# inside the network. +NetworkAlias={{ network_name }} + +Environment=MENU_ROOT=/data +Environment=MENU_SITE_CONFIG=/app/config/{{ site }}.site.json +Environment=SECURE_COOKIES=true +Environment=LOG_LEVEL=info +Environment=PORT={{ container_port }} + +# Everything the app writes: data/ (originals, drafts, published, site.json, +# sections.json) and site.env. The Site's own config is baked into the image and +# read from /app. +Volume={{ app_root }}:/data + +Network={{ network_name }}.network + +# podman builds an OCI image by default and drops the Containerfile's +# HEALTHCHECK instruction, so the same check is declared here and the play gates +# on it with `podman healthcheck run`. node is in the image. +HealthCmd={{ health_cmd }} +HealthInterval=30s +HealthTimeout=5s +HealthRetries=3 +HealthStartPeriod=30s + +[Service] +Restart=always +# The app validates its data directory before it serves and refuses to boot on a +# fatal integrity problem, so a first start over copied data can take a moment. +TimeoutStartSec=120 + +[Install] +WantedBy=default.target diff --git a/ansible/roles/network/tasks/main.yaml b/ansible/roles/network/tasks/main.yaml deleted file mode 100644 index 979dde4..0000000 --- a/ansible/roles/network/tasks/main.yaml +++ /dev/null @@ -1,8 +0,0 @@ ---- -- name: 'Ensure {{ docker_network }} exists and Caddy is on it' - docker_network: - name: '{{ docker_network }}' - state: present - connected: - - '{{ caddy_container }}' - appends: yes diff --git a/ansible/roles/register/tasks/main.yaml b/ansible/roles/register/tasks/main.yaml new file mode 100644 index 0000000..a5336ec --- /dev/null +++ b/ansible/roles/register/tasks/main.yaml @@ -0,0 +1,51 @@ +--- +# Register this Site's logical network with Caddy. +# +# Copied from ansible-playbooks/caddy/roles/register/tasks/main.yaml: this repo +# carries its own deployment, so it cannot include that file by path. Keep the +# two in step. +# +# Caddy is the *base* unit and knows nothing about its upstreams. A service adds +# itself by writing two files into the Quadlet directory: +# +# 1. its own .network unit, and +# 2. a drop-in next to caddy.container that pulls that network into Caddy. +# +# Caddy itself is owned by ansible-playbooks/caddy/playbook.yml; this play only +# registers with it. The including play defines a "Restart Caddy" handler. + +- name: 'Create the {{ network_name }} network unit' + ansible.builtin.copy: + dest: '{{ quadlet_dir }}/{{ network_name }}.network' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' + mode: '0644' + content: | + # {{ network_name }} — logical app network. + # Managed by Ansible. Do not edit by hand. + # + # NetworkName pins the Podman network name. Without it Quadlet would + # create "systemd-{{ network_name }}". + # + # isolate blocks traffic between separate Podman networks, so a container + # on this network cannot reach one that is only on another. Caddy and any + # shared service still reach it, because they are members of it. + [Network] + NetworkName={{ network_name }}_net + Options=isolate=true + notify: Restart Caddy + +- name: 'Register {{ network_name }} with Caddy' + ansible.builtin.copy: + dest: '{{ quadlet_dir }}/{{ caddy_name }}.container.d/50-{{ network_name }}.conf' + owner: '{{ podman_user }}' + group: '{{ podman_user }}' + mode: '0644' + content: | + # Added by the {{ network_name }} playbook. Do not edit by hand. + # + # Quadlet merges this into caddy.container. Deleting this file removes + # the network from Caddy on the next daemon-reload. + [Container] + Network={{ network_name }}.network + notify: Restart Caddy diff --git a/ansible/vars/default.yml b/ansible/vars/default.yml index ab4ba28..07f6a17 100644 --- a/ansible/vars/default.yml +++ b/ansible/vars/default.yml @@ -2,19 +2,58 @@ # Defaults shared by every Site. Each Site's own file is named by the `var_file` # in ansible/hosts, and its slug is its inventory name. +# --------------------------------------------------------------------------- +# Rootless Podman / Quadlet +# --------------------------------------------------------------------------- +# podman_user is derived from the connection: this play carries no become, so the +# Quadlet units belong to whoever Ansible connects as. Requires gather_facts +# (set in playbook.yml). +# +# These three duplicate caddy/vars/default.yml in the ansible-playbooks repo, +# which owns the Caddy unit these Sites sit behind. Keep them in sync. +podman_user: "{{ ansible_facts['user_id'] }}" +podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data" +quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd" + # Which Site config inside the image to run, and which web address it serves. site: '{{ inventory_hostname }}' +# --------------------------------------------------------------------------- +# The app +# --------------------------------------------------------------------------- image_name: registry.tanshu.com/menu-admin image_tag: "{{ tag | default('latest') }}" +# The unit file is menu-.container and the Podman network is +# menu-_net. The container keeps the Docker-era name, dots and all, because +# scripts/provision-site-env.sh and anything an operator types look it up that +# way — but a name with dots is a DNS liability, so the Caddyfile proxies to the +# dotless NetworkAlias the unit sets beside it. +network_name: 'menu-{{ site }}' +unit_name: '{{ network_name }}.service' +container_name: 'menu-app.{{ host }}' + +# The image's own account (`node`), which keep-id maps onto podman_user. app_uid: 1000 app_gid: 1000 -caddy_container: caddy -caddyfile_path: /var/lib/caddy/conf/Caddyfile +container_port: 3000 + +# podman builds an OCI image by default and drops the Containerfile's +# HEALTHCHECK instruction, so the unit declares the same check and the play gates +# on it. node is in the image and the sh -c podman wraps it in handles the quotes. +health_cmd: >- + node -e "fetch('http://127.0.0.1:{{ container_port }}/healthz').then(r => + process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))" + +# --------------------------------------------------------------------------- +# The shared Caddy unit and its Caddyfile, both owned by the caddy playbook in +# ansible-playbooks (caddy/playbook.yml). +# --------------------------------------------------------------------------- +caddy_name: caddy +caddyfile_path: '{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile' caddy_snippet: | {{ host }} { - reverse_proxy {{ docker_container }}:3000 + reverse_proxy {{ network_name }}:{{ container_port }} } diff --git a/ansible/vars/greatbear.yml b/ansible/vars/greatbear.yml index 96aa27c..60c3e11 100644 --- a/ansible/vars/greatbear.yml +++ b/ansible/vars/greatbear.yml @@ -1,13 +1,11 @@ --- -# The Great Bear. Its name, tagline, icon, categories and menus live on the server under -# app_root and are edited in the admin; config/greatbear.site.json only seeds the -# theme and the first category list. +# The Great Bear. Its name, tagline, icon, categories and menus live on the +# server under app_root and are edited in the admin; config/greatbear.site.json +# only seeds the theme and the first category list. title: greatbear.in host: 'menu.{{ title }}' -docker_container: 'menu-app.{{ title }}' -docker_network: '{{ docker_container }}_net' - -# Everything the app writes — data/ and the untracked site.env. Deliberately the -# directory the old static menus were served from. -app_root: '/var/lib/{{ host }}' +# Everything the app writes — data/ (the content) and site.env (the admin +# password hash and session secret). One directory per Site, each of which is +# also its own Podman network and its own Caddy upstream. +app_root: '{{ podman_data_dir }}/{{ network_name }}' diff --git a/ansible/vars/hngmohali.yml b/ansible/vars/hngmohali.yml index 986b0d3..3bfc7f2 100644 --- a/ansible/vars/hngmohali.yml +++ b/ansible/vars/hngmohali.yml @@ -1,13 +1,11 @@ --- -# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the server under -# app_root and are edited in the admin; config/hngmohali.site.json only seeds the -# theme and the first category list. +# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the +# server under app_root and are edited in the admin; config/hngmohali.site.json +# only seeds the theme and the first category list. title: hngmohali.com host: 'menu.{{ title }}' -docker_container: 'menu-app.{{ title }}' -docker_network: '{{ docker_container }}_net' - -# Everything the app writes — data/ and the untracked site.env. Deliberately the -# directory the old static menus were served from. -app_root: '/var/lib/{{ host }}' +# Everything the app writes — data/ (the content) and site.env (the admin +# password hash and session secret). One directory per Site, each of which is +# also its own Podman network and its own Caddy upstream. +app_root: '{{ podman_data_dir }}/{{ network_name }}' diff --git a/ansible/vars/hopsngrains.yml b/ansible/vars/hopsngrains.yml index 4ee10c7..0d22fdd 100644 --- a/ansible/vars/hopsngrains.yml +++ b/ansible/vars/hopsngrains.yml @@ -1,7 +1,7 @@ --- -# Hops n Grains. Its name, tagline, icon, categories and menus live on the -# server under app_root and are edited in the admin; config/hopsngrains.site.json -# only seeds the theme. This file says where the Site runs and what it answers. +# Hops n Grains. Its name, tagline, icon, categories and menus live on the server +# under app_root and are edited in the admin; config/hopsngrains.site.json only +# seeds the theme. This file says where the Site runs and what it answers. # # Every value that interpolates a template is quoted: YAML reads a leading `{` # as a flow mapping, which is the parse error Ansible reports as "missing quotes @@ -9,13 +9,7 @@ title: hopsngrains.com host: 'menu.{{ title }}' -docker_container: 'menu-app.{{ title }}' -docker_network: '{{ docker_container }}_net' - -# Everything the app writes — data/ and the untracked site.env. -# -# Deliberately the directory the old static menus were served from, so the app -# inherits the location rather than a second one. The app only ever reads and -# writes its own data/ inside it; the old menu files can be emptied out when -# they are no longer wanted, and the directory itself must stay. -app_root: '/var/lib/{{ host }}' +# Everything the app writes — data/ (the content) and site.env (the admin +# password hash and session secret). One directory per Site, each of which is +# also its own Podman network and its own Caddy upstream. +app_root: '{{ podman_data_dir }}/{{ network_name }}' diff --git a/compose.yaml b/compose.yaml index e91b801..9b578af 100644 --- a/compose.yaml +++ b/compose.yaml @@ -1,7 +1,10 @@ -# Local development. Production is the menu-admin playbook in ansible-playbooks; -# this file exists so the container can be tried out without touching a server. +# Local development. Production is the playbook in ansible/; this file exists so +# the container can be tried out without touching a server. # -# npm run seed && docker compose up --build +# npm run seed && podman compose up --build +# +# `podman compose` needs a provider on PATH — podman-compose or docker-compose. +# The image is built from the Containerfile beside this file. services: menu: build: . diff --git a/deploy.sh b/deploy.sh index 543137f..500ea08 100755 --- a/deploy.sh +++ b/deploy.sh @@ -1,7 +1,9 @@ #!/usr/bin/env bash # # Builds the image with podman, pushes it to the private registry, then deploys -# the Site with the playbook that lives in ansible/. +# the Site with the playbook that lives in ansible/. The playbook runs rootless +# Podman Quadlet units on emily, behind the shared Caddy unit from +# ansible-playbooks/caddy. # # ./deploy.sh 2026-09-22 deploy the default Site # SITE=greatbear ./deploy.sh 2026-09-22 deploy another Site @@ -24,6 +26,11 @@ if ! command -v podman >/dev/null 2>&1; then exit 1 fi +if [[ -z "${SKIP_DEPLOY:-}" ]] && ! command -v ansible-playbook >/dev/null 2>&1; then + echo "ansible-playbook is required to deploy; SKIP_DEPLOY=1 builds and pushes only" >&2 + exit 1 +fi + if [[ -z "${SKIP_CHECKS:-}" ]]; then echo "==> lint, format, types and tests" npm run lint diff --git a/docs/adr/0009-rootless-podman-quadlets.md b/docs/adr/0009-rootless-podman-quadlets.md new file mode 100644 index 0000000..336f629 --- /dev/null +++ b/docs/adr/0009-rootless-podman-quadlets.md @@ -0,0 +1,31 @@ +# Sites run as rootless Podman Quadlets behind a shared Caddy + +Each Site is deployed as a **rootless Podman** container managed by a systemd +user unit (Quadlet) on `emily`, reached through the one shared Caddy unit rather +than a long-lived rootful Docker container. The image is built by `podman` too, +from a `Containerfile`. This matches how every other service on that host is +deployed, and it takes the deployment off Docker entirely. + +The app owns its whole serving path, so it needed the smallest deployment of the +fleet: one `.container` file, one `.network` file, and a block appended to the +Caddyfile. Two details carry the design. `UserNS=keep-id:uid=1000,gid=1000` maps +the image's own `node` account onto the user that owns the units, which is what +keeps `data/` and `site.env` owned by a real host user — plain `tar` backups, no +subuids — and `scripts/provision-site-env.sh` uses the same mapping so both sides +agree on who owns the file. And the Site registers its own network with Caddy by +writing a drop-in beside `caddy.container`, so Caddy never lists its upstreams +and never has to be edited when a Site is added or removed. + +**Considered Options**: staying on rootful Docker on `monoco`, rejected because +the fleet is moving and leaving one app behind means two runtimes and two Caddy +configs to reason about; running rootful Podman, rejected because it keeps the +same root-owned data and loses the plain-backup property; and naming the Quadlet +unit after the container (`menu-app.`), rejected because the dots make +the unit name awkward and the name fragile in DNS — the unit is `menu-` +while the container keeps its old name for the scripts that look it up. + +**Consequences**: the app is not published on a host port at all — only Caddy, +on the shared Podman network, can reach it — and deployment, logs and restarts +move from `docker` to `systemctl --user` / `podman`. The data root moves to +`~/podman-data/menu-`, so each Site's content has to be copied across from +`/var/lib/menu.` once, before DNS points at it. diff --git a/docs/operating.md b/docs/operating.md index 7a2581a..df480e1 100644 --- a/docs/operating.md +++ b/docs/operating.md @@ -2,19 +2,26 @@ For whoever is on the other end of "the menu is wrong". The design is in `plan.md`, the vocabulary in `CONTEXT.md`, and the deployment in -`ansible-playbooks/menu-admin/readme.md`. +`../ansible/readme.md`. + +Each Site is one rootless Podman **Quadlet** unit on `emily` — a systemd *user* +service named `menu-` — behind the shared Caddy unit. There is no Docker in +this path. ## Where things are -| Thing | Where | -| --- | --- | -| The app | container `menu-app.` on `monoco`, image `registry.tanshu.com/menu-admin:` | -| Menu content | `/var/lib/menu./data` — `originals/`, `drafts/`, `published/` | -| Admin | `https:///admin` | -| The password | `/var/lib/menu./site.env` (mode 600, never in git) | -| Categories | `data/sections.json`, edited in the admin | +| Thing | Where | +| --------------- | ----------------------------------------------------------------------------------------- | +| The app | user unit `menu-.service` on `emily`, container `menu-app.`, image `registry.tanshu.com/menu-admin:` | +| Menu content | `~/podman-data/menu-/data` — `originals/`, `drafts/`, `published/` | +| Admin | `https:///admin` | +| The password | `~/podman-data/menu-/site.env` (mode 600, never in git) | +| Categories | `data/sections.json`, edited in the admin | | Brand and theme | `config/.site.json` in this repo, baked into the image (the palette follows the Stitch system in `docs/design/stitch/`) | +Everything runs as the user `tanshu`, so every command below is run on `emily` as +that user. `sudo` is never needed, and never works for these units. + ## Everyday tasks **Publish a new menu.** Sign in at `/admin`, find the Section, fill in "New Menu @@ -30,8 +37,8 @@ open the set you want and press "Make current". Its URL does not change. which signs out anyone who was signed in. Nothing else changes. ```sh -ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com --force -ssh monoco docker restart menu-app.hopsngrains.com +ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains --force +ssh emily systemctl --user restart menu-hopsngrains ``` There is no minimum length and no password policy — the one rule is that the @@ -45,8 +52,8 @@ HASH=$(printf '%s' "$PW" | node -e "let s='';process.stdin.setEncoding('utf8');p SECRET=$(node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))") printf 'ADMIN_PASSWORD_HASH=%s SESSION_SECRET=%s -' "$HASH" "$SECRET" | ssh monoco "docker run --rm -i --user 1000:1000 -v /var/lib/menu.hopsngrains.com:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'" -ssh monoco docker restart menu-app.hopsngrains.com +' "$HASH" "$SECRET" | ssh emily "podman run --rm -i --userns=keep-id:uid=1000,gid=1000 -v ~/podman-data/menu-hopsngrains:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'" +ssh emily systemctl --user restart menu-hopsngrains ``` **Add, rename, reorder or remove a category.** In the admin, open the @@ -69,14 +76,15 @@ rebuild and push (`./deploy.sh `) and the playbook with the new tag. Original survives, the app can rebuild them: ```sh -docker run --rm --user 1000:1000 -e MENU_ROOT=/data \ +podman run --rm --userns=keep-id:uid=1000,gid=1000 -e MENU_ROOT=/data \ -e MENU_SITE_CONFIG=/app/config/hopsngrains.site.json \ - -v /var/lib/menu.hopsngrains.com:/data \ + -v ~/podman-data/menu-hopsngrains:/data \ --entrypoint node registry.tanshu.com/menu-admin: src/cli/repair.ts ``` `src/cli/verify.ts` does the same inventory without changing anything, and is -what the server refuses to boot on. +what the server refuses to boot on. Both need the `--userns=keep-id` mapping, or +they would see the files as a different user and fail to write. ## When something looks wrong @@ -84,15 +92,21 @@ what the server refuses to boot on. | --- | --- | | Landing page says the menus are being updated | no Section has a Current Menu Set — publish one in the admin | | A Section 404s | that Section has no Current set, or the URL is misspelled | -| Sign-in rejects the right password | `docker logs menu-app.`, then check `site.env` exists and is owned by the app user | -| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `docker restart` | +| Sign-in rejects the right password | `podman logs menu-app.`, then check `site.env` exists and is owned by `tanshu` | +| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `systemctl --user restart menu-` | | A photo uploads but looks wrong | the browser shrinks to 3000px before upload; that is expected, and the Original is kept | -| The container will not start | `docker logs` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust | +| The container will not start | `systemctl --user status menu-` and `podman logs menu-app.` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust | +| The unit is "not found" | its `.network` file is missing, so Quadlet refused to generate it: check `~/.config/containers/systemd/` | | Caddy reload failed during a deploy | nothing is down: Caddy validates before applying. Read the error in the playbook output | | Everything is slow | check the tag is current; a Viewing image is ~1600px, so a Section page should not be megabytes | -**Logs.** `docker logs -f menu-app.` — JSON lines, one per event, including -every upload, publish, restore and failed sign-in. +**Logs.** `podman logs -f menu-app.`, or `journalctl --user -u menu- +-f` for the unit. The app logs JSON lines, one per event, including every upload, +publish, restore and failed sign-in. + +**Restarting.** `systemctl --user restart menu-`. A *reload* of the Caddy +config is `podman exec caddy caddy reload --config /etc/caddy/Caddyfile`, which +does not drop the other sites. **Checking from outside.** No sign-in, no writes, exits non-zero on failure, so it can gate a cutover: @@ -104,133 +118,19 @@ npm run check-site -- https://menu.hopsngrains.com ## Backups The app is the only writer. The irreplaceable part is -`/var/lib/menu./data/originals` — the photos exactly as uploaded, from +`~/podman-data/menu-/data/originals` — the photos exactly as uploaded, from which everything else can be regenerated with `repair`. Back that directory up; `site.env` is worth keeping too (otherwise you re-provision the password). -## Rolling back - -Deploy the previous tag — `./deploy.sh ` — which rebuilds the Site at -that version. `docker logs menu-app.` is where a container that will not start -says why. - - -For whoever is on the other end of "the menu is wrong". The design is in -`plan.md`, the vocabulary in `CONTEXT.md`, and the deployment in -`ansible-playbooks/menu-admin/readme.md`. - -## Where things are - -| Thing | Where | -| --- | --- | -| The app | container `menu-app.` on `monoco`, image `registry.tanshu.com/menu-admin:` | -| Menu content | `/var/lib/menu./data` — `originals/`, `drafts/`, `published/` | -| Admin | `https:///admin` | -| The password | `/var/lib/menu./site.env` (mode 600, never in git) | -| Categories | `data/sections.json`, edited in the admin | -| Brand and theme | `config/.site.json` in this repo, baked into the image (the palette follows the Stitch system in `docs/design/stitch/`) | - -## Everyday tasks - -**Publish a new menu.** Sign in at `/admin`, find the Section, fill in "New Menu -Set" (label, optional valid-until date), Create Draft, then drop the photos in. -Drag to reorder — the order in the editor is the order customers see. Press -Preview to check it, then Publish. Publishing archives whatever was current. - -**Put an older menu back.** Open the Section's "older menus" fold in the admin, -open the set you want and press "Make current". Its URL does not change. - -**Change or reset the admin password.** Run the provision script again with -`--force`: it overwrites the password hash, and generates a fresh session secret, -which signs out anyone who was signed in. Nothing else changes. - -```sh -ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com --force -ssh monoco docker restart menu-app.hopsngrains.com -``` - -There is no minimum length and no password policy — the one rule is that the -password cannot be empty. It can also be set from a machine that has this repo -checked out, with the prompt on that machine rather than the server: - -```sh -cd ~/programming/menu-admin -read -r -s -p 'Admin password: ' PW; echo -HASH=$(printf '%s' "$PW" | node -e "let s='';process.stdin.setEncoding('utf8');process.stdin.on('data',c=>s+=c).on('end',async()=>{const m=await import('./src/auth.ts');process.stdout.write(await m.hashPassword(s))})") -SECRET=$(node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('base64url'))") -printf 'ADMIN_PASSWORD_HASH=%s -SESSION_SECRET=%s -' "$HASH" "$SECRET" | ssh monoco "docker run --rm -i --user 1000:1000 -v /var/lib/menu.hopsngrains.com:/data --entrypoint sh registry.tanshu.com/menu-admin:2026-09-22 -c 'umask 077 && cat > /data/site.env'" -ssh monoco docker restart menu-app.hopsngrains.com -``` - -**Add, rename, reorder or remove a category.** In the admin, open the -**Categories** fold at the bottom of the dashboard. A new category starts empty, -so customers do not see it until you publish a menu into it. Reordering there is -the order customers see. Two rules: a category's web address is fixed when it is -created (rename the name as often as you like — a printed QR code keeps working), -and a category can only be removed once it holds no menus. - -**Change the restaurant's name or tagline, the icon, or the published dates.** In -the admin, open the **Site identity** fold: the name and tagline customers see, -a checkbox for whether a menu says when it was published, and an SVG upload for -the icon (under 256 KB; the upload replaces the bundled mark and "Remove the -uploaded icon" brings it back). None of this changes your web addresses. - -**Change the colours.** Those are still `config/.site.json`, so they need a -rebuild and push (`./deploy.sh `) and the playbook with the new tag. - -**Fix a missing image.** If a Page's thumbnail or viewing image is lost while its -Original survives, the app can rebuild them: - -```sh -docker run --rm --user 1000:1000 -e MENU_ROOT=/data \ - -e MENU_SITE_CONFIG=/app/config/hopsngrains.site.json \ - -v /var/lib/menu.hopsngrains.com:/data \ - --entrypoint node registry.tanshu.com/menu-admin: src/cli/repair.ts -``` - -`src/cli/verify.ts` does the same inventory without changing anything, and is -what the server refuses to boot on. - -## When something looks wrong - -| Symptom | Look here | -| --- | --- | -| Landing page says the menus are being updated | no Section has a Current Menu Set — publish one in the admin | -| A Section 404s | that Section has no Current set, or the URL is misspelled | -| Sign-in rejects the right password | `docker logs menu-app.`, then check `site.env` exists and is owned by the app user | -| Admin is missing entirely | `site.env` was unreadable at boot, so the admin is disabled: run the provision script and `docker restart` | -| A photo uploads but looks wrong | the browser shrinks to 3000px before upload; that is expected, and the Original is kept | -| The container will not start | `docker logs` — a fatal integrity problem makes it refuse to serve a data directory it cannot trust | -| Caddy reload failed during a deploy | nothing is down: Caddy validates before applying. Read the error in the playbook output | -| Everything is slow | check the tag is current; a Viewing image is ~1600px, so a Section page should not be megabytes | - -**Logs.** `docker logs -f menu-app.` — JSON lines, one per event, including -every upload, publish, restore and failed sign-in. - -**Checking from outside.** No sign-in, no writes, exits non-zero on failure, so -it can gate a cutover: - -```sh -npm run check-site -- https://menu.hopsngrains.com -``` - -## Backups - -The app is the only writer. The irreplaceable part is -`/var/lib/menu./data/originals` — the photos exactly as uploaded, from -which everything else can be regenerated with `repair`. Back that directory up; -`site.env` is worth keeping too (otherwise you re-provision the password). +Because of `keep-id`, everything there is owned by `tanshu`, so a plain +`tar`/`rsync` of `~/podman-data/menu-` is complete — no subuid-owned files +to trip over and no `podman unshare` needed. ## Rolling back -The old nginx Site is kept, not deleted. To go back: +Deploy the previous tag — `./deploy.sh ` — which rebuilds the Site +at that version. `podman logs menu-app.` is where a container that will not +start says why. -```sh -docker start nginx-menu.hopsngrains.com -``` - -then point Caddy back at it — the cutover section of the playbook readme has the -exact invocation, including the flag that stops the app's snippet from being -re-added. +The Docker-era container was left on `monoco`, stopped and not removed, so the +last resort is to start it again and point Caddy back at it. diff --git a/scripts/copy-site-data.sh b/scripts/copy-site-data.sh new file mode 100755 index 0000000..a4f5fd2 --- /dev/null +++ b/scripts/copy-site-data.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# +# Copies one Site's data from the Docker deployment on monoco to the rootless +# Podman deployment on emily. +# +# scripts/copy-site-data.sh hopsngrains +# scripts/copy-site-data.sh hopsngrains --live # copy while the app keeps serving +# +# Run it from a machine that can ssh to both hosts (the ssh aliases monoco and +# emily, overridable with MONOCO= and EMILY=). This is the one-off part of the +# move: the playbook deploys the container, this hands it the content. +# +# By default it stops the old container first, so the copy is of quiesced data, +# and starts it again afterwards — monoco keeps serving until DNS moves. Use +# --live to skip the stop; the app writes its manifests atomically and its images +# are immutable once written, so a live copy is usually clean, but an upload +# landing mid-copy is missed. Nothing on monoco is removed either way. +# +# Two details are load-bearing: +# +# * monoco cannot ssh to emily, so this machine is the relay — the tar streams +# through it and nothing large is written locally. +# * sudo on monoco, because /var/lib/menu. belongs to uid 1000 (the +# ubuntu account) while the ssh user there is 1001. The extraction runs as +# tanshu on emily, which is the uid keep-id maps the app onto, so the files +# land owned by the right user rather than by root or a subuid. +set -euo pipefail + +usage() { + cat <<'USAGE' +usage: copy-site-data.sh [--live] + + inventory name from ansible/hosts: hopsngrains, greatbear, hngmohali + --live copy without stopping the old container (not quiesced) + +Environment overrides: + MONOCO=host ssh target holding the Docker deployment (default: monoco) + EMILY=host ssh target holding the rootless Podman deployment (default: emily) + SRC_ROOT=dir absolute data root on monoco (default: /var/lib) + DEST_ROOT=dir absolute data root on emily (default: /podman-data, + resolved over ssh — ~ and $HOME are not expanded for you) +USAGE +} + +SITE="${1:-}" +if [[ -z "$SITE" || "$SITE" == "-h" || "$SITE" == "--help" ]]; then + usage + [[ -n "$SITE" ]] || exit 1 + exit 0 +fi +shift + +LIVE=0 +for argument in "$@"; do + case "$argument" in + --live) LIVE=1 ;; + *) echo "unknown argument: $argument" >&2; usage >&2; exit 1 ;; + esac +done + +cd "$(dirname "${BASH_SOURCE[0]}")/.." + +# The repo is the source of truth for which Sites exist and what they answer. +VARS="ansible/vars/$SITE.yml" +if ! grep -qE "^$SITE[[:space:]]" ansible/hosts || [[ ! -f "$VARS" ]]; then + echo "'$SITE' is not a Site in ansible/hosts" >&2 + exit 1 +fi +TITLE="$(sed -n 's/^title:[[:space:]]*//p' "$VARS" | head -n 1)" +HOST="menu.$TITLE" +CONTAINER="menu-app.$HOST" + +MONOCO="${MONOCO:-monoco}" +EMILY="${EMILY:-emily}" +SRC_ROOT="${SRC_ROOT:-/var/lib}" + +for tool in ssh tar; do + command -v "$tool" >/dev/null 2>&1 || { echo "$tool is required" >&2; exit 1; } +done + +# Resolved rather than left as a literal ~ or $HOME, so every path below is +# absolute and can be quoted safely on the remote side. +if [[ -z "${DEST_ROOT:-}" ]]; then + DEST_ROOT="$(ssh -n "$EMILY" 'printf %s "$HOME"')/podman-data" +fi + +SRC="$SRC_ROOT/$HOST" +DEST="$DEST_ROOT/menu-$SITE" + +echo "==> $SITE: $MONOCO:$SRC -> $EMILY:$DEST" + +# Start the old container again however this exits, so a failed copy never +# leaves the live Site down. +restart_needed=0 +restart_old() { + if [[ "$restart_needed" -eq 1 ]]; then + echo "==> starting $CONTAINER again on $MONOCO" + ssh -n "$MONOCO" docker start "$CONTAINER" >/dev/null || true + restart_needed=0 + fi +} +trap restart_old EXIT + +# -n on every monoco call: ssh would otherwise eat this script's stdin, and the +# copy below is a pipeline whose input matters. +if [[ "$LIVE" -eq 0 ]]; then + echo "==> stopping $CONTAINER on $MONOCO" + ssh -n "$MONOCO" docker stop "$CONTAINER" >/dev/null + restart_needed=1 +fi + +echo "==> copying" +ssh -n "$MONOCO" "sudo -n tar -C '$SRC' -cf - ." \ + | ssh "$EMILY" "mkdir -p '$DEST' && tar -C '$DEST' -xf -" + +restart_old + +echo "==> verifying" +src_sum="$(ssh -n "$MONOCO" "sudo -n sh -c 'cd $SRC && find . -type f -exec sha256sum {} +' | sort | sha256sum" | awk '{print $1}')" +dst_sum="$(ssh -n "$EMILY" "cd '$DEST' && find . -type f -exec sha256sum {} + | sort | sha256sum" | awk '{print $1}')" + +if [[ -z "$src_sum" || "$src_sum" != "$dst_sum" ]]; then + echo "the copy does not match the source: $src_sum != $dst_sum" >&2 + exit 1 +fi + +user="$(id -un)" +strays="$(ssh -n "$EMILY" "find '$DEST' ! -user '$user'" | head -n 5)" +if [[ -n "$strays" ]]; then + echo "the copy holds files $user does not own, so keep-id will not see them:" >&2 + echo "$strays" >&2 + exit 1 +fi + +echo +echo "ok ${dst_sum:0:12} $(ssh -n "$EMILY" "du -sb '$DEST'" | awk '{print $1}') bytes, all owned by $user" +echo +echo "next:" +echo " cd ansible && ansible-playbook --inventory=hosts playbook.yml --limit $SITE -e tag=" +echo " then check the Site on a Host: override before moving DNS to $EMILY" diff --git a/scripts/provision-site-env.sh b/scripts/provision-site-env.sh index f6ff4fe..17dac89 100755 --- a/scripts/provision-site-env.sh +++ b/scripts/provision-site-env.sh @@ -1,9 +1,10 @@ #!/usr/bin/env bash # # Writes the one file that never goes in git: the admin password hash and the -# session secret. Run this on the server, once per Site. +# session secret. Run this on the server, once per Site, as the user that owns +# the rootless Podman units — tanshu: # -# scripts/provision-site-env.sh /var/lib/menu-app.hopsngrains.com +# scripts/provision-site-env.sh ~/podman-data/menu-hopsngrains # # The hash is produced by the app image itself, so the server needs no Node and # the password is never visible in the process list. @@ -14,8 +15,8 @@ usage() { usage: provision-site-env.sh [image] [--force] directory the app mounts as MENU_ROOT, e.g. - /var/lib/menu-app.hopsngrains.com - [image] app image to borrow Node from (default menu-admin:latest) + ~/podman-data/menu-hopsngrains + [image] app image to borrow Node from (default: the running app's) --force overwrite an existing site.env USAGE } @@ -43,19 +44,17 @@ if [[ -f "$ENV_FILE" && "$FORCE" -ne 1 ]]; then exit 1 fi -if ! command -v docker >/dev/null 2>&1; then - echo "docker is required" >&2 +if ! command -v podman >/dev/null 2>&1; then + echo "podman is required" >&2 exit 1 fi -# The image is only needed to borrow node for hashing, so take the one the app -# is already running rather than guessing a name. The playbook names the -# container menu-app. for the web address menu.. +# The image is only needed to borrow node for hashing, so take the one an app +# container is already running rather than guessing a tag. The playbook names +# every Site's container menu-app.. All Sites share the image, so which +# one answers does not matter. if [[ -z "$IMAGE" ]]; then - for candidate in "menu-app.${DATA_ROOT##*/menu.}" "menu-app.${DATA_ROOT##*/}"; do - IMAGE="$(docker inspect --format '{{.Config.Image}}' "$candidate" 2>/dev/null || true)" - [[ -n "$IMAGE" ]] && break - done + IMAGE="$(podman ps --filter 'name=^menu-app\.' --format '{{.Image}}' | head -n 1 || true)" fi if [[ -z "$IMAGE" ]]; then @@ -64,7 +63,7 @@ if [[ -z "$IMAGE" ]]; then exit 1 fi -if ! docker image inspect "$IMAGE" >/dev/null 2>&1; then +if ! podman image inspect "$IMAGE" >/dev/null 2>&1; then echo "image $IMAGE is not present on this host." >&2 echo "Pass the one the app runs: $0 $DATA_ROOT registry.tanshu.com/menu-admin:" >&2 exit 1 @@ -96,11 +95,11 @@ if [[ "$PASSWORD" != "$REPEAT" ]]; then exit 1 fi -HASH="$(printf '%s' "$PASSWORD" | docker run --rm -i --entrypoint node "$IMAGE" \ +HASH="$(printf '%s' "$PASSWORD" | podman run --rm -i --entrypoint node "$IMAGE" \ --input-type=module \ -e "let s='';process.stdin.setEncoding('utf8');process.stdin.on('data',c=>s+=c);process.stdin.on('end',async()=>{const {pathToFileURL}=await import('node:url');const m=await import(pathToFileURL('/app/src/auth.ts').href);process.stdout.write(await m.hashPassword(s))})")" -SECRET="$(docker run --rm --entrypoint node "$IMAGE" --input-type=module \ +SECRET="$(podman run --rm --entrypoint node "$IMAGE" --input-type=module \ -e "import {randomBytes} from 'node:crypto';process.stdout.write(randomBytes(32).toString('base64url'))")" if [[ -z "$HASH" || -z "$SECRET" ]]; then @@ -109,10 +108,14 @@ if [[ -z "$HASH" || -z "$SECRET" ]]; then fi # The file is written by a container running as the app's own uid, not by the -# SSH user: the data root belongs to the app user (0750), so the SSH user -# usually cannot write into it, and a file they did write would be unreadable by -# the app. This way the ownership is right the first time and there is no -# "re-run the playbook afterwards" step. +# SSH user, so its ownership is right the first time and there is no "re-run the +# playbook afterwards" step. +# +# Rootless Podman maps container uid 1000 onto a subuid by default, which would +# produce a file the host user cannot read and a tar backup would drop. keep-id +# maps it onto this script's user instead — the same mapping the app's Quadlet +# unit uses (UserNS=keep-id in roles/menu-app), which is what keeps both sides +# agreeing on who owns site.env. if [[ ! -d "$DATA_ROOT" ]]; then echo "$DATA_ROOT does not exist; run the menu-admin playbook first" >&2 exit 1 @@ -120,18 +123,20 @@ fi APP_UID="${APP_UID:-1000}" APP_GID="${APP_GID:-1000}" +USERNS="keep-id:uid=$APP_UID,gid=$APP_GID" -if ! printf 'ADMIN_PASSWORD_HASH=%s\nSESSION_SECRET=%s\n' "$HASH" "$SECRET" | docker run --rm -i \ - --user "$APP_UID:$APP_GID" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \ +if ! printf 'ADMIN_PASSWORD_HASH=%s\nSESSION_SECRET=%s\n' "$HASH" "$SECRET" | podman run --rm -i \ + --userns="$USERNS" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \ -c 'umask 077 && cat > /data/site.env'; then - echo "could not write $ENV_FILE as $APP_UID:$APP_GID" >&2 + echo "could not write $ENV_FILE as the app's uid" >&2 exit 1 fi unset PASSWORD REPEAT HASH SECRET -docker run --rm --user "$APP_UID:$APP_GID" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \ +podman run --rm --userns="$USERNS" -v "$DATA_ROOT:/data" --entrypoint sh "$IMAGE" \ -c 'ls -l /data/site.env' -echo "wrote $ENV_FILE, owned by $APP_UID:$APP_GID" -echo "restart the app container to pick it up; changing the password later means running this again with --force" +echo "wrote $ENV_FILE, owned by $(id -un)" +echo "restart the app to pick it up: systemctl --user restart menu-" +echo "changing the password later means running this again with --force"