Run the Sites as rootless Podman Quadlets, and build with podman

The fleet on emily is rootless Podman with Quadlet units under
~/.config/containers/systemd and state in ~/podman-data, all behind one shared
Caddy unit. This brings the menus in line instead of leaving them the last
Docker deployment. The build moves with them: podman reads a Containerfile as
readily as a Dockerfile, so the name now says which toolchain is real.

The app is one .container and one .network file. Two details carry it.
UserNS=keep-id maps the image's own node account onto the user that owns the
units, so data/ and site.env stay owned by tanshu: plain tar backups, no
subuids. provision-site-env.sh writes site.env with the same mapping rather
than --user, which under rootless Podman would have produced a subuid-owned
file the app could not read. And the Site registers its own network with Caddy
by writing a drop-in beside caddy.container, so Caddy never lists its upstreams
and never changes when a Site is added. The unit keeps the Docker-era container
name for the scripts and muscle memory that look it up, while the Caddyfile
proxies to a dotless NetworkAlias, because a container name with dots in it is
a liability in DNS.

The unit keeps podman's Pull=missing, so a restart or a reboot never depends on
the registry being reachable, while the play pulls explicitly at deploy time.
That is also where a missing `podman login` for the private registry now fails,
with the instruction, rather than as a unit that will not start.

copy-site-data.sh moves a Site's data across, and is not a plain rsync:
/var/lib/menu.* on monoco belongs to uid 1000 while the ssh user there is 1001,
so the read needs sudo; monoco cannot ssh to emily, so the workstation relays
the tar; and the extraction runs as tanshu on emily, which is what makes the
files land owned by the user keep-id maps the app onto. It proves the copy by
comparing a SHA-256 of the whole tree against the source and checking every
file's owner.

Not yet done: the copies themselves and the DNS move, which need a human.

ansible/roles/menu-app/tasks/main.yaml

@@ -1,134 +1,178 @@
 ---
-- name: Pull the app image
-  docker_image:
-    name: '{{ image_name }}'
-    tag: '{{ image_tag }}'
-    source: pull
-    force_source: yes
+# The menu app as a rootless Podman Quadlet unit.
+#
+# Everything here runs as {{ podman_user }} — the connection user. No become, no
+# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
+# user manager is reachable through the standard environment.
 
+# The pull is explicit, rather than left to the Quadlet unit, for two reasons.
+# The image lives in a private registry, so a failure here is where a missing
+# `podman login` should show up — not two tasks later as "unit not found". And
+# the unit keeps podman's default `Pull=missing`, so it never reaches for the
+# registry on a restart or after a reboot: that is what lets a box come back up
+# when the registry is down. Pulling here is what keeps a redeployed tag current.
+- name: Pull the app image
+  ansible.builtin.command: 'podman pull {{ image_name }}:{{ image_tag }}'
+  register: menu_pull
+  changed_when: false
+  failed_when: false
+
+- name: Fail when the image cannot be pulled
+  ansible.builtin.fail:
+    msg: >-
+      podman could not pull {{ image_name }}:{{ image_tag }}.
+      {{ menu_pull.stderr | default('') | trim }}
+      The registry is private: run `podman login registry.tanshu.com` once as
+      {{ podman_user }} on this host.
+  when: menu_pull.rc != 0
+
+# The data directory is created here rather than left to the image. Under
+# keep-id the image's own root phase maps to a subuid, so anything it tries to
+# create under the mount fails with EPERM; the directory has to exist already,
+# owned by {{ podman_user }}.
 - name: Ensure the app root exists
-  file:
+  ansible.builtin.file:
     path: '{{ app_root }}'
     state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
     mode: '0750'
 
+# MENU_ROOT is the app root, and the app keeps its content one level down in
+# data/ (originals/, drafts/, published/, site.json, sections.json).
 - name: Ensure the data directory exists
-  file:
+  ansible.builtin.file:
     path: '{{ app_root }}/data'
     state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
     mode: '0750'
 
 - name: Look for the untracked site.env
-  stat:
+  ansible.builtin.stat:
     path: '{{ app_root }}/site.env'
   register: menu_site_env
 
 - name: Warn when the admin has no password yet
-  debug:
+  ansible.builtin.debug:
     msg: >-
       {{ app_root }}/site.env is missing, so the admin is disabled: the public
-      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on
-      this host, then re-run this playbook.
+      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on this
+      host, then re-run this playbook.
   when: not menu_site_env.stat.exists
 
-- name: Run the menu app
-  docker_container:
-    name: '{{ docker_container }}'
-    image: '{{ image_name }}:{{ image_tag }}'
-    state: started
-    restart_policy: unless-stopped
-    user: '{{ app_uid }}:{{ app_gid }}'
-    # Force a rebuild of the container with -e recreate_container=true: that is
-    # what applies a changed healthcheck or env to an existing container, since
-    # the module otherwise leaves a running one alone.
-    recreate: '{{ recreate_container | default(false) | bool }}'
-    # Declared here rather than relied on from the image: podman builds an OCI
-    # image by default and drops Docker's HEALTHCHECK instruction, so the
-    # Dockerfile's own healthcheck never reaches the container. node is in the
-    # image and the exec form needs no shell quoting.
-    healthcheck:
-      test:
-        - CMD
-        - node
-        - -e
-        - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-      interval: 30s
-      timeout: 5s
-      start_period: 30s
-      retries: 3
-    env:
-      MENU_ROOT: /data
-      MENU_SITE_CONFIG: '/app/config/{{ site }}.site.json'
-      SECURE_COOKIES: 'true'
-      LOG_LEVEL: info
-      PORT: '3000'
-    volumes:
-      - '{{ app_root }}:/data'
-    networks:
-      - name: '{{ docker_network }}'
+- name: Install the menu app Quadlet unit
+  ansible.builtin.template:
+    src: menu-app.container.j2
+    dest: '{{ quadlet_dir }}/{{ network_name }}.container'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+  notify:
+    - Reload systemd user units
+    - Restart the menu app
 
-- name: Keep site.env readable only by the app user
-  file:
-    path: '{{ app_root }}/site.env'
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
-    mode: '0600'
-  when: menu_site_env.stat.exists
+- name: Start the menu app
+  block:
+    - name: Start the menu app
+      ansible.builtin.systemd_service:
+        name: '{{ unit_name }}'
+        scope: user
+        state: started
+        daemon_reload: true
+  rescue:
+    # systemd reports "control process exited with error code" and nothing else,
+    # which is useless for the usual failures (a missing network unit, an
+    # unwritable /data, a bad uid mapping). Pull the real reason out.
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_diagnostics
+      changed_when: false
+      failed_when: false
 
-# Deliberately not the image's HEALTHCHECK: podman builds an OCI image by
-# default and drops the Docker HEALTHCHECK instruction, so State.Health is absent
-# and a wait on it fails on a perfectly healthy container — which stops the play
-# before Caddy is ever pointed at the app.
-- name: Wait for the app to answer inside its own container
-  ansible.builtin.command:
-    argv:
-      - docker
-      - exec
-      - '{{ docker_container }}'
-      - node
-      - -e
-      - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-  register: menu_health
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: 'The menu app did not start — see the diagnostics above.'
+
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+- name: Verify the menu app is running
+  ansible.builtin.command: 'systemctl --user is-active {{ unit_name }}'
+  register: menu_active
   changed_when: false
-  retries: 20
-  delay: 3
+  failed_when: menu_active.stdout != 'active'
+
+# `is-active` says the container is running; it does not say the app can serve.
+# Restart=always means even a crash-looping container is reported active — which
+# is exactly what a fatal integrity problem in the data directory produces. The
+# unit declares the health check, so run that once, synchronously.
+- name: Wait for the app to pass its health check
+  ansible.builtin.command: 'podman healthcheck run {{ container_name }}'
+  register: menu_health
   until: menu_health.rc == 0
+  retries: 24
+  delay: 5
+  changed_when: false
   failed_when: false
 
-- name: Fail when the app never answers
-  fail:
-    msg: >-
-      {{ docker_container }} did not answer /healthz within a minute. Its logs
-      say why: docker logs {{ docker_container }}
+- name: Report a container that never answered
   when: menu_health.rc != 0
+  block:
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_health_diagnostics
+      changed_when: false
+      failed_when: false
+
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_health_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: >-
+          {{ container_name }} did not pass its health check within two minutes.
+          Its logs are above, and: podman logs {{ container_name }}
+
+- name: Keep site.env readable only by the app user
+  ansible.builtin.file:
+    path: '{{ app_root }}/site.env'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0600'
+  when: menu_site_env.stat.exists
 
 - name: Check the data directory for integrity problems
   # Reads the same directory the container writes, as the same user, so a
   # permissions mistake shows up here rather than at 8pm service.
-  command:
+  ansible.builtin.command:
     argv:
-      - docker
-      - run
-      - --rm
-      - --user
-      - '{{ app_uid }}:{{ app_gid }}'
-      - -e
+      - podman
+      - exec
+      - --env
       - MENU_ROOT=/data
-      - -e
+      - --env
       - 'MENU_SITE_CONFIG=/app/config/{{ site }}.site.json'
-      - --entrypoint
+      - '{{ container_name }}'
       - node
-      - '{{ image_name }}:{{ image_tag }}'
       - src/cli/verify.ts
   register: menu_verify
   changed_when: false
   failed_when: false
 
 - name: Report integrity problems
-  debug:
+  ansible.builtin.debug:
     msg: '{{ menu_verify.stdout_lines | default([]) }}'
   when: menu_verify.rc != 0