Run the Sites as rootless Podman Quadlets, and build with podman

The fleet on emily is rootless Podman with Quadlet units under
~/.config/containers/systemd and state in ~/podman-data, all behind one shared
Caddy unit. This brings the menus in line instead of leaving them the last
Docker deployment. The build moves with them: podman reads a Containerfile as
readily as a Dockerfile, so the name now says which toolchain is real.

The app is one .container and one .network file. Two details carry it.
UserNS=keep-id maps the image's own node account onto the user that owns the
units, so data/ and site.env stay owned by tanshu: plain tar backups, no
subuids. provision-site-env.sh writes site.env with the same mapping rather
than --user, which under rootless Podman would have produced a subuid-owned
file the app could not read. And the Site registers its own network with Caddy
by writing a drop-in beside caddy.container, so Caddy never lists its upstreams
and never changes when a Site is added. The unit keeps the Docker-era container
name for the scripts and muscle memory that look it up, while the Caddyfile
proxies to a dotless NetworkAlias, because a container name with dots in it is
a liability in DNS.

The unit keeps podman's Pull=missing, so a restart or a reboot never depends on
the registry being reachable, while the play pulls explicitly at deploy time.
That is also where a missing `podman login` for the private registry now fails,
with the instruction, rather than as a unit that will not start.

copy-site-data.sh moves a Site's data across, and is not a plain rsync:
/var/lib/menu.* on monoco belongs to uid 1000 while the ssh user there is 1001,
so the read needs sudo; monoco cannot ssh to emily, so the workstation relays
the tar; and the extraction runs as tanshu on emily, which is what makes the
files land owned by the user keep-id maps the app onto. It proves the copy by
comparing a SHA-256 of the whole tree against the source and checking every
file's owner.

Not yet done: the copies themselves and the DNS move, which need a human.

ansible/roles/caddy/handlers/main.yaml

@@ -1,6 +1,31 @@
 ---
-- name: Reload Caddy
-  # Caddy validates before it applies, so a mistake here leaves the running
-  # config untouched and fails the play loudly.
-  command: 'docker exec -w /etc/caddy {{ caddy_container }} caddy reload'
-  listen: Reload Caddy
+# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
+# Quadlet unit there. These handlers run as the connection user, so
+# `systemctl --user` and `podman exec` work without any XDG_RUNTIME_DIR juggling.
+
+# Notified by this role when the Caddyfile changes. A reload keeps existing
+# connections; a restart would briefly drop every hosted site.
+#
+# Note the config path: /etc/caddy/Caddyfile is the mount inside the container,
+# not {{ caddyfile_path }}, which is the host side of it.
+- name: Reload Caddy configuration
+  ansible.builtin.command: >-
+    podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
+  changed_when: true
+  listen: 'Reload Caddy'
+
+# Notified by roles/register when this Site's network unit, or the drop-in that
+# attaches Caddy to it, is written. Quadlet has no live equivalent of
+# `docker network connect`, so the container has to be recreated — a brief
+# restart of every hosted site, not just this one. See the Caddy playbook's
+# readme, "Trade-off worth knowing".
+#
+# Defined after the reload above so that when both are notified the cheap config
+# reload happens first.
+- name: Restart Caddy
+  ansible.builtin.systemd_service:
+    name: '{{ caddy_name }}.service'
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Caddy

ansible/roles/caddy/tasks/main.yaml

@@ -1,14 +1,29 @@
 ---
+# Append this Site's block to the shared Caddyfile.
+#
+# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
+# Quadlet unit there, so this role does not manage the container — it only points
+# it at the app. The Caddyfile is a bind mount into that container, so Caddy sees
+# a written change as soon as the file lands; the handler only asks Caddy to
+# re-read it. That is a reload, not a restart, so the other hosted sites are
+# untouched.
+#
+# blockinfile is idempotent on its own via the marker, so there is no need to
+# slurp the file and compare first — which also matters because slurp *fails*
+# when the Caddyfile does not exist yet, on a host where the Caddy playbook has
+# not run.
+
 - name: Check the Caddyfile exists
-  stat:
+  ansible.builtin.stat:
     path: '{{ caddyfile_path }}'
   register: menu_caddyfile
 
 - name: Fail when Caddy is not where we expect it
-  fail:
+  ansible.builtin.fail:
     msg: >-
-      {{ caddyfile_path }} is missing. Deploy the caddy playbook, which owns the
-      Caddy container and its config, before this one.
+      {{ caddyfile_path }} is missing. Deploy the Caddy playbook
+      (ansible-playbooks/caddy/playbook.yml), which owns the Caddy unit and its
+      config, before this one.
   when: not menu_caddyfile.stat.exists
 
 # The Site's web address may still be claimed by an older snippet: the retired
@@ -16,7 +31,7 @@
 # Caddy refuse the whole config, and a refused reload leaves the old proxy in
 # place, which is a 502 once the old container is gone.
 - name: Drop any older snippet claiming this web address
-  blockinfile:
+  ansible.builtin.blockinfile:
     path: '{{ caddyfile_path }}'
     marker: '# {mark} Ansible managed Caddy snippet for {{ host }}'
     state: absent
@@ -25,9 +40,12 @@
 # blockinfile replaces whatever sits between its markers, so redeploying updates
 # the snippet in place rather than appending a second one.
 - name: Point the Site's web address at the app
-  blockinfile:
+  ansible.builtin.blockinfile:
     path: '{{ caddyfile_path }}'
     marker: '# {mark} ANSIBLE MANAGED: menu app ({{ site }})'
     block: '{{ caddy_snippet }}'
-    create: no
+    create: false
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
   notify: Reload Caddy

ansible/roles/menu-app/handlers/main.yaml

@@ -0,0 +1,19 @@
+---
+# All handlers run as the connection user ({{ podman_user }}), so systemctl
+# --user works without setting XDG_RUNTIME_DIR.
+
+- name: Reload systemd user units
+  ansible.builtin.command: systemctl --user daemon-reload
+  changed_when: true
+  listen: Reload systemd user units
+
+# A changed unit file changes the generated unit, so the app is recreated. All of
+# its state is on the bind mount, so a recreate loses nothing — but it is a few
+# seconds of downtime for the one Site.
+- name: Restart the menu app
+  ansible.builtin.systemd_service:
+    name: '{{ unit_name }}'
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart the menu app

ansible/roles/menu-app/tasks/main.yaml

@@ -1,134 +1,178 @@
 ---
-- name: Pull the app image
-  docker_image:
-    name: '{{ image_name }}'
-    tag: '{{ image_tag }}'
-    source: pull
-    force_source: yes
+# The menu app as a rootless Podman Quadlet unit.
+#
+# Everything here runs as {{ podman_user }} — the connection user. No become, no
+# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
+# user manager is reachable through the standard environment.
 
+# The pull is explicit, rather than left to the Quadlet unit, for two reasons.
+# The image lives in a private registry, so a failure here is where a missing
+# `podman login` should show up — not two tasks later as "unit not found". And
+# the unit keeps podman's default `Pull=missing`, so it never reaches for the
+# registry on a restart or after a reboot: that is what lets a box come back up
+# when the registry is down. Pulling here is what keeps a redeployed tag current.
+- name: Pull the app image
+  ansible.builtin.command: 'podman pull {{ image_name }}:{{ image_tag }}'
+  register: menu_pull
+  changed_when: false
+  failed_when: false
+
+- name: Fail when the image cannot be pulled
+  ansible.builtin.fail:
+    msg: >-
+      podman could not pull {{ image_name }}:{{ image_tag }}.
+      {{ menu_pull.stderr | default('') | trim }}
+      The registry is private: run `podman login registry.tanshu.com` once as
+      {{ podman_user }} on this host.
+  when: menu_pull.rc != 0
+
+# The data directory is created here rather than left to the image. Under
+# keep-id the image's own root phase maps to a subuid, so anything it tries to
+# create under the mount fails with EPERM; the directory has to exist already,
+# owned by {{ podman_user }}.
 - name: Ensure the app root exists
-  file:
+  ansible.builtin.file:
     path: '{{ app_root }}'
     state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
     mode: '0750'
 
+# MENU_ROOT is the app root, and the app keeps its content one level down in
+# data/ (originals/, drafts/, published/, site.json, sections.json).
 - name: Ensure the data directory exists
-  file:
+  ansible.builtin.file:
     path: '{{ app_root }}/data'
     state: directory
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
     mode: '0750'
 
 - name: Look for the untracked site.env
-  stat:
+  ansible.builtin.stat:
     path: '{{ app_root }}/site.env'
   register: menu_site_env
 
 - name: Warn when the admin has no password yet
-  debug:
+  ansible.builtin.debug:
     msg: >-
       {{ app_root }}/site.env is missing, so the admin is disabled: the public
-      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on
-      this host, then re-run this playbook.
+      menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on this
+      host, then re-run this playbook.
   when: not menu_site_env.stat.exists
 
-- name: Run the menu app
-  docker_container:
-    name: '{{ docker_container }}'
-    image: '{{ image_name }}:{{ image_tag }}'
-    state: started
-    restart_policy: unless-stopped
-    user: '{{ app_uid }}:{{ app_gid }}'
-    # Force a rebuild of the container with -e recreate_container=true: that is
-    # what applies a changed healthcheck or env to an existing container, since
-    # the module otherwise leaves a running one alone.
-    recreate: '{{ recreate_container | default(false) | bool }}'
-    # Declared here rather than relied on from the image: podman builds an OCI
-    # image by default and drops Docker's HEALTHCHECK instruction, so the
-    # Dockerfile's own healthcheck never reaches the container. node is in the
-    # image and the exec form needs no shell quoting.
-    healthcheck:
-      test:
-        - CMD
-        - node
-        - -e
-        - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-      interval: 30s
-      timeout: 5s
-      start_period: 30s
-      retries: 3
-    env:
-      MENU_ROOT: /data
-      MENU_SITE_CONFIG: '/app/config/{{ site }}.site.json'
-      SECURE_COOKIES: 'true'
-      LOG_LEVEL: info
-      PORT: '3000'
-    volumes:
-      - '{{ app_root }}:/data'
-    networks:
-      - name: '{{ docker_network }}'
+- name: Install the menu app Quadlet unit
+  ansible.builtin.template:
+    src: menu-app.container.j2
+    dest: '{{ quadlet_dir }}/{{ network_name }}.container'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+  notify:
+    - Reload systemd user units
+    - Restart the menu app
 
-- name: Keep site.env readable only by the app user
-  file:
-    path: '{{ app_root }}/site.env'
-    owner: '{{ app_uid }}'
-    group: '{{ app_gid }}'
-    mode: '0600'
-  when: menu_site_env.stat.exists
+- name: Start the menu app
+  block:
+    - name: Start the menu app
+      ansible.builtin.systemd_service:
+        name: '{{ unit_name }}'
+        scope: user
+        state: started
+        daemon_reload: true
+  rescue:
+    # systemd reports "control process exited with error code" and nothing else,
+    # which is useless for the usual failures (a missing network unit, an
+    # unwritable /data, a bad uid mapping). Pull the real reason out.
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_diagnostics
+      changed_when: false
+      failed_when: false
 
-# Deliberately not the image's HEALTHCHECK: podman builds an OCI image by
-# default and drops the Docker HEALTHCHECK instruction, so State.Health is absent
-# and a wait on it fails on a perfectly healthy container — which stops the play
-# before Caddy is ever pointed at the app.
-- name: Wait for the app to answer inside its own container
-  ansible.builtin.command:
-    argv:
-      - docker
-      - exec
-      - '{{ docker_container }}'
-      - node
-      - -e
-      - "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
-  register: menu_health
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: 'The menu app did not start — see the diagnostics above.'
+
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+- name: Verify the menu app is running
+  ansible.builtin.command: 'systemctl --user is-active {{ unit_name }}'
+  register: menu_active
   changed_when: false
-  retries: 20
-  delay: 3
+  failed_when: menu_active.stdout != 'active'
+
+# `is-active` says the container is running; it does not say the app can serve.
+# Restart=always means even a crash-looping container is reported active — which
+# is exactly what a fatal integrity problem in the data directory produces. The
+# unit declares the health check, so run that once, synchronously.
+- name: Wait for the app to pass its health check
+  ansible.builtin.command: 'podman healthcheck run {{ container_name }}'
+  register: menu_health
   until: menu_health.rc == 0
+  retries: 24
+  delay: 5
+  changed_when: false
   failed_when: false
 
-- name: Fail when the app never answers
-  fail:
-    msg: >-
-      {{ docker_container }} did not answer /healthz within a minute. Its logs
-      say why: docker logs {{ docker_container }}
+- name: Report a container that never answered
   when: menu_health.rc != 0
+  block:
+    - name: Collect diagnostics
+      ansible.builtin.command: '{{ item }}'
+      loop:
+        - 'systemctl --user status {{ unit_name }} --no-pager'
+        - 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
+        - 'podman logs --tail 40 {{ container_name }}'
+      register: menu_health_diagnostics
+      changed_when: false
+      failed_when: false
+
+    - name: Show diagnostics
+      ansible.builtin.debug:
+        msg: '{{ menu_health_diagnostics.results | map(attribute="stdout") | list }}'
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: >-
+          {{ container_name }} did not pass its health check within two minutes.
+          Its logs are above, and: podman logs {{ container_name }}
+
+- name: Keep site.env readable only by the app user
+  ansible.builtin.file:
+    path: '{{ app_root }}/site.env'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0600'
+  when: menu_site_env.stat.exists
 
 - name: Check the data directory for integrity problems
   # Reads the same directory the container writes, as the same user, so a
   # permissions mistake shows up here rather than at 8pm service.
-  command:
+  ansible.builtin.command:
     argv:
-      - docker
-      - run
-      - --rm
-      - --user
-      - '{{ app_uid }}:{{ app_gid }}'
-      - -e
+      - podman
+      - exec
+      - --env
       - MENU_ROOT=/data
-      - -e
+      - --env
       - 'MENU_SITE_CONFIG=/app/config/{{ site }}.site.json'
-      - --entrypoint
+      - '{{ container_name }}'
       - node
-      - '{{ image_name }}:{{ image_tag }}'
       - src/cli/verify.ts
   register: menu_verify
   changed_when: false
   failed_when: false
 
 - name: Report integrity problems
-  debug:
+  ansible.builtin.debug:
     msg: '{{ menu_verify.stdout_lines | default([]) }}'
   when: menu_verify.rc != 0

ansible/roles/menu-app/templates/menu-app.container.j2

@@ -0,0 +1,60 @@
+# {{ container_name }} — rootless Podman Quadlet unit.
+# Managed by Ansible (menu-admin/roles/menu-app). Do not edit by hand.
+#
+# Joins {{ network_name }}.network, written by roles/register — the same include
+# that attaches Caddy to it. Caddy reaches the app by container name, so nothing
+# is published on the host and the app is not reachable from outside the Podman
+# network at all.
+
+[Unit]
+Description=Menu app for {{ host }}
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName={{ container_name }}
+Image={{ image_name }}:{{ image_tag }}
+
+# keep-id maps the image's own account — `node`, uid/gid {{ app_uid }} — onto
+# {{ podman_user }}, which is what keeps every file the app writes owned by
+# {{ podman_user }} on the host: plain tar/rsync backups, no subuids, no
+# `podman unshare`. The same mapping is why scripts/provision-site-env.sh writes
+# site.env with --userns=keep-id.
+UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
+
+# The alias is deliberately dotless. ContainerName keeps the Docker-era name so
+# operators and scripts find it, but a name with dots in it is a liability in
+# DNS, so {{ network_name }} is what the Caddyfile proxies to. Both resolve from
+# inside the network.
+NetworkAlias={{ network_name }}
+
+Environment=MENU_ROOT=/data
+Environment=MENU_SITE_CONFIG=/app/config/{{ site }}.site.json
+Environment=SECURE_COOKIES=true
+Environment=LOG_LEVEL=info
+Environment=PORT={{ container_port }}
+
+# Everything the app writes: data/ (originals, drafts, published, site.json,
+# sections.json) and site.env. The Site's own config is baked into the image and
+# read from /app.
+Volume={{ app_root }}:/data
+
+Network={{ network_name }}.network
+
+# podman builds an OCI image by default and drops the Containerfile's
+# HEALTHCHECK instruction, so the same check is declared here and the play gates
+# on it with `podman healthcheck run`. node is in the image.
+HealthCmd={{ health_cmd }}
+HealthInterval=30s
+HealthTimeout=5s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+# The app validates its data directory before it serves and refuses to boot on a
+# fatal integrity problem, so a first start over copied data can take a moment.
+TimeoutStartSec=120
+
+[Install]
+WantedBy=default.target

ansible/roles/network/tasks/main.yaml

@@ -1,8 +0,0 @@
----
-- name: 'Ensure {{ docker_network }} exists and Caddy is on it'
-  docker_network:
-    name: '{{ docker_network }}'
-    state: present
-    connected:
-      - '{{ caddy_container }}'
-    appends: yes

ansible/roles/register/tasks/main.yaml

@@ -0,0 +1,51 @@
+---
+# Register this Site's logical network with Caddy.
+#
+# Copied from ansible-playbooks/caddy/roles/register/tasks/main.yaml: this repo
+# carries its own deployment, so it cannot include that file by path. Keep the
+# two in step.
+#
+# Caddy is the *base* unit and knows nothing about its upstreams. A service adds
+# itself by writing two files into the Quadlet directory:
+#
+#   1. its own <name>.network unit, and
+#   2. a drop-in next to caddy.container that pulls that network into Caddy.
+#
+# Caddy itself is owned by ansible-playbooks/caddy/playbook.yml; this play only
+# registers with it. The including play defines a "Restart Caddy" handler.
+
+- name: 'Create the {{ network_name }} network unit'
+  ansible.builtin.copy:
+    dest: '{{ quadlet_dir }}/{{ network_name }}.network'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+    content: |
+      # {{ network_name }} — logical app network.
+      # Managed by Ansible. Do not edit by hand.
+      #
+      # NetworkName pins the Podman network name. Without it Quadlet would
+      # create "systemd-{{ network_name }}".
+      #
+      # isolate blocks traffic between separate Podman networks, so a container
+      # on this network cannot reach one that is only on another. Caddy and any
+      # shared service still reach it, because they are members of it.
+      [Network]
+      NetworkName={{ network_name }}_net
+      Options=isolate=true
+  notify: Restart Caddy
+
+- name: 'Register {{ network_name }} with Caddy'
+  ansible.builtin.copy:
+    dest: '{{ quadlet_dir }}/{{ caddy_name }}.container.d/50-{{ network_name }}.conf'
+    owner: '{{ podman_user }}'
+    group: '{{ podman_user }}'
+    mode: '0644'
+    content: |
+      # Added by the {{ network_name }} playbook. Do not edit by hand.
+      #
+      # Quadlet merges this into caddy.container. Deleting this file removes
+      # the network from Caddy on the next daemon-reload.
+      [Container]
+      Network={{ network_name }}.network
+  notify: Restart Caddy