Run the Sites as rootless Podman Quadlets, and build with podman

The fleet on emily is rootless Podman with Quadlet units under
~/.config/containers/systemd and state in ~/podman-data, all behind one shared
Caddy unit. This brings the menus in line instead of leaving them the last
Docker deployment. The build moves with them: podman reads a Containerfile as
readily as a Dockerfile, so the name now says which toolchain is real.

The app is one .container and one .network file. Two details carry it.
UserNS=keep-id maps the image's own node account onto the user that owns the
units, so data/ and site.env stay owned by tanshu: plain tar backups, no
subuids. provision-site-env.sh writes site.env with the same mapping rather
than --user, which under rootless Podman would have produced a subuid-owned
file the app could not read. And the Site registers its own network with Caddy
by writing a drop-in beside caddy.container, so Caddy never lists its upstreams
and never changes when a Site is added. The unit keeps the Docker-era container
name for the scripts and muscle memory that look it up, while the Caddyfile
proxies to a dotless NetworkAlias, because a container name with dots in it is
a liability in DNS.

The unit keeps podman's Pull=missing, so a restart or a reboot never depends on
the registry being reachable, while the play pulls explicitly at deploy time.
That is also where a missing `podman login` for the private registry now fails,
with the instruction, rather than as a unit that will not start.

copy-site-data.sh moves a Site's data across, and is not a plain rsync:
/var/lib/menu.* on monoco belongs to uid 1000 while the ssh user there is 1001,
so the read needs sudo; monoco cannot ssh to emily, so the workstation relays
the tar; and the extraction runs as tanshu on emily, which is what makes the
files land owned by the user keep-id maps the app onto. It proves the copy by
comparing a SHA-256 of the whole tree against the source and checking every
file's owner.

Not yet done: the copies themselves and the DNS move, which need a human.
This commit is contained in:
tanshu committed 2026-10-16 09:54:52 +05:30
1 parent c412181159
commit e68f8ab00d
23 files changed
+876 -389

No files matched your search

+14 -7
View File
@@ -1,17 +1,24 @@
# The Ansible inventory for the menus. Every Site runs on the same box, so each
# entry names it and points at the vars file that describes it.
# The Ansible inventory for the menus. Every Site runs on the same box — emily,
# the rootless Podman host — so each entry names it and points at the vars file
# that describes it.
#
# cd ansible
# ansible-playbook --inventory=hosts playbook.yml # every Site
# ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains
#
# ansible_host is written as the resolvable name rather than an ssh alias so the
# inventory works on any machine, whatever its ~/.ssh/config says.
# This play carries no become: it writes rootless Quadlet units, which belong to
# whoever Ansible connects as, so it must connect as that user.
#
# ansible_host is written as the resolvable name rather than an ssh alias, so the
# inventory works on any machine, whatever its ~/.ssh/config says. emily has no
# DNS name yet, so that name is its address today; when DNS — or monoco's elastic
# address — moves to it at cutover, this is the one line that changes.
[menus]
hopsngrains ansible_host=monoco.tanshu.com var_file=vars/hopsngrains.yml
greatbear ansible_host=monoco.tanshu.com var_file=vars/greatbear.yml
hngmohali ansible_host=monoco.tanshu.com var_file=vars/hngmohali.yml
hopsngrains ansible_host=35.154.14.59 var_file=vars/hopsngrains.yml
greatbear ansible_host=35.154.14.59 var_file=vars/greatbear.yml
hngmohali ansible_host=35.154.14.59 var_file=vars/hngmohali.yml
[all:vars]
ansible_user=tanshu
ansible_python_interpreter=/usr/bin/python3
+30 -10
View File
@@ -1,15 +1,26 @@
---
# Deploys the menu app for one or more Sites. It lives here, beside the app it
# deploys, and is run from this directory:
# Deploys the menu app for one or more Sites as rootless Podman Quadlet units on
# emily, behind the shared Caddy unit that ansible-playbooks/caddy owns. It lives
# here, beside the app it deploys, and is run from this directory:
#
# make build-production TAG=<tag> (from the repo root)
# ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains
# ./deploy.sh <tag> (repo root)
# ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=<tag>
#
# One run deploys the Site: it starts the container and points the Site's web
# address at it. Nothing older is kept serving, so there is no staged rollout —
# deploying an earlier tag is the rollback.
- hosts: menus
become: true
# Connects as the container user and runs everything as that user: the Quadlet
# units belong to it, so the connection's PAM session is what makes
# `systemctl --user` work without any environment juggling.
#
# gather_facts is required: podman_user and the home directory come from the
# user_id / user_dir facts.
#
# The rootless-Podman prerequisites (installing podman, enabling lingering, the
# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied
# here. They are the shared task file at
# ansible-playbooks/caddy/roles/podman/tasks/main.yaml, and a host that has
# already run ansible-playbooks/caddy/playbook.yml has them.
- name: Deploy the menu app as rootless Podman Quadlets
hosts: menus
gather_facts: true
vars_files:
- vars/default.yml
@@ -19,7 +30,16 @@
file: '{{ var_file }}'
when: var_file is defined
# This must happen before the app unit is started, not after: the generated
# unit references {{ network_name }}.network, and Quadlet fails *unit
# generation* — so `systemctl --user start` reports "unit not found" — when a
# referenced .network file does not exist yet. The same include writes the
# Caddy-side drop-in that attaches Caddy to this network.
- name: Register this Site's network with Caddy
ansible.builtin.include_tasks: '{{ playbook_dir }}/roles/register/tasks/main.yaml'
roles:
- network
# The app: the data directory tree, the Quadlet unit, start + verify.
- menu-app
# Appends this Site's block to the shared Caddyfile and reloads Caddy.
- caddy
+169 -46
View File
@@ -1,26 +1,90 @@
# The menu deployment
Lives here, beside the app it deploys. One run deploys a Site: it starts the
app's container and points the Site's web address at it. There is no older
system to keep serving, so there is no staged rollout and nothing to retire —
deploying an earlier tag is the rollback.
app's container as a **rootless Podman Quadlet unit** on `emily` and points the
Site's web address at it through the shared **Caddy** unit that
`ansible-playbooks/caddy` owns. There is no older system to keep serving, so
there is no staged rollout and nothing to retire — deploying an earlier tag is
the rollback.
**The app's data root is the directory the old static menus were served from**
(`/var/lib/menu.<site>`), on purpose, so the app inherits the location instead of
a second one. The app only reads and writes `data/` and `site.env` inside it. If
you later want the old menu files gone, delete those files — not the directory,
which now holds the live content.
This is the same shape as the sibling services in `ansible-playbooks` (gitea,
nextcloud, tandoor, bitwarden, valkey, postgres, trillium): rootless Podman, one
Quadlet `.container` / `.network` per service in `~/.config/containers/systemd/`,
state under `~/podman-data/`, and Caddy reaching each app by name on a Podman
network. Nothing here talks to Docker.
## The pieces
```
ansible/
├── hosts # one entry per Site, all on emily
├── playbook.yml # connects as tanshu, no become
├── vars/{default,hopsngrains,greatbear,hngmohali}.yml
└── roles/
├── register/ # writes menu-<site>.network and the Caddy drop-in
├── menu-app/ # the Quadlet unit, start + health + verify
└── caddy/ # appends the site block, reloads Caddy
```
Two things are worth internalising, because they are the whole design:
**Caddy knows nothing about its upstreams.** A service registers _itself_ by
writing two files into the Quadlet directory: its own `<name>.network` unit and a
drop-in beside `caddy.container` that pulls that network into Caddy. This play
does that in `pre_tasks`, before the app unit starts, because Quadlet fails unit
generation when a unit references a `.network` file that does not exist yet.
**The container's uid is mapped onto `tanshu`.** `UserNS=keep-id:uid=1000,gid=1000`
maps the image's own `node` account onto the user that owns the units, so every
file the app writes is owned by `tanshu` on the host: plain `tar`/`rsync`
backups, no subuids, no `podman unshare`. `scripts/provision-site-env.sh` uses
the same mapping for the same reason.
## What lives where
| Thing | Where |
| -------------------------------------- | ---------------------------------------------------------------------- |
| Name, tagline, icon, categories, menus | on the server under `/var/lib/menu.<site>/data` — the admin edits them |
| Theme colours | `config/<site>.site.json` in the repo, baked into the image |
| Admin password hash, session secret | `/var/lib/menu.<site>/site.env` (untracked, mode 600) |
| Which image tag runs | `-e tag=<tag>` |
| Which Site is deployed | the inventory entry in `hosts` |
| The web address it serves | `host:` in `vars/<site>.yml` |
| Thing | Where |
| -------------------------------------- | --------------------------------------------------------------------------- |
| Name, tagline, icon, categories, menus | on the server under `~/podman-data/menu-<site>/data` — the admin edits them |
| Admin password hash, session secret | `~/podman-data/menu-<site>/site.env` (untracked, mode 600) |
| Theme colours | `config/<site>.site.json` in the repo, baked into the image |
| The unit that runs | `~/.config/containers/systemd/menu-<site>.container` |
| The network and Caddy drop-in | `menu-<site>.network`, `caddy.container.d/50-menu-<site>.conf` |
| Which image tag runs | `-e tag=<tag>` |
| Which Site is deployed | the inventory entry in `hosts` |
| The web address it serves | `host:` in `vars/<site>.yml` |
`<site>` is the inventory name (`hopsngrains`, `greatbear`, `hngmohali`). The
container itself keeps the Docker-era name `menu-app.<domain>`, because
`provision-site-env.sh` and muscle memory look it up that way; the Caddyfile
proxies to the dotless `menu-<site>` alias, because a container name with dots in
it is a liability in DNS.
## What the host needs
`emily` is already set up — it runs Caddy, Postgres, Tandoor and Valkey the same
way. If you are standing up a fresh host, run `ansible-playbooks/caddy` first: it
installs podman, enables lingering, lowers the privileged-port floor so rootless
Caddy can bind 80/443, creates the Quadlet directory, and starts the Caddy unit.
Those prerequisites deliberately are **not** duplicated here, so a host that has
not run that playbook will fail at the first `systemctl --user` — see
`ansible-playbooks/caddy/readme.md`.
The image is in the private registry, so log in once as `tanshu` on `emily`,
before the first deploy:
```sh
podman login registry.tanshu.com
```
The play pulls the image itself, so a missing login fails there with that
instruction rather than as a container that will not start. The unit keeps
podman's default `Pull=missing` on purpose, so a restart or a reboot never
depends on the registry being reachable.
`ssh` on some machines complains about
`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is
a local file-permission bug whose fix is `sudo chown -R root:root
/etc/ssh/ssh_config.d`, not something this playbook works around.
## Deploying
@@ -33,8 +97,8 @@ SKIP_DEPLOY=1 ./deploy.sh 2026-09-22 # build and push only
`deploy.sh` runs `make build-production TAG=<tag>`, which builds a multi-arch
podman manifest (`linux/amd64,linux/arm64`) and pushes `latest` and the tag to
`registry.tanshu.com/menu-admin`. Then it runs this playbook against the Site
named by `SITE` (default `hopsngrains`).
`registry.tanshu.com/menu-admin`. Then it runs this playbook against every Site,
or the one named by `SITE`.
By hand:
@@ -45,16 +109,89 @@ ansible-playbook --inventory=hosts playbook.yml --limit hopsngrains -e tag=2026-
```
Every Site runs on the same box, so `hosts` has one entry per Site pointing at
the same `ansible_host` with its own `var_file`. `ansible_host` is written as the
resolvable name rather than an ssh alias, so the inventory works whatever a
machine's `~/.ssh/config` says. (`ssh` on some machines complains about
`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` being owned by `nobody`; that is
a local file-permission bug whose fix is `sudo chown -R root:root
/etc/ssh/ssh_config.d`, not something this playbook works around.)
the same `ansible_host` with its own `var_file`. The play carries no `become`: the
Quadlet units belong to `ansible_user` (`tanshu`), and running as that user is
what gives us a real PAM session, which is what makes `systemctl --user` work.
Connecting as anyone else would write units into the wrong home.
Adding a Site is one line in `hosts`, a copy of `vars/hopsngrains.yml`, and a
`config/<site>.site.json`. The image is shared, so the config has to be in it:
a new Site needs a new tag.
`config/<site>.site.json`. The image is shared, so the config has to be in it: a
new Site needs a new tag.
## Migrating a Site's data from `monoco`
The menus still serve from Docker containers on `monoco`. A Site moves across
with its data, before it is ever pointed at from DNS. Run this from a machine
that can ssh to both hosts — the `monoco` and `emily` aliases this repo's own
tooling uses:
```sh
scripts/copy-site-data.sh hopsngrains
scripts/copy-site-data.sh greatbear
scripts/copy-site-data.sh hngmohali
```
For one Site it stops the old container, streams `/var/lib/menu.<domain>` across,
starts the old container again so `monoco` keeps serving until DNS moves, and
then proves the copy: the SHA-256 of the whole tree is compared against the
source, and every file is checked to belong to `tanshu`. `--live` skips the stop;
`MONOCO=`, `EMILY=`, `SRC_ROOT=` and `DEST_ROOT=` override the defaults.
Two things make the copy look different from a plain `rsync`:
- **`sudo` on monoco.** `/var/lib/menu.<domain>` belongs to uid 1000 (the
`ubuntu` account) and the ssh user there is 1001, so it is not readable without
it. monoco's `tanshu` has passwordless sudo.
- **The workstation relays it.** monoco cannot ssh to emily, so the tar streams
through the machine running the script; nothing large is written locally.
Extracting as `tanshu` on emily is what makes the files land owned by the user
`keep-id` maps the app onto — as root they would arrive owned by uid 1000,
which on emily is nobody.
By hand, that is:
```sh
SITE=hopsngrains
HOST=menu.hopsngrains.com
ssh monoco "docker stop menu-app.$HOST"
ssh -n monoco "sudo -n tar -C /var/lib/$HOST -cf - ." \
| ssh emily "mkdir -p ~/podman-data/menu-$SITE && tar -C ~/podman-data/menu-$SITE -xf -"
ssh monoco "docker start menu-app.$HOST"
# Same checks the script runs.
ssh -n monoco "sudo -n sh -c 'cd /var/lib/$HOST && find . -type f -exec sha256sum {} +' | sort | sha256sum"
ssh -n emily "cd ~/podman-data/menu-$SITE && find . -type f -exec sha256sum {} + | sort | sha256sum"
ssh -n emily "find ~/podman-data/menu-$SITE ! -user tanshu"
```
The old layout put everything under `/var/lib/menu.<domain>/`, with the content
one level down in `data/`; the new one is `~/podman-data/menu-<site>/` with the
same `data/` inside it and `site.env` beside it. Nothing inside changes.
`site.env` carries over so the admin keeps working; without it the app starts
with the admin disabled and the playbook says so rather than failing.
Then deploy that Site and check it on a `Host:` override before touching DNS.
`monoco` keeps serving until the cutover; only then stop its container — and
leave it in place, not removed, so `docker start` is the rollback.
## The admin password
The app starts with the admin disabled while `~/podman-data/menu-<site>/site.env`
is missing, and the playbook says so rather than failing. Create it once per Site;
run it again with `--force` to reset the password (a fresh session secret signs
everyone out). There is no minimum password length.
```sh
scp scripts/provision-site-env.sh emily:/tmp/
ssh -t emily bash /tmp/provision-site-env.sh ~/podman-data/menu-hopsngrains
systemctl --user restart menu-hopsngrains.service
```
The `-t` matters: the script prompts for the password, and without a terminal on
the remote side the prompt cannot be read, so it would sit there looking frozen.
With no image argument it borrows the one the running container uses. It has to
run as `tanshu`, the user whose rootless Podman owns the container.
## Checking it landed
@@ -63,25 +200,8 @@ npm run check-site -- https://menu.hopsngrains.com
```
Read-only, exits non-zero on failure. Then from a phone: the landing page, a
Section page, the page counter, tap-to-zoom, and signing into `/admin` to
publish something.
## The admin password
The app starts with the admin disabled while `/var/lib/menu.<site>/site.env`
is missing, and the playbook says so rather than failing. Create it once per
Site; run it again with `--force` to reset the password (a fresh session secret
signs everyone out). There is no minimum password length.
```sh
scp scripts/provision-site-env.sh monoco:/tmp/
ssh -t monoco bash /tmp/provision-site-env.sh /var/lib/menu.hopsngrains.com
ssh monoco docker restart menu-app.hopsngrains.com
```
The `-t` matters: the script prompts for the password, and without a terminal on
the remote side the prompt cannot be read, so it would sit there looking frozen.
With no image argument it borrows the one the running container uses.
Section page, the page counter, tap-to-zoom, and signing into `/admin` to publish
something.
## Rolling back
@@ -91,4 +211,7 @@ Deploy the previous tag:
./deploy.sh <previous-tag>
```
`docker logs menu-app.<site>` is where a failed container says why.
`podman logs menu-app.<site>` and `journalctl --user -u menu-<site>` are where a
container that will not start says why. To go all the way back to the Docker
deployment, start the container that was left on `monoco` and point Caddy at it
again.
+30 -5
View File
@@ -1,6 +1,31 @@
---
- name: Reload Caddy
# Caddy validates before it applies, so a mistake here leaves the running
# config untouched and fails the play loudly.
command: 'docker exec -w /etc/caddy {{ caddy_container }} caddy reload'
listen: Reload Caddy
# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
# Quadlet unit there. These handlers run as the connection user, so
# `systemctl --user` and `podman exec` work without any XDG_RUNTIME_DIR juggling.
# Notified by this role when the Caddyfile changes. A reload keeps existing
# connections; a restart would briefly drop every hosted site.
#
# Note the config path: /etc/caddy/Caddyfile is the mount inside the container,
# not {{ caddyfile_path }}, which is the host side of it.
- name: Reload Caddy configuration
ansible.builtin.command: >-
podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
changed_when: true
listen: 'Reload Caddy'
# Notified by roles/register when this Site's network unit, or the drop-in that
# attaches Caddy to it, is written. Quadlet has no live equivalent of
# `docker network connect`, so the container has to be recreated — a brief
# restart of every hosted site, not just this one. See the Caddy playbook's
# readme, "Trade-off worth knowing".
#
# Defined after the reload above so that when both are notified the cheap config
# reload happens first.
- name: Restart Caddy
ansible.builtin.systemd_service:
name: '{{ caddy_name }}.service'
scope: user
state: restarted
daemon_reload: true
listen: Restart Caddy
+25 -7
View File
@@ -1,14 +1,29 @@
---
# Append this Site's block to the shared Caddyfile.
#
# Caddy is its own playbook (ansible-playbooks/caddy/playbook.yml) and a rootless
# Quadlet unit there, so this role does not manage the container — it only points
# it at the app. The Caddyfile is a bind mount into that container, so Caddy sees
# a written change as soon as the file lands; the handler only asks Caddy to
# re-read it. That is a reload, not a restart, so the other hosted sites are
# untouched.
#
# blockinfile is idempotent on its own via the marker, so there is no need to
# slurp the file and compare first — which also matters because slurp *fails*
# when the Caddyfile does not exist yet, on a host where the Caddy playbook has
# not run.
- name: Check the Caddyfile exists
stat:
ansible.builtin.stat:
path: '{{ caddyfile_path }}'
register: menu_caddyfile
- name: Fail when Caddy is not where we expect it
fail:
ansible.builtin.fail:
msg: >-
{{ caddyfile_path }} is missing. Deploy the caddy playbook, which owns the
Caddy container and its config, before this one.
{{ caddyfile_path }} is missing. Deploy the Caddy playbook
(ansible-playbooks/caddy/playbook.yml), which owns the Caddy unit and its
config, before this one.
when: not menu_caddyfile.stat.exists
# The Site's web address may still be claimed by an older snippet: the retired
@@ -16,7 +31,7 @@
# Caddy refuse the whole config, and a refused reload leaves the old proxy in
# place, which is a 502 once the old container is gone.
- name: Drop any older snippet claiming this web address
blockinfile:
ansible.builtin.blockinfile:
path: '{{ caddyfile_path }}'
marker: '# {mark} Ansible managed Caddy snippet for {{ host }}'
state: absent
@@ -25,9 +40,12 @@
# blockinfile replaces whatever sits between its markers, so redeploying updates
# the snippet in place rather than appending a second one.
- name: Point the Site's web address at the app
blockinfile:
ansible.builtin.blockinfile:
path: '{{ caddyfile_path }}'
marker: '# {mark} ANSIBLE MANAGED: menu app ({{ site }})'
block: '{{ caddy_snippet }}'
create: no
create: false
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0644'
notify: Reload Caddy
+19
View File
@@ -0,0 +1,19 @@
---
# All handlers run as the connection user ({{ podman_user }}), so systemctl
# --user works without setting XDG_RUNTIME_DIR.
- name: Reload systemd user units
ansible.builtin.command: systemctl --user daemon-reload
changed_when: true
listen: Reload systemd user units
# A changed unit file changes the generated unit, so the app is recreated. All of
# its state is on the bind mount, so a recreate loses nothing — but it is a few
# seconds of downtime for the one Site.
- name: Restart the menu app
ansible.builtin.systemd_service:
name: '{{ unit_name }}'
scope: user
state: restarted
daemon_reload: true
listen: Restart the menu app
+134 -90
View File
@@ -1,134 +1,178 @@
---
- name: Pull the app image
docker_image:
name: '{{ image_name }}'
tag: '{{ image_tag }}'
source: pull
force_source: yes
# The menu app as a rootless Podman Quadlet unit.
#
# Everything here runs as {{ podman_user }} — the connection user. No become, no
# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
# user manager is reachable through the standard environment.
# The pull is explicit, rather than left to the Quadlet unit, for two reasons.
# The image lives in a private registry, so a failure here is where a missing
# `podman login` should show up — not two tasks later as "unit not found". And
# the unit keeps podman's default `Pull=missing`, so it never reaches for the
# registry on a restart or after a reboot: that is what lets a box come back up
# when the registry is down. Pulling here is what keeps a redeployed tag current.
- name: Pull the app image
ansible.builtin.command: 'podman pull {{ image_name }}:{{ image_tag }}'
register: menu_pull
changed_when: false
failed_when: false
- name: Fail when the image cannot be pulled
ansible.builtin.fail:
msg: >-
podman could not pull {{ image_name }}:{{ image_tag }}.
{{ menu_pull.stderr | default('') | trim }}
The registry is private: run `podman login registry.tanshu.com` once as
{{ podman_user }} on this host.
when: menu_pull.rc != 0
# The data directory is created here rather than left to the image. Under
# keep-id the image's own root phase maps to a subuid, so anything it tries to
# create under the mount fails with EPERM; the directory has to exist already,
# owned by {{ podman_user }}.
- name: Ensure the app root exists
file:
ansible.builtin.file:
path: '{{ app_root }}'
state: directory
owner: '{{ app_uid }}'
group: '{{ app_gid }}'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0750'
# MENU_ROOT is the app root, and the app keeps its content one level down in
# data/ (originals/, drafts/, published/, site.json, sections.json).
- name: Ensure the data directory exists
file:
ansible.builtin.file:
path: '{{ app_root }}/data'
state: directory
owner: '{{ app_uid }}'
group: '{{ app_gid }}'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0750'
- name: Look for the untracked site.env
stat:
ansible.builtin.stat:
path: '{{ app_root }}/site.env'
register: menu_site_env
- name: Warn when the admin has no password yet
debug:
ansible.builtin.debug:
msg: >-
{{ app_root }}/site.env is missing, so the admin is disabled: the public
menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on
this host, then re-run this playbook.
menu still serves. Run scripts/provision-site-env.sh {{ app_root }} on this
host, then re-run this playbook.
when: not menu_site_env.stat.exists
- name: Run the menu app
docker_container:
name: '{{ docker_container }}'
image: '{{ image_name }}:{{ image_tag }}'
state: started
restart_policy: unless-stopped
user: '{{ app_uid }}:{{ app_gid }}'
# Force a rebuild of the container with -e recreate_container=true: that is
# what applies a changed healthcheck or env to an existing container, since
# the module otherwise leaves a running one alone.
recreate: '{{ recreate_container | default(false) | bool }}'
# Declared here rather than relied on from the image: podman builds an OCI
# image by default and drops Docker's HEALTHCHECK instruction, so the
# Dockerfile's own healthcheck never reaches the container. node is in the
# image and the exec form needs no shell quoting.
healthcheck:
test:
- CMD
- node
- -e
- "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
interval: 30s
timeout: 5s
start_period: 30s
retries: 3
env:
MENU_ROOT: /data
MENU_SITE_CONFIG: '/app/config/{{ site }}.site.json'
SECURE_COOKIES: 'true'
LOG_LEVEL: info
PORT: '3000'
volumes:
- '{{ app_root }}:/data'
networks:
- name: '{{ docker_network }}'
- name: Install the menu app Quadlet unit
ansible.builtin.template:
src: menu-app.container.j2
dest: '{{ quadlet_dir }}/{{ network_name }}.container'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0644'
notify:
- Reload systemd user units
- Restart the menu app
- name: Keep site.env readable only by the app user
file:
path: '{{ app_root }}/site.env'
owner: '{{ app_uid }}'
group: '{{ app_gid }}'
mode: '0600'
when: menu_site_env.stat.exists
- name: Start the menu app
block:
- name: Start the menu app
ansible.builtin.systemd_service:
name: '{{ unit_name }}'
scope: user
state: started
daemon_reload: true
rescue:
# systemd reports "control process exited with error code" and nothing else,
# which is useless for the usual failures (a missing network unit, an
# unwritable /data, a bad uid mapping). Pull the real reason out.
- name: Collect diagnostics
ansible.builtin.command: '{{ item }}'
loop:
- 'systemctl --user status {{ unit_name }} --no-pager'
- 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
- 'podman logs --tail 40 {{ container_name }}'
register: menu_diagnostics
changed_when: false
failed_when: false
# Deliberately not the image's HEALTHCHECK: podman builds an OCI image by
# default and drops the Docker HEALTHCHECK instruction, so State.Health is absent
# and a wait on it fails on a perfectly healthy container — which stops the play
# before Caddy is ever pointed at the app.
- name: Wait for the app to answer inside its own container
ansible.builtin.command:
argv:
- docker
- exec
- '{{ docker_container }}'
- node
- -e
- "fetch('http://127.0.0.1:3000/healthz').then(r => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
register: menu_health
- name: Show diagnostics
ansible.builtin.debug:
msg: '{{ menu_diagnostics.results | map(attribute="stdout") | list }}'
- name: Fail
ansible.builtin.fail:
msg: 'The menu app did not start — see the diagnostics above.'
# Quadlet honours [Install] WantedBy= when it generates the unit, so
# `systemctl --user enable` is neither needed nor possible on a generated unit.
- name: Verify the menu app is running
ansible.builtin.command: 'systemctl --user is-active {{ unit_name }}'
register: menu_active
changed_when: false
retries: 20
delay: 3
failed_when: menu_active.stdout != 'active'
# `is-active` says the container is running; it does not say the app can serve.
# Restart=always means even a crash-looping container is reported active — which
# is exactly what a fatal integrity problem in the data directory produces. The
# unit declares the health check, so run that once, synchronously.
- name: Wait for the app to pass its health check
ansible.builtin.command: 'podman healthcheck run {{ container_name }}'
register: menu_health
until: menu_health.rc == 0
retries: 24
delay: 5
changed_when: false
failed_when: false
- name: Fail when the app never answers
fail:
msg: >-
{{ docker_container }} did not answer /healthz within a minute. Its logs
say why: docker logs {{ docker_container }}
- name: Report a container that never answered
when: menu_health.rc != 0
block:
- name: Collect diagnostics
ansible.builtin.command: '{{ item }}'
loop:
- 'systemctl --user status {{ unit_name }} --no-pager'
- 'journalctl --user -u {{ unit_name }} --no-pager -n 40'
- 'podman logs --tail 40 {{ container_name }}'
register: menu_health_diagnostics
changed_when: false
failed_when: false
- name: Show diagnostics
ansible.builtin.debug:
msg: '{{ menu_health_diagnostics.results | map(attribute="stdout") | list }}'
- name: Fail
ansible.builtin.fail:
msg: >-
{{ container_name }} did not pass its health check within two minutes.
Its logs are above, and: podman logs {{ container_name }}
- name: Keep site.env readable only by the app user
ansible.builtin.file:
path: '{{ app_root }}/site.env'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0600'
when: menu_site_env.stat.exists
- name: Check the data directory for integrity problems
# Reads the same directory the container writes, as the same user, so a
# permissions mistake shows up here rather than at 8pm service.
command:
ansible.builtin.command:
argv:
- docker
- run
- --rm
- --user
- '{{ app_uid }}:{{ app_gid }}'
- -e
- podman
- exec
- --env
- MENU_ROOT=/data
- -e
- --env
- 'MENU_SITE_CONFIG=/app/config/{{ site }}.site.json'
- --entrypoint
- '{{ container_name }}'
- node
- '{{ image_name }}:{{ image_tag }}'
- src/cli/verify.ts
register: menu_verify
changed_when: false
failed_when: false
- name: Report integrity problems
debug:
ansible.builtin.debug:
msg: '{{ menu_verify.stdout_lines | default([]) }}'
when: menu_verify.rc != 0
@@ -0,0 +1,60 @@
# {{ container_name }} — rootless Podman Quadlet unit.
# Managed by Ansible (menu-admin/roles/menu-app). Do not edit by hand.
#
# Joins {{ network_name }}.network, written by roles/register — the same include
# that attaches Caddy to it. Caddy reaches the app by container name, so nothing
# is published on the host and the app is not reachable from outside the Podman
# network at all.
[Unit]
Description=Menu app for {{ host }}
After=network-online.target
Wants=network-online.target
[Container]
ContainerName={{ container_name }}
Image={{ image_name }}:{{ image_tag }}
# keep-id maps the image's own account — `node`, uid/gid {{ app_uid }} — onto
# {{ podman_user }}, which is what keeps every file the app writes owned by
# {{ podman_user }} on the host: plain tar/rsync backups, no subuids, no
# `podman unshare`. The same mapping is why scripts/provision-site-env.sh writes
# site.env with --userns=keep-id.
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
# The alias is deliberately dotless. ContainerName keeps the Docker-era name so
# operators and scripts find it, but a name with dots in it is a liability in
# DNS, so {{ network_name }} is what the Caddyfile proxies to. Both resolve from
# inside the network.
NetworkAlias={{ network_name }}
Environment=MENU_ROOT=/data
Environment=MENU_SITE_CONFIG=/app/config/{{ site }}.site.json
Environment=SECURE_COOKIES=true
Environment=LOG_LEVEL=info
Environment=PORT={{ container_port }}
# Everything the app writes: data/ (originals, drafts, published, site.json,
# sections.json) and site.env. The Site's own config is baked into the image and
# read from /app.
Volume={{ app_root }}:/data
Network={{ network_name }}.network
# podman builds an OCI image by default and drops the Containerfile's
# HEALTHCHECK instruction, so the same check is declared here and the play gates
# on it with `podman healthcheck run`. node is in the image.
HealthCmd={{ health_cmd }}
HealthInterval=30s
HealthTimeout=5s
HealthRetries=3
HealthStartPeriod=30s
[Service]
Restart=always
# The app validates its data directory before it serves and refuses to boot on a
# fatal integrity problem, so a first start over copied data can take a moment.
TimeoutStartSec=120
[Install]
WantedBy=default.target
-8
View File
@@ -1,8 +0,0 @@
---
- name: 'Ensure {{ docker_network }} exists and Caddy is on it'
docker_network:
name: '{{ docker_network }}'
state: present
connected:
- '{{ caddy_container }}'
appends: yes
+51
View File
@@ -0,0 +1,51 @@
---
# Register this Site's logical network with Caddy.
#
# Copied from ansible-playbooks/caddy/roles/register/tasks/main.yaml: this repo
# carries its own deployment, so it cannot include that file by path. Keep the
# two in step.
#
# Caddy is the *base* unit and knows nothing about its upstreams. A service adds
# itself by writing two files into the Quadlet directory:
#
# 1. its own <name>.network unit, and
# 2. a drop-in next to caddy.container that pulls that network into Caddy.
#
# Caddy itself is owned by ansible-playbooks/caddy/playbook.yml; this play only
# registers with it. The including play defines a "Restart Caddy" handler.
- name: 'Create the {{ network_name }} network unit'
ansible.builtin.copy:
dest: '{{ quadlet_dir }}/{{ network_name }}.network'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0644'
content: |
# {{ network_name }} — logical app network.
# Managed by Ansible. Do not edit by hand.
#
# NetworkName pins the Podman network name. Without it Quadlet would
# create "systemd-{{ network_name }}".
#
# isolate blocks traffic between separate Podman networks, so a container
# on this network cannot reach one that is only on another. Caddy and any
# shared service still reach it, because they are members of it.
[Network]
NetworkName={{ network_name }}_net
Options=isolate=true
notify: Restart Caddy
- name: 'Register {{ network_name }} with Caddy'
ansible.builtin.copy:
dest: '{{ quadlet_dir }}/{{ caddy_name }}.container.d/50-{{ network_name }}.conf'
owner: '{{ podman_user }}'
group: '{{ podman_user }}'
mode: '0644'
content: |
# Added by the {{ network_name }} playbook. Do not edit by hand.
#
# Quadlet merges this into caddy.container. Deleting this file removes
# the network from Caddy on the next daemon-reload.
[Container]
Network={{ network_name }}.network
notify: Restart Caddy
+42 -3
View File
@@ -2,19 +2,58 @@
# Defaults shared by every Site. Each Site's own file is named by the `var_file`
# in ansible/hosts, and its slug is its inventory name.
# ---------------------------------------------------------------------------
# Rootless Podman / Quadlet
# ---------------------------------------------------------------------------
# podman_user is derived from the connection: this play carries no become, so the
# Quadlet units belong to whoever Ansible connects as. Requires gather_facts
# (set in playbook.yml).
#
# These three duplicate caddy/vars/default.yml in the ansible-playbooks repo,
# which owns the Caddy unit these Sites sit behind. Keep them in sync.
podman_user: "{{ ansible_facts['user_id'] }}"
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
# Which Site config inside the image to run, and which web address it serves.
site: '{{ inventory_hostname }}'
# ---------------------------------------------------------------------------
# The app
# ---------------------------------------------------------------------------
image_name: registry.tanshu.com/menu-admin
image_tag: "{{ tag | default('latest') }}"
# The unit file is menu-<site>.container and the Podman network is
# menu-<site>_net. The container keeps the Docker-era name, dots and all, because
# scripts/provision-site-env.sh and anything an operator types look it up that
# way — but a name with dots is a DNS liability, so the Caddyfile proxies to the
# dotless NetworkAlias the unit sets beside it.
network_name: 'menu-{{ site }}'
unit_name: '{{ network_name }}.service'
container_name: 'menu-app.{{ host }}'
# The image's own account (`node`), which keep-id maps onto podman_user.
app_uid: 1000
app_gid: 1000
caddy_container: caddy
caddyfile_path: /var/lib/caddy/conf/Caddyfile
container_port: 3000
# podman builds an OCI image by default and drops the Containerfile's
# HEALTHCHECK instruction, so the unit declares the same check and the play gates
# on it. node is in the image and the sh -c podman wraps it in handles the quotes.
health_cmd: >-
node -e "fetch('http://127.0.0.1:{{ container_port }}/healthz').then(r =>
process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"
# ---------------------------------------------------------------------------
# The shared Caddy unit and its Caddyfile, both owned by the caddy playbook in
# ansible-playbooks (caddy/playbook.yml).
# ---------------------------------------------------------------------------
caddy_name: caddy
caddyfile_path: '{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile'
caddy_snippet: |
{{ host }} {
reverse_proxy {{ docker_container }}:3000
reverse_proxy {{ network_name }}:{{ container_port }}
}
+7 -9
View File
@@ -1,13 +1,11 @@
---
# The Great Bear. Its name, tagline, icon, categories and menus live on the server under
# app_root and are edited in the admin; config/greatbear.site.json only seeds the
# theme and the first category list.
# The Great Bear. Its name, tagline, icon, categories and menus live on the
# server under app_root and are edited in the admin; config/greatbear.site.json
# only seeds the theme and the first category list.
title: greatbear.in
host: 'menu.{{ title }}'
docker_container: 'menu-app.{{ title }}'
docker_network: '{{ docker_container }}_net'
# Everything the app writes — data/ and the untracked site.env. Deliberately the
# directory the old static menus were served from.
app_root: '/var/lib/{{ host }}'
# Everything the app writes — data/ (the content) and site.env (the admin
# password hash and session secret). One directory per Site, each of which is
# also its own Podman network and its own Caddy upstream.
app_root: '{{ podman_data_dir }}/{{ network_name }}'
+7 -9
View File
@@ -1,13 +1,11 @@
---
# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the server under
# app_root and are edited in the admin; config/hngmohali.site.json only seeds the
# theme and the first category list.
# Hops n Grains Mohali. Its name, tagline, icon, categories and menus live on the
# server under app_root and are edited in the admin; config/hngmohali.site.json
# only seeds the theme and the first category list.
title: hngmohali.com
host: 'menu.{{ title }}'
docker_container: 'menu-app.{{ title }}'
docker_network: '{{ docker_container }}_net'
# Everything the app writes — data/ and the untracked site.env. Deliberately the
# directory the old static menus were served from.
app_root: '/var/lib/{{ host }}'
# Everything the app writes — data/ (the content) and site.env (the admin
# password hash and session secret). One directory per Site, each of which is
# also its own Podman network and its own Caddy upstream.
app_root: '{{ podman_data_dir }}/{{ network_name }}'
+7 -13
View File
@@ -1,7 +1,7 @@
---
# Hops n Grains. Its name, tagline, icon, categories and menus live on the
# server under app_root and are edited in the admin; config/hopsngrains.site.json
# only seeds the theme. This file says where the Site runs and what it answers.
# Hops n Grains. Its name, tagline, icon, categories and menus live on the server
# under app_root and are edited in the admin; config/hopsngrains.site.json only
# seeds the theme. This file says where the Site runs and what it answers.
#
# Every value that interpolates a template is quoted: YAML reads a leading `{`
# as a flow mapping, which is the parse error Ansible reports as "missing quotes
@@ -9,13 +9,7 @@
title: hopsngrains.com
host: 'menu.{{ title }}'
docker_container: 'menu-app.{{ title }}'
docker_network: '{{ docker_container }}_net'
# Everything the app writes — data/ and the untracked site.env.
#
# Deliberately the directory the old static menus were served from, so the app
# inherits the location rather than a second one. The app only ever reads and
# writes its own data/ inside it; the old menu files can be emptied out when
# they are no longer wanted, and the directory itself must stay.
app_root: '/var/lib/{{ host }}'
# Everything the app writes — data/ (the content) and site.env (the admin
# password hash and session secret). One directory per Site, each of which is
# also its own Podman network and its own Caddy upstream.
app_root: '{{ podman_data_dir }}/{{ network_name }}'