The first deploy put all three Sites up serving and permanently unhealthy: every
healthcheck run died with `/bin/sh: syntax error: unterminated quoted string`.
The unit file carried the closing quote, but podman's stored config did not.
Quadlet's parser strips a trailing double quote, so `node -e "fetch(...)"`
reached the container as an unterminated shell string. The sibling apps never hit
this because their HealthCmd is a bare `curl`.
The check is now `wget -q -O /dev/null http://127.0.0.1:<port>/healthz` — busybox
wget, no quoting, non-zero on anything but a 2xx — and the Dockerfile's own
HEALTHCHECK says the same thing. The vars comment, the unit template and the
deployment readme all say why it has to stay quote-free.
Found from the running container rather than the play output: `podman
healthcheck run` returned 1, `.State.Health.Log` held the sh error, and the app
log showed it listening on 127.0.0.1:3000 the whole time.
menu-admin was the odd one out: it built from the working directory with a date
tag and a multi-arch manifest, deployed with `-e tag=`, and kept the Docker-era
container name. The other two apps build from `git archive HEAD` and tag with the
commit, so what runs on the host is exactly what git knows about; build only for
the architecture emily actually runs; and use one slug for the container, its
network and its data directory.
The build is now their three lines: `podman build --platform linux/arm64` from
`git archive HEAD`, tagged `:<commit>` and `:latest`, pushed with `podman push`
rather than a manifest. `deploy.sh` pushes the commit first, then refuses a dirty
tracked tree or a HEAD that is not origin/main — which is what makes the commit
tag trustworthy — takes the same `--force` and `--no-push`, and ends in the same
`-u tanshu ... -e image_tag=<commit>`. Its own lint/typecheck/test gate stays,
but behind `--checks`, so a plain deploy behaves as theirs does.
The registry credential moves into vars/default.yml and the role logs in, pulls
and logs out around it, as their roles do. The image is only pulled at deploy
time, so the unit keeps podman's default `Pull=missing` and a reboot never needs
the registry. This is the same plaintext credential the other two repos carry.
One real bug goes with it: the healthcheck gate ran before the handler that
restarts a changed unit, so a run that only changed the unit checked the stale
container. `meta: flush_handlers` before the gate is the fix the sibling repos
already made.
The container, network and data directory are all menu-<site> now. The dotted
Docker-era name and the NetworkAlias it needed are gone, so `podman logs
menu-<site>` and `systemctl --user status menu-<site>` are the same two commands
across all three apps. The Dockerfile keeps its name for the same reason: podman
reads it as readily as a Containerfile, and the siblings use that name too.
The fleet on emily is rootless Podman with Quadlet units under
~/.config/containers/systemd and state in ~/podman-data, all behind one shared
Caddy unit. This brings the menus in line instead of leaving them the last
Docker deployment. The build moves with them: podman reads a Containerfile as
readily as a Dockerfile, so the name now says which toolchain is real.
The app is one .container and one .network file. Two details carry it.
UserNS=keep-id maps the image's own node account onto the user that owns the
units, so data/ and site.env stay owned by tanshu: plain tar backups, no
subuids. provision-site-env.sh writes site.env with the same mapping rather
than --user, which under rootless Podman would have produced a subuid-owned
file the app could not read. And the Site registers its own network with Caddy
by writing a drop-in beside caddy.container, so Caddy never lists its upstreams
and never changes when a Site is added. The unit keeps the Docker-era container
name for the scripts and muscle memory that look it up, while the Caddyfile
proxies to a dotless NetworkAlias, because a container name with dots in it is
a liability in DNS.
The unit keeps podman's Pull=missing, so a restart or a reboot never depends on
the registry being reachable, while the play pulls explicitly at deploy time.
That is also where a missing `podman login` for the private registry now fails,
with the instruction, rather than as a unit that will not start.
copy-site-data.sh moves a Site's data across, and is not a plain rsync:
/var/lib/menu.* on monoco belongs to uid 1000 while the ssh user there is 1001,
so the read needs sudo; monoco cannot ssh to emily, so the workstation relays
the tar; and the extraction runs as tanshu on emily, which is what makes the
files land owned by the user keep-id maps the app onto. It proves the copy by
comparing a SHA-256 of the whole tree against the source and checking every
file's owner.
Not yet done: the copies themselves and the DNS move, which need a human.
Reverts the explicit subnets: the pools were freed by deleting five networks, so
the network role is back to its single idempotent task - ensure the network
exists and Caddy is on it - and the Sites no longer declare a subnet. The
deployment readme keeps the note that a new Site needs a config, and therefore a
new tag, but no longer asks for a free /24.
If the pools fill up again the failure is recognisable: 'all predefined address
pools have been fully subnetted', and the choices are deleting stale networks or
declaring a subnet per Site.
The deploy failed on hngmohali with 'all predefined address pools have been
fully subnetted': every 172.17-172.30/16 is in use, 172.31/16 is the host's own
VPC so Docker will not allocate from it, and all sixteen 192.168.0-240/20 blocks
are taken. Docker therefore refuses a network created without a subnet.
Each Site now names a free network_subnet (10.210.0.0/16, one /24 each), the role
looks the network up first and only creates it when it is absent and with that
subnet, and it fails with an explanation when a Site declares none. Existing
networks are left alone, because Docker cannot change a subnet in place and the
Sites already running have theirs. The test checks every Site declares one, and
the deployment readme says a new Site needs a free subnet.
deploy.sh now deploys every Site by default, with SITE=<name> to narrow it, which
is what the local edit was reaching for.
The script refused anything under ten characters. That is a policy the owner
should hold, not one the tooling invents; the only rule left is that the password
cannot be empty, since an empty one would let anyone in. Resetting is the same
script with --force, which overwrites the hash and generates a fresh session
secret, signing out anyone who was signed in - now written down in the operating
notes and the deployment readme, including the form that prompts on your own
machine instead of the server.
That was the intent of the uncommitted edit, and it is a better location than a
second directory: the app inherits the place the menus were already served from
and only ever reads and writes its own data/ and site.env inside it.
app_root is written as /var/lib/{{ host }}, quoted, because the unquoted form is
what failed to parse. Every documented path follows, and the deployment readme
now says that emptying the old menu files means deleting those files rather than
the directory, which holds the live content.
There is no previous system kept serving, so the playbook no longer knows about
one: the retire flags, the temporary web address, the task that removed the old
nginx snippet and the task that stopped the old container are gone. One run
starts the container and points the Site's web address at it, and deploying an
earlier tag is the rollback. The vars lose vhost, retire_nginx_snippet,
retire_nginx_container and legacy_container; the Caddy role keeps only the
snippet upsert.
The deployment readme drops its staged-rollout sections and the glossary of words
that no longer exist, and the plan's sequence and the operating notes describe
the single run instead.
The deployment readme used three pieces of jargon without defining any of them,
which reads as a locked door to anyone doing their first deploy. It now opens
with what each word means, the table of which address serves what at each stage,
and the note that skipping the parallel run is a legitimate choice.
Domain terms stay in CONTEXT.md; these are operational words for this directory,
so they are defined here.
The playbook, its roles, its vars and a new inventory now live in ansible/,
beside the app they deploy, the way the other apps on this machine are laid out.
The inventory follows the same shape: one entry per Site, all on the same box,
each naming its own vars file through var_file, so adding a Site is one line and
a file rather than a command-line flag. ansible_host is written as the
resolvable name rather than an ssh alias so the inventory works on any machine.
The image is now built with podman as a multi-arch manifest and pushed to
registry.tanshu.com/menu-admin, mirroring the Makefile used elsewhere on this
machine: build-production, plus build-check and build-check-local for a build
that does not push. deploy.sh runs the checks, builds, pushes, then runs the
playbook for the Site named by SITE, with SKIP_DEPLOY for a push without a
deploy.
ESLint comes out again: this project's typescript@7 has no JavaScript compiler
API, so typescript-eslint cannot install, and linting only the JavaScript was
not worth keeping. Prettier stays, and the README records why ESLint is absent
and what would bring it back.
tanshu