mozimo.in/ansible

Deploying mozimo

Rootless podman + Quadlet on the host emily (aarch64, Ubuntu 26.04), fronted by that host's shared Caddy. Nothing here runs as root.

./deploy.sh                                   # build the pushed commit + deploy
ansible-playbook playbook.yml -e image_tag=<commit> [--check --diff]

How it fits together

Piece	Where
Quadlet units	~/.config/containers/systemd/{mozimo,vince}.container, mozimo.network
Service data	~/podman-data/mozimo/ (PocketBase db + files, .env), ~/podman-data/vince/
Shared Caddy config	~/podman-data/caddy/conf/Caddyfile — one marked block per service
Caddy's network membership	~/.config/containers/systemd/caddy.container.d/50-mozimo.conf

Quadlet units are not read by systemd directly: the podman user generator turns them into user services on daemon-reload, and [Install] WantedBy=default.target brings them up at boot. systemctl --user is-enabled mozimo reports generated — that is normal, there is no unit to enable.

Nothing is published on the host. Caddy joins mozimo_net (through the drop-in) and proxies to the containers by name:

mozimo.in        → mozimo:3000   (plus /_/* and /api/* → mozimo:8090)
www.mozimo.in    → 301 mozimo.in
cms.mozimo.in    → mozimo:8090
stats.mozimo.in  → vince:8000
mozimo.in/stats/* → vince:8000 (prefix stripped)

The user id mapping matters: the app image runs as uid 1000 (node) and Vince as root, so both units use UserNS=keep-id: to map that uid onto tanshu. Without it, PocketBase and Vince would write files owned by a subuid, and backups, rsync and tar would all trip over them.

Services register themselves

This host's convention (also used by its tandoor, postgres and caddy playbooks) is that a service owns its own registration rather than editing a central list:

1. it ships a <name>.network Quadlet with NetworkName=<name>_net;
2. it drops caddy.container.d/50-<name>.conf containing [Container] Network=<name>.network, which Quadlet merges into Caddy;
3. it appends a marked block to Caddy's Caddyfile and reloads it.

Removing a service is the reverse: delete the drop-in, delete the Caddyfile block, systemctl --user daemon-reload, and the network is no longer Caddy's.

mozimo-network.service is a oneshot with RemainAfterExit=yes, so systemd keeps reporting it as active (exited) even if the network underneath it was removed out of band (podman network rm, podman network prune, a wiped /run/user/<uid>); restarting Caddy then starts it against a network that is not there, and Caddy's own restart loop takes every site on the host down with it. The playbook therefore asks podman (podman network exists) rather than systemd, and recreates the network before Caddy is restarted. To recover by hand:

ssh emily
systemctl --user restart mozimo-network.service   # recreate mozimo_net
systemctl --user reset-failed caddy.service
systemctl --user start caddy.service

Migrating monoco → emily

monoco is the current host: rootful Docker, data in /var/lib/mozimo, its own Caddy, DNS for mozimo.in / www / cms / stats all pointing at 13.202.6.2. emily is 35.154.14.59.

Certificates are not copied. emily's Caddy is shared with other sites, and its ACME storage already holds their certificates — overwriting it would break them. After the DNS switch emily issues fresh certificates for mozimo.in; until the DNS switch it retries, which is why the Caddy role can be skipped first (--skip-tags caddy).

0. Compare the deployed environment with the repo's copy

The playbook uploads ansible/files/.env (gitignored) over whatever is on the host, so anything that has drifted on monoco would be silently dropped:

ssh monoco 'sudo -n cat /var/lib/mozimo/.env' > /tmp/monoco.env
diff /tmp/monoco.env ansible/files/.env

1. Rehearse against emily without touching DNS

cd ansible
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD) --check --diff   # what would change
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD) --skip-tags caddy

This creates the network, the units and a fresh, empty PocketBase database on emily. Nothing public changes yet.

2. Copy the data (the cutover window)

Stop both sides first. PocketBase keeps its database in SQLite, so a copy taken underneath a running writer can be inconsistent — and on emily the rehearsal left an empty database that this snapshot replaces:

# emily: stop its copy and drop the empty database the rehearsal created
ssh emily 'systemctl --user stop mozimo vince; rm -rf ~/podman-data/mozimo/pb_data ~/podman-data/vince'

# monoco: the site is down from here until DNS + emily are live
ssh monoco 'docker stop mozimo vince'

# PocketBase database + files (23 MB)
ssh monoco 'sudo -n tar -C /var/lib/mozimo -cf - pb_data' \
  | ssh emily 'mkdir -p ~/podman-data/mozimo && tar -C ~/podman-data/mozimo -xf -'

# Vince analytics history (119 MB), contents dropped directly into its data dir
ssh monoco 'sudo -n tar -C /var/lib/mozimo -cf - vince_data' \
  | ssh emily 'mkdir -p ~/podman-data/vince && tar -C ~/podman-data/vince -xf - --strip-components=1'

The extraction runs as tanshu, so everything lands owned by tanshu — which is exactly what the keep-id mapping expects, whatever it was owned by on monoco. Resumable equivalents, if the link is unreliable:

rsync -a --info=progress2 --rsync-path="sudo -n rsync" \
  monoco:/var/lib/mozimo/pb_data/   emily:podman-data/mozimo/pb_data/
rsync -a --info=progress2 --rsync-path="sudo -n rsync" \
  monoco:/var/lib/mozimo/vince_data/ emily:podman-data/vince/

3. Point DNS at emily and deploy the Caddy block

mozimo.in, www.mozimo.in, cms.mozimo.in, stats.mozimo.in → 35.154.14.59. Then:

cd ansible
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD)

# the play restarts a service whose unit or .env changed; starting is explicit
# here because step 2 stopped both, and `start` is a no-op if already running
ssh emily 'systemctl --user start mozimo vince'

The first request after the reload makes Caddy issue certificates for the four names.

4. Verify, then retire monoco

ssh emily 'systemctl --user status mozimo vince caddy --no-pager | head -30'
ssh emily 'podman ps --format "{{.Names}} {{.Status}}"'
curl -sI https://mozimo.in | head -3
curl -s https://cms.mozimo.in/api/health | head -3      # PocketBase
curl -sI https://stats.mozimo.in | head -3

Check a product page renders live prices and that the CMS admin login works with the same superuser credentials. Only then:

ssh monoco 'docker rm -f mozimo vince'     # or leave stopped for a few days

Rollback at any point before that: point DNS back at 13.202.6.2 and ssh monoco 'docker start mozimo vince'. On emily, systemctl --user stop mozimo vince and delete the mozimo.in block from its Caddyfile.