tanshu
c926f45faa
The deploy failed in the network role: the Caddy restart exited 125 with "unable to find network with name or ID mozimo_net", and with Restart=always and a 100ms delay the start limit was spent in under a second — taking every other site this host serves down with it. The network was gone, but mozimo-network.service was not: it is a oneshot with RemainAfterExit=yes, so systemd kept reporting it active after the network had been removed out of band. The generated Requires=mozimo-network.service looked satisfied, so the flushed "Restart Caddy" handler started Caddy against a network that no longer existed, and nothing recreated it. - ansible/roles/network/tasks/main.yaml: daemon-reload after templating the unit so it can be started on a first deploy; ask podman (`podman network exists`) rather than systemd whether the network is really there; restart mozimo-network.service when it is not; confirm it exists before the handler flush, so a network that cannot be repaired fails the play rather than the host's reverse proxy. - The Caddy restart condition includes the recreated case: a network that comes back keeps its name but not its identity, so Caddy's network namespace is stale even when neither template changed. - ansible/README.md: the oneshot/stale-state trap and the manual recovery. Verified on emily after recovering Caddy by hand: the play completes with 22 ok / 6 changed / 0 failed, and mozimo and vince run behind Caddy (TLS for mozimo.in still waits on the DNS cutover).
174 lines
6.9 KiB
Markdown
174 lines
6.9 KiB
Markdown
# Deploying mozimo
|
|
|
|
Rootless **podman + Quadlet** on the host `emily` (aarch64, Ubuntu 26.04),
|
|
fronted by that host's **shared Caddy**. Nothing here runs as root.
|
|
|
|
```bash
|
|
./deploy.sh # build the pushed commit + deploy
|
|
ansible-playbook playbook.yml -e image_tag=<commit> [--check --diff]
|
|
```
|
|
|
|
## How it fits together
|
|
|
|
| Piece | Where |
|
|
| --- | --- |
|
|
| Quadlet units | `~/.config/containers/systemd/{mozimo,vince}.container`, `mozimo.network` |
|
|
| Service data | `~/podman-data/mozimo/` (PocketBase db + files, `.env`), `~/podman-data/vince/` |
|
|
| Shared Caddy config | `~/podman-data/caddy/conf/Caddyfile` — one marked block per service |
|
|
| Caddy's network membership | `~/.config/containers/systemd/caddy.container.d/50-mozimo.conf` |
|
|
|
|
Quadlet units are not read by systemd directly: the podman user generator turns
|
|
them into *user services* on `daemon-reload`, and `[Install] WantedBy=default.target`
|
|
brings them up at boot. `systemctl --user is-enabled mozimo` reports `generated`
|
|
— that is normal, there is no unit to enable.
|
|
|
|
**Nothing is published on the host.** Caddy joins `mozimo_net` (through the
|
|
drop-in) and proxies to the containers by name:
|
|
|
|
```
|
|
mozimo.in → mozimo:3000 (plus /_/* and /api/* → mozimo:8090)
|
|
www.mozimo.in → 301 mozimo.in
|
|
cms.mozimo.in → mozimo:8090
|
|
stats.mozimo.in → vince:8000
|
|
mozimo.in/stats/* → vince:8000 (prefix stripped)
|
|
```
|
|
|
|
The user id mapping matters: the app image runs as uid 1000 (`node`) and Vince
|
|
as root, so both units use `UserNS=keep-id:` to map that uid onto `tanshu`.
|
|
Without it, PocketBase and Vince would write files owned by a subuid, and
|
|
backups, rsync and tar would all trip over them.
|
|
|
|
## Services register themselves
|
|
|
|
This host's convention (also used by its `tandoor`, `postgres` and `caddy`
|
|
playbooks) is that a service owns its own registration rather than editing a
|
|
central list:
|
|
|
|
1. it ships a `<name>.network` Quadlet with `NetworkName=<name>_net`;
|
|
2. it drops `caddy.container.d/50-<name>.conf` containing
|
|
`[Container] Network=<name>.network`, which Quadlet merges into Caddy;
|
|
3. it appends a marked block to Caddy's Caddyfile and reloads it.
|
|
|
|
Removing a service is the reverse: delete the drop-in, delete the Caddyfile
|
|
block, `systemctl --user daemon-reload`, and the network is no longer Caddy's.
|
|
|
|
`mozimo-network.service` is a `oneshot` with `RemainAfterExit=yes`, so systemd
|
|
keeps reporting it as `active (exited)` even if the network underneath it was
|
|
removed out of band (`podman network rm`, `podman network prune`, a wiped
|
|
`/run/user/<uid>`); restarting Caddy then starts it against a network that is
|
|
not there, and Caddy's own restart loop takes *every* site on the host down
|
|
with it. The playbook therefore asks podman (`podman network exists`) rather
|
|
than systemd, and recreates the network before Caddy is restarted. To recover
|
|
by hand:
|
|
|
|
```bash
|
|
ssh emily
|
|
systemctl --user restart mozimo-network.service # recreate mozimo_net
|
|
systemctl --user reset-failed caddy.service
|
|
systemctl --user start caddy.service
|
|
```
|
|
|
|
## Migrating monoco → emily
|
|
|
|
monoco is the current host: rootful Docker, data in `/var/lib/mozimo`, its own
|
|
Caddy, DNS for `mozimo.in` / `www` / `cms` / `stats` all pointing at
|
|
`13.202.6.2`. emily is `35.154.14.59`.
|
|
|
|
**Certificates are not copied.** emily's Caddy is shared with other sites, and
|
|
its ACME storage already holds their certificates — overwriting it would break
|
|
them. After the DNS switch emily issues fresh certificates for `mozimo.in`;
|
|
until the DNS switch it retries, which is why the Caddy role can be skipped
|
|
first (`--skip-tags caddy`).
|
|
|
|
### 0. Compare the deployed environment with the repo's copy
|
|
|
|
The playbook uploads `ansible/files/.env` (gitignored) over whatever is on the
|
|
host, so anything that has drifted on monoco would be silently dropped:
|
|
|
|
```bash
|
|
ssh monoco 'sudo -n cat /var/lib/mozimo/.env' > /tmp/monoco.env
|
|
diff /tmp/monoco.env ansible/files/.env
|
|
```
|
|
|
|
### 1. Rehearse against emily without touching DNS
|
|
|
|
```bash
|
|
cd ansible
|
|
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD) --check --diff # what would change
|
|
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD) --skip-tags caddy
|
|
```
|
|
|
|
This creates the network, the units and a **fresh, empty** PocketBase database
|
|
on emily. Nothing public changes yet.
|
|
|
|
### 2. Copy the data (the cutover window)
|
|
|
|
Stop both sides first. PocketBase keeps its database in SQLite, so a copy taken
|
|
underneath a running writer can be inconsistent — and on emily the rehearsal
|
|
left an empty database that this snapshot replaces:
|
|
|
|
```bash
|
|
# emily: stop its copy and drop the empty database the rehearsal created
|
|
ssh emily 'systemctl --user stop mozimo vince; rm -rf ~/podman-data/mozimo/pb_data ~/podman-data/vince'
|
|
|
|
# monoco: the site is down from here until DNS + emily are live
|
|
ssh monoco 'docker stop mozimo vince'
|
|
|
|
# PocketBase database + files (23 MB)
|
|
ssh monoco 'sudo -n tar -C /var/lib/mozimo -cf - pb_data' \
|
|
| ssh emily 'mkdir -p ~/podman-data/mozimo && tar -C ~/podman-data/mozimo -xf -'
|
|
|
|
# Vince analytics history (119 MB), contents dropped directly into its data dir
|
|
ssh monoco 'sudo -n tar -C /var/lib/mozimo -cf - vince_data' \
|
|
| ssh emily 'mkdir -p ~/podman-data/vince && tar -C ~/podman-data/vince -xf - --strip-components=1'
|
|
```
|
|
|
|
The extraction runs as `tanshu`, so everything lands owned by `tanshu` — which
|
|
is exactly what the `keep-id` mapping expects, whatever it was owned by on
|
|
monoco. Resumable equivalents, if the link is unreliable:
|
|
|
|
```bash
|
|
rsync -a --info=progress2 --rsync-path="sudo -n rsync" \
|
|
monoco:/var/lib/mozimo/pb_data/ emily:podman-data/mozimo/pb_data/
|
|
rsync -a --info=progress2 --rsync-path="sudo -n rsync" \
|
|
monoco:/var/lib/mozimo/vince_data/ emily:podman-data/vince/
|
|
```
|
|
|
|
### 3. Point DNS at emily and deploy the Caddy block
|
|
|
|
`mozimo.in`, `www.mozimo.in`, `cms.mozimo.in`, `stats.mozimo.in` →
|
|
`35.154.14.59`. Then:
|
|
|
|
```bash
|
|
cd ansible
|
|
ansible-playbook playbook.yml -e image_tag=$(git rev-parse --short HEAD)
|
|
|
|
# the play restarts a service whose unit or .env changed; starting is explicit
|
|
# here because step 2 stopped both, and `start` is a no-op if already running
|
|
ssh emily 'systemctl --user start mozimo vince'
|
|
```
|
|
|
|
The first request after the reload makes Caddy issue certificates for the four
|
|
names.
|
|
|
|
### 4. Verify, then retire monoco
|
|
|
|
```bash
|
|
ssh emily 'systemctl --user status mozimo vince caddy --no-pager | head -30'
|
|
ssh emily 'podman ps --format "{{.Names}} {{.Status}}"'
|
|
curl -sI https://mozimo.in | head -3
|
|
curl -s https://cms.mozimo.in/api/health | head -3 # PocketBase
|
|
curl -sI https://stats.mozimo.in | head -3
|
|
```
|
|
|
|
Check a product page renders live prices and that the CMS admin login works
|
|
with the same superuser credentials. Only then:
|
|
|
|
```bash
|
|
ssh monoco 'docker rm -f mozimo vince' # or leave stopped for a few days
|
|
```
|
|
|
|
**Rollback** at any point before that: point DNS back at `13.202.6.2` and
|
|
`ssh monoco 'docker start mozimo vince'`. On emily, `systemctl --user stop
|
|
mozimo vince` and delete the `mozimo.in` block from its Caddyfile.
|