Add the Yamtrack playbook

This commit is contained in:
tanshu committed 2026-10-18 21:35:53 +05:30
1 parent 26a0654fea
commit af2e7394b8
11 files changed
+1023

No files matched your search

+54
View File
@@ -0,0 +1,54 @@
---
# Yamtrack as a rootless Podman Quadlet unit, behind the shared Caddy unit
# (caddy/playbook.yml), with its cache in the shared Valkey unit
# (valkey/playbook.yml). The database stays SQLite, inside the app's own data
# directory — see readme.md, "The database stays SQLite".
#
# Connects as the container user and runs everything as that user: the Quadlet
# units belong to it, so the connection's PAM session is what makes
# `systemctl --user` work without any environment juggling.
#
# gather_facts is required: podman_user and the home directory come from the
# user_id / user_dir facts.
#
# The rootless-Podman prerequisites (installing podman, enabling lingering, the
# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied
# here. They are the shared task file at caddy/roles/podman/tasks/main.yaml, and
# a host that has already run caddy/playbook.yml has them:
#
# - name: Rootless Podman prerequisites
# ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
#
# The variables this play needs are duplicated at the top of vars/default.yml, so
# it stands on its own.
#
# Run with: ansible-playbook -u tanshu yamtrack/playbook.yml
- name: Deploy Yamtrack as a rootless Podman Quadlet
hosts: emily
gather_facts: true
vars_files:
- vars/default.yml
pre_tasks:
# This must happen before the app unit is started, not after: the generated
# unit references {{ network_name }}.network, and Quadlet fails *unit
# generation* (so `systemctl --user start` reports "unit not found") when a
# referenced .network file does not exist yet. The same include writes the
# Caddy-side drop-in that attaches Caddy to this network, and the Valkey
# attach below needs the network unit to exist as well.
#
# network_name is a play var, so the include can see it directly.
- name: Register Yamtrack's network with Caddy
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
roles:
# Attaches this app's network to the shared Valkey unit, which is what makes
# REDIS_URL's host resolve, and owns the "Restart Valkey" handler that
# valkey/roles/attach notifies. Runs first so the cache is reachable before
# the app starts.
- valkey
# The app: data directory, the 0600 environment file, the Quadlet unit,
# start + verify.
- yamtrack
# Appends this site's block to the shared Caddyfile and reloads Caddy.
- caddy
+518
View File
@@ -0,0 +1,518 @@
# Yamtrack (rootless Podman + Quadlet)
Runs [Yamtrack](https://github.com/FuzzyGrim/Yamtrack) on `emily` as a
**rootless Podman** container managed by a **systemd user unit** (Quadlet),
behind the shared [rootless Caddy](../caddy/readme.md), with its cache and Celery
broker in the shared [rootless Valkey](../valkey/readme.md). Replaces the
`docker_container` deployment on `monoco`.
The database stays **SQLite**, inside the app's own data directory
(`~/podman-data/yamtrack/data/db.sqlite3`). That is deliberate, not an
oversight — see [The database stays SQLite](#the-database-stays-sqlite).
Everything else about the old container is reproduced rather than redesigned:
the same image tag, the same four environment variables, the same upstream
(`yamtrack:8000`), the same network name (`yamtrack_net`) and the same
`SECRET`.
## Ordering
Both shared services are already migrated and running on `emily`:
1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns
the Caddyfile this site appends to, supplies the `caddy.service` handlers, and
is the one play that applies the rootless-Podman prerequisites.
2. [valkey/playbook.yml](../valkey/playbook.yml) — the shared cache and Celery
broker. **Yamtrack hard-requires Redis**: its cache backend is `django_redis`
with no fallback, Celery worker and beat need a broker, and several metadata
providers talk to Redis directly. The old container was given
`redis://valkey:6379` and that is exactly what this playbook sets.
3. This playbook — **run it after the file copy below**, not before: the
entrypoint migrates the schema on first start, so an empty
`~/podman-data/yamtrack/data` must not be what the app wakes up to.
[postgres/playbook.yml](../postgres/playbook.yml) is **not** part of this
deployment: there is no `yamtrack` database in the restored cluster, because the
app never used Postgres.
This playbook attaches `yamtrack_net` to Valkey. Until it has, `valkey` does not
resolve on that network, so running this playbook is what makes the app
reachable — there is no separate step.
## Layout
```
yamtrack/
├── playbook.yml
├── vars/default.yml
└── roles/
├── valkey/ # attaches this app's network to the shared cache (+ its handler)
├── yamtrack/ # data dir, the 0600 env file, the Quadlet unit, start + verify
└── caddy/ # appends this site's block to the shared Caddyfile
```
Quadlet units and state:
| On the host | What |
|---|---|
| `~/.config/containers/systemd/yamtrack.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/yamtrack.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-yamtrack.conf` | attaches Caddy to that network |
| `~/.config/containers/systemd/valkey.container.d/50-yamtrack.conf` | attaches Valkey to it |
| `~/.config/containers/systemd/valkey.container.d/10-network-aliases.conf` | the shared `redis` alias |
| `~/podman-data/yamtrack/data/` | the image's `/yamtrack/db`: `db.sqlite3` (+ `-wal`/`-shm`) |
| `~/podman-data/yamtrack/yamtrack.env` | `SECRET`, `TZ`, `URLS`, `REDIS_URL`, `PUID`/`PGID`, mode `0600`, read by systemd |
Two Docker-era pieces are **deleted rather than ported**: `roles/network`, which
existed to attach an already-created container to a network
(`docker_network`'s `connected:`/`appends:`, which `podman_network` has no
equivalent for — [docs/podman-compat-audit.md §3.3](../docs/podman-compat-audit.md)),
and `files/nginx.conf.j2`, which was a copy-pasted *Vaultwarden* nginx template
that this playbook never used. Nothing has to be connected after the fact now:
the app's own unit names its `Network=`, and Caddy and Valkey learn about the
network from drop-ins written in the same run.
## How it registers with Caddy
Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:
```yaml
- name: Register Yamtrack's network with Caddy
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```
It runs in `pre_tasks`, **before** the app unit is started, because
`yamtrack.container` references `yamtrack.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet. The Valkey attach below depends on
it for the same reason.
`network_name` is `yamtrack`, so the *file* is `yamtrack.network` and the *Podman
network* is `yamtrack_net` (`NetworkName=` in the network unit pins it, keeping
the Docker-era name).
The site block itself is appended to the shared Caddyfile by
[roles/caddy](roles/caddy/tasks/main.yaml) with `blockinfile` — the same bytes
the Docker-era Caddyfile on `monoco` already carries for this host:
```
track.tanshu.com {
reverse_proxy yamtrack:8000
}
```
`8000` is the port the image's own nginx listens on inside the container (it
proxies to gunicorn); the image `EXPOSE`s exactly that, and
`vars/default.yml`'s `container_port` is where it is written down. The old
`roles/caddy` slurped and compared the Caddyfile before editing it, which
*fails* on a host where Caddy has not run yet; `blockinfile` is idempotent on
its own marker, so that whole dance is gone.
## How it attaches to Valkey
Same idea at the other end: Valkey knows nothing about its clients, and this
playbook uses the shared include in
[valkey/roles/attach](../valkey/roles/attach/tasks/main.yaml) rather than writing
those drop-ins itself. That writes two files, both mergeable into
`valkey.container` by Quadlet:
| File | Content |
|---|---|
| `valkey.container.d/50-yamtrack.conf` | `Network=yamtrack.network` |
| `valkey.container.d/10-network-aliases.conf` | `NetworkAlias=redis` |
`REDIS_URL` uses the host `valkey`, which is `ContainerName=valkey` and is what
the Docker-era container was given. The `redis` alias in the shared file is for
the *other* cache clients (Nextcloud writes `REDIS_HOST=redis` into its
`config.php`), not for this app; the play checks that `valkey` resolves.
[roles/valkey](roles/valkey/tasks/main.yaml) wraps that include for two reasons:
- it defines the `Restart Valkey` handler the include notifies; and
- it **flushes handlers** immediately afterwards. On a first attach the drop-in
is new, so Valkey has to be *recreated* to join `yamtrack_net` — and that
restart is a handler, which by default runs at the end of the play, after the
`yamtrack` role has already started the app. The app would then come up before
`valkey` resolves, fail its own health check, and fail the play on a machine
where nothing is wrong. Flushing applies the drop-in first; on later runs
nothing is pending and it is a no-op.
Unlike Postgres, the cost here is **not** downtime for a database: Valkey is a
cache, so every attached app simply starts with a cold cache and the first
requests after the restart are slower. See
[valkey/readme.md](../valkey/readme.md), "The trade-off". Attached means *every*
cache client, not just Yamtrack.
## Who runs as whom
`UserNS=keep-id:uid=0,gid=0` — container root is mapped onto `podman_user`, the
same choice Caddy, Vaultwarden and Tandoor make. That is what keeps
`~/podman-data/yamtrack` owned by `podman_user` rather than by a subuid, so
plain `tar`, `cp` and `rm -rf` work there without `podman unshare` or `sudo`,
and extracting the old data as `podman_user` is already correct.
**uid 0 is required here, not a convenience.** `--userns=keep-id` does two
things: it maps one container uid onto the connecting user, *and it runs the
container as that uid*. Yamtrack's entrypoint begins by rewriting `/etc/group`
and `/etc/passwd` (`groupmod`/`usermod`, applying `PUID`/`PGID`) — files owned by
container-root in the image. With `keep-id:uid=1000` the entrypoint runs as
`abc`, cannot lock `/etc/group`, and the unit dies with:
```
groupmod: /etc/group.5: Permission denied
groupmod: cannot lock /etc/group; try again later.
yamtrack.service: Main process exited, code=exited, status=10/n/a
```
before supervisord is ever reached. `manage.py migrate` has *already* run and
succeeded at that point, because the bind mount is writable as `abc` — which
makes the failure look like a database problem when it is not. Docker on
`monoco` ran the same entrypoint as real root, which is why it never surfaced
there.
`PUID`/`PGID` are therefore pinned to the same `0` in the environment file. The
entrypoint's own default is `1000`, and under this mapping container uid `1000`
is a subuid on the host, so its `chown -R abc:abc db` would leave the database
owned by e.g. `165536` — unreadable by a plain `tar` backup. With both pinned to
`0`, `abc` resolves to container root, which `keep-id` maps onto `podman_user`.
The entrypoint still chowns `db` on every start, so the copy only has to be
*readable* by `podman_user`.
The one visible side effect is this warning from Celery, once per start, in
`podman logs yamtrack`:
```
SecurityWarning: You're running the worker with superuser privileges
```
That is Celery spotting uid 0 *inside the container*, where uid 0 is an
unprivileged user on the host. Expected, and harmless in a rootless unit.
Like Caddy, `podman_user` is derived from the connection
(`ansible_facts['user_id']`) and the play carries no `become`. Because the podman
tasks are not run here, there is **no guard against connecting as `root`**; the
prerequisites, including that check, live in `caddy/playbook.yml`.
## The database stays SQLite
Yamtrack supports Postgres (`DB_HOST`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`,
`DB_PORT`), and this repo now has a shared Postgres unit. It is still not used
here, on purpose:
- **The old data is SQLite.** 2.3 MB in one `db.sqlite3` file, owned by uid
`1000` ([docs/backup-plan.md §2.1](../docs/backup-plan.md)). Moving it to
Postgres is a *data conversion* — `dumpdata`/`loaddata` or a
`sqlite`→`postgres` copy — with its own failure modes, and it buys nothing
this deployment needs.
- **There is no `yamtrack` database in the restored cluster.** The twelve
databases brought across in
[postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md) do not
include one, which is the same fact seen from the other side.
- **A single file is not a worse fit here than a shared server.** The app is a
one-user media tracker; the whole database is smaller than a single query
result in some of the other services.
The consequence to be deliberate about: **the entrypoint runs
`manage.py migrate` on every start**, so an image bump with `latest` can migrate
the schema, and that migration is not reversible by rolling the image back. The
copy in [Migrating the file data](#migrating-the-file-data-from-monoco) is the
rollback, which is one more reason to take it before the first start and to keep
`monoco` intact until this deployment has been lived with.
Moving to Postgres later is a normal, separate change: point the app at it,
convert the data, and add a `postgres` role here exactly like Nextcloud's or
Tandoor's (`postgres/roles/attach` plus a handler and a flush).
## Settings
| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `track.tanshu.com` — the site block Caddy serves |
| `container_name` / `unit_name` | `yamtrack`, and the Quadlet unit / container name |
| `image` | `ghcr.io/fuzzygrim/yamtrack:latest` — fully qualified on purpose |
| `network_name` | `yamtrack` → `yamtrack.network` / `yamtrack_net` |
| `host_directory` | `~/podman-data/yamtrack` |
| `data_dir` | `~/podman-data/yamtrack/data` → the image's `/yamtrack/db` |
| `container_port` | `8000`, the port inside the container that Caddy proxies to |
| `env_file` | `~/podman-data/yamtrack/yamtrack.env`, mode `0600` |
| `app_uid` / `app_gid` | `0` — container root, mapped by `keep-id` onto `podman_user`, and the entrypoint's `PUID`/`PGID` |
| `yamtrack_tz` | `Asia/Kolkata`, the `TZ` the old container was given |
| `yamtrack_secret` | The Django `SECRET`, carried over verbatim |
| `urls` | `https://track.tanshu.com`, the old container's `URLS` (CSRF origins) |
| `valkey_container` / `valkey_unit` | The shared cache unit this app attaches to |
| `redis_url` | `redis://valkey:6379`, unchanged from the Docker-era container |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |
The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run — Yamtrack publishes nothing host-side.
`SECRET`, `TZ` and `REDIS_URL` are written to `yamtrack.env` at mode `0600` and
read by systemd via `EnvironmentFile=`, not by Ansible. `SECRET` is the same
value `monoco` uses, so existing sessions and password-reset tokens keep working
— rotating it logs every user out. It is committed in plaintext as before;
flagged as a follow-up, not changed here.
`ALLOWED_HOSTS` is deliberately absent: Yamtrack's default is `*`, which is what
the Docker-era container relied on. Setting `URLS` — as the old playbook did — is
what fills in the CSRF trusted origins.
## Running
```command
ansible-playbook -u tanshu yamtrack/playbook.yml
```
The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.
This play applies **only** Yamtrack's own units and data. Its `pre_tasks` run
Caddy's `register` include and nothing else. The rootless-Podman prerequisites —
installing podman, `loginctl enable-linger`, the low-port sysctl and the Quadlet
directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.
Neither is Valkey deployed here: run `valkey/playbook.yml` first, or the flush in
`roles/valkey` fails on a `valkey.service` that does not exist.
`systemctl --user enable yamtrack` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.
## Migrating the file data from monoco
There is no database to restore from Postgres for this app: the whole of its
state is `db.sqlite3`, and the playbook does not move it because the file is on
the old host while the playbook runs on the new one. Run this from the machine
that holds the `monoco` and `emily` SSH aliases.
**Copy before deploying.** The entrypoint migrates the schema on first start, so
a first run against an empty directory creates a new, empty database and the
copy would then be overwriting a live one.
If a first run has *already* happened — the unit crash-looped, or a deploy was
tried before the copy — that directory now holds a fresh database with every
migration applied and no rows. Delete it before copying rather than copying on
top of it, so no `-wal`/`-shm` from the empty database can mix with the real
one:
```bash
# Prove it is the empty one before removing it: 0 users, 0 items, 0 episodes.
ssh emily 'podman exec -i yamtrack python3 -' <<'PY'
import sqlite3
c = sqlite3.connect("/yamtrack/db/db.sqlite3")
for t in ("users_user", "app_item", "app_episode"):
print(t, c.execute("select count(*) from " + t).fetchone()[0])
PY
ssh emily 'systemctl --user stop yamtrack.service; rm -rf ~/podman-data/yamtrack/data'
```
```bash
# 1. Look before you copy: size, contents, and whether it is readable without
# sudo. `ls -ln` shows numeric owners — the old data is uid 1000, which is
# `ubuntu` on monoco and the container's `abc` inside it.
ssh monoco 'du -sh /var/lib/yamtrack/data; ls -ln /var/lib/yamtrack/data'
ssh monoco 'find /var/lib/yamtrack/data -type f ! -readable' # empty = no sudo needed
# 2. Stop the old container so nothing writes while you copy. Leave it stopped:
# two live copies of one SQLite database would diverge.
#
# This is also what makes the copy *consistent*. SQLite runs in WAL mode here,
# and a plain tar of a live database can produce a file that will not open —
# the main file and the WAL are copied at different instants
# (docs/backup-plan.md §3.1). A clean stop checkpoints the WAL into
# db.sqlite3 and removes the sidecars, so the copy is a single frozen file.
ssh monoco 'docker stop yamtrack'
# 3. Create the destination.
ssh emily 'mkdir -p ~/podman-data/yamtrack/data'
# 4. Copy. tar streams it across the workstation; rsync cannot copy
# remote-to-remote in one command.
#
# The -wal/-shm sidecars are copied rather than filtered out on purpose: if
# anything is still in the WAL, it is part of the database. After a clean stop
# they are usually already gone, and copying "nothing" costs nothing.
#
# `set -o pipefail` matters here: without it the pipeline reports only the
# *receiving* tar's status, so a read error or a dropped stream on the source
# side exits 0 and leaves a silently short copy.
set -o pipefail
ssh monoco 'tar -C /var/lib/yamtrack/data -cf - .' \
| ssh emily 'tar --no-same-owner -C ~/podman-data/yamtrack/data -xf -'
# If step 1 listed unreadable files, read them as root instead:
# ssh monoco 'sudo tar -C /var/lib/yamtrack/data -cf - .' \
# | ssh emily 'tar --no-same-owner -C ~/podman-data/yamtrack/data -xf -'
# 5. Verify by content. Both sides are frozen by now (the old container is
# stopped and nothing writes on emily yet), so identical digests mean a
# byte-faithful copy.
ssh monoco 'cd /var/lib/yamtrack/data && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
ssh emily 'cd ~/podman-data/yamtrack/data && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# 6. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/yamtrack ! -user tanshu' # empty = correct
# 7. Deploy.
cd prepare-server && ansible-playbook -u tanshu ../yamtrack/playbook.yml
```
Ownership needs no `chown` pass: the app runs as uid `1000` in-container, which
`keep-id` maps to `tanshu`, so files extracted as `tanshu` are already right, and
the entrypoint's own `chown abc:abc db` runs on every start anyway. That is also
why `--no-same-owner` is explicit on the receiving side — the source is owned by
uid `1000`, and restoring that ownership would be wrong here.
If the link is slow or drops, do it in two hops instead: resumable, and it shows
progress.
```bash
rsync -aP monoco:/var/lib/yamtrack/data/ ~/yamtrack-copy/
rsync -aP ~/yamtrack-copy/ emily:~/podman-data/yamtrack/data/
```
If stopping the app is unacceptable — for instance while rehearsing this
deployment before the DNS move — ask SQLite for a consistent snapshot instead of
copying the files while they are hot:
```bash
ssh monoco 'docker exec -i yamtrack python3 -' <<'PY'
import sqlite3
sqlite3.connect("/yamtrack/db/db.sqlite3").execute(
"VACUUM INTO '/yamtrack/db/pre-migration.sqlite3'"
)
print("wrote /yamtrack/db/pre-migration.sqlite3")
PY
# then copy it as the database, and delete the snapshot on monoco afterwards
ssh monoco 'cat /var/lib/yamtrack/data/pre-migration.sqlite3' \
| ssh emily 'cat > ~/podman-data/yamtrack/data/db.sqlite3'
```
`VACUUM INTO` needs SQLite 3.27+; the image's Python 3.12 has far newer. This
still has to be followed by stopping the container before DNS moves: it shortens
the window, it does not remove the cutover.
Rolling back means rolling back both halves: `ssh monoco 'docker start yamtrack'`
is only meaningful while `monoco`'s data and reverse proxy are intact, and the
DNS record has to point back. `monoco` was never modified — its container was
only stopped — so the rollback is starting it again.
## Verification
```bash
systemctl --user status yamtrack
journalctl --user -u yamtrack -f
podman logs yamtrack
# `podman ps` shows (healthy) once the image's own probe passes.
podman ps --filter name=yamtrack
podman healthcheck run yamtrack
# When it does not pass, the health view names the failing check — cache,
# database, Redis or Celery — instead of just "unhealthy":
podman exec yamtrack wget -qO- http://127.0.0.1:8000/health/
# What the playbook checks for you:
podman exec yamtrack getent hosts valkey
# The file really is the migrated database, untouched by a first-start migrate:
podman exec yamtrack python3 -c "import sqlite3; print(sqlite3.connect('/yamtrack/db/db.sqlite3').execute('pragma integrity_check').fetchone()[0])"
# Row counts compared with monoco — equal numbers mean the library came across.
# (1517 items and 1312 episodes when this was written. Use `docker exec` on
# monoco, `podman exec` on emily.)
ssh monoco 'docker exec yamtrack python3 -c "import sqlite3; print(sqlite3.connect(\"/yamtrack/db/db.sqlite3\").execute(\"select count(*) from app_item\").fetchone()[0])"'
ssh emily 'podman exec yamtrack python3 -c "import sqlite3; print(sqlite3.connect(\"/yamtrack/db/db.sqlite3\").execute(\"select count(*) from app_item\").fetchone()[0])"'
# The site, end to end:
curl -sSI https://track.tanshu.com
```
Then log in **in a browser** and confirm the library, an episode's history and
one metadata lookup feel right — the health check proves the process, not the
data. Back it up with a **write**: mark an episode watched, then confirm on the
host that `db.sqlite3` changed and is owned by `tanshu`:
```bash
ssh emily 'ls -ln ~/podman-data/yamtrack/data'
```
A write landing as root or as a `165536+` subuid is the `keep-id` check, and a
read-only smoke test will not catch it.
Then reboot the host and confirm the unit returns unattended — without
`loginctl enable-linger tanshu` it will not, and the failure is silent.
## Notes and gotchas
- **`SECRET` is committed in plaintext**, as it was before. It is the same value
`monoco` used, so sessions survive the move; changing it is a separate,
user-visible decision (everyone is logged out, pending password-reset links
break).
- **`latest` floats, and the entrypoint migrates.** Neither this playbook nor the
old one pins a digest or a version, so an image bump can run new Django
migrations against the SQLite file on first start. Rolling the image back does
not roll the schema back; the copy taken in
[Migrating the file data](#migrating-the-file-data-from-monoco) is the
rollback. This is not hypothetical: `monoco` is running an image pulled in
April (`v0.25.2`, 200 applied migrations) while `emily` pulled a newer
`latest`, so the copied database gets 12 migrations on its first start under
the new unit.
- **The entrypoint needs root, so `keep-id`'s uid matters.** See
[Who runs as whom](#who-runs-as-whom) — `keep-id:uid=1000` produces a crash
loop whose log looks like a database or permission problem and is neither.
After that failure the data directory contains a **fresh, empty** database
(0 users, 0 items) with every migration applied, which will happily serve an
empty library if the copy has not landed. Check
`select count(*) from app_item` before pointing DNS anywhere.
- **Yamtrack requires Redis.** There is no "cache disabled" mode: the cache
backend is `django_redis` and Celery needs the broker. If `valkey` is down, the
site does not degrade — it fails.
- **`staticfiles/` is not on the host.** Unlike Tandoor, this image bakes its
collected static files in, so `~/podman-data/yamtrack/data` is the whole of
what has to be backed up. The old container's `/yamtrack` showed
`staticfiles/` owned by `abc`; that is image content, not state.
- **Client IPs.** Yamtrack's nginx sits behind Caddy, and whether the real client
address survives the rootless port forward depends on the forwarder. See
[caddy/readme.md](../caddy/readme.md), "Client IPs", before relying on it.
- **Registering restarts all of Caddy**, and attaching restarts Valkey for every
cache client — a brief restart of every hosted site, and a cold cache
everywhere. See the trade-offs in [caddy/readme.md](../caddy/readme.md) and
[valkey/readme.md](../valkey/readme.md).
- **If the Caddyfile does not exist yet**, this playbook creates it containing
only the site block. That is a valid Caddyfile; the Caddy playbook's seeded
global-options block is then not added, because its seed only fires on a
missing or zero-length file. Deploy Caddy first on a fresh host.
- **DNS.** `track.tanshu.com` currently points at `monoco` (`13.202.6.2`) and
must be repointed at `emily` (`35.154.14.59`) before Caddy can obtain a
certificate for it. Until then the site answers from the old host.
- **Do not restart the container on `monoco`** after the cutover. It would keep
writing to the old database, and two live copies behind one hostname diverge
silently.
- **There is no `yamtrack-backup.sh`.** `~/bin` on `monoco` has scripts for
Nextcloud, Gitea, Bitwarden, Tandoor and others, but none for this app, and
root's crontab has only the Nextcloud entry — so nothing has ever been
scheduled ([docs/backup-plan.md §11](../docs/backup-plan.md)). Nothing has to
be ported here, but the 2.3 MB file that *is* this service has no backup
either: a `systemd --user` timer doing `podman exec yamtrack python3 -c "…"
"VACUUM INTO …"` next to the Quadlet units is the shape that fits this host,
because root cannot `podman exec` into another user's rootless containers.
## Not done here
- Rotating `SECRET` and the other secrets that are committed in plaintext.
- Moving the database to the shared Postgres unit (see
[The database stays SQLite](#the-database-stays-sqlite)).
- Replacing the missing backup with a systemd user timer (see above).
+31
View File
@@ -0,0 +1,31 @@
---
# Caddy is its own playbook (caddy/playbook.yml) and a rootless Quadlet unit
# there. These handlers run as the connection user, so `systemctl --user` and
# `podman exec` work without any XDG_RUNTIME_DIR juggling.
# Notified by this role when the Caddyfile changes. A reload keeps existing
# connections; a restart would briefly drop every hosted site.
#
# Note the config path: /etc/caddy/Caddyfile is the mount inside the container,
# not {{ caddyfile_path }}, which is the host side of it.
- name: Reload Caddy configuration
ansible.builtin.command: >-
podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
changed_when: true
listen: "Reload Caddy"
# Notified by caddy/roles/register when this service's network unit, or the
# drop-in that attaches Caddy to it, is written. Quadlet has no live equivalent
# of `docker network connect`, so the container has to be recreated — a brief
# restart of every hosted site, not just this one. See caddy/readme.md,
# "Trade-off worth knowing".
#
# Defined after the reload above so that when both are notified the cheap
# config reload happens first.
- name: Restart Caddy
ansible.builtin.systemd_service:
name: "{{ caddy_name }}.service"
scope: user
state: restarted
daemon_reload: true
listen: Restart Caddy
+22
View File
@@ -0,0 +1,22 @@
---
# Append this site's block to the shared Caddyfile.
#
# The Caddyfile is a bind mount into the Caddy container, so Caddy sees the
# change as soon as the file is written; the handler only asks Caddy to re-read
# it. That is a reload, not a restart, so the other hosted sites are untouched.
#
# blockinfile is idempotent on its own via the marker, so there is no need to
# slurp the file and compare first — which also matters because slurp *fails*
# when the Caddyfile does not exist yet, on a host where the Caddy playbook has
# not run. (The Docker-era version of this role did exactly that.)
- name: "Add the {{ host }} site block to the Caddyfile"
ansible.builtin.blockinfile:
path: "{{ caddyfile_path }}"
marker: "# {mark} Ansible managed Caddy snippet for {{ host }}"
block: "{{ snippet_block }}"
create: yes
owner: "{{ podman_user }}"
group: "{{ podman_user }}"
mode: '0644'
notify: Reload Caddy
+20
View File
@@ -0,0 +1,20 @@
---
# Valkey is its own playbook (valkey/playbook.yml) and a rootless Quadlet unit
# there. This handler runs as the connection user, so `systemctl --user` works
# without setting XDG_RUNTIME_DIR.
#
# Required by valkey/roles/attach, which notifies it whenever this app's drop-in
# changes: a new drop-in changes the generated unit, so Valkey must be recreated
# to join the network — Quadlet has no live equivalent of
# `podman network connect`.
#
# The cost is a cold cache for every attached app, not lost data — but note that
# attached means every cache client, not just Yamtrack. See valkey/readme.md,
# "The trade-off".
- name: Restart Valkey
ansible.builtin.systemd_service:
name: "{{ valkey_unit }}"
scope: user
state: restarted
daemon_reload: true
listen: Restart Valkey
+31
View File
@@ -0,0 +1,31 @@
---
# Attach this app's network to the shared Valkey unit.
#
# Valkey knows nothing about its clients, exactly like Caddy and Postgres: a
# service joins by dropping 50-yamtrack.conf and the shared
# 10-network-aliases.conf next to valkey.container. The canonical way to do that
# is the shared include below — it writes the same bytes every service writes,
# asserts that yamtrack.network already exists, and notifies the "Restart Valkey"
# handler defined in this role.
#
# `valkey` is the host REDIS_URL asks for — the container name, not the shared
# `redis` alias, which is there for the clients whose config.php says `redis`.
# The Docker-era container was given `redis://valkey:6379` too, so the URL is
# unchanged.
#
# This runs in the roles phase, which is after the register include in
# playbook.yml's pre_tasks has created yamtrack.network.
- name: Attach Yamtrack to Valkey
ansible.builtin.include_tasks: "{{ playbook_dir }}/../valkey/roles/attach/tasks/main.yaml"
# On a first attach the drop-in is new, so Valkey has to be recreated to join
# yamtrack_net — and that restart is a *handler*, which by default runs at the
# end of the play, after the yamtrack role has already started the app. The app
# would then come up before `valkey` resolves, fail its own checks, and fail the
# play on a machine where nothing is actually wrong.
#
# Flushing applies the new drop-in before the app starts. On later runs the
# drop-in is unchanged, nothing is pending, and this is a no-op.
- name: Apply the Valkey network change before starting Yamtrack
ansible.builtin.meta: flush_handlers
@@ -0,0 +1,20 @@
---
# All handlers run as the connection user ({{ podman_user }}), so systemctl
# --user works without setting XDG_RUNTIME_DIR.
- name: Reload systemd user units
ansible.builtin.command: systemctl --user daemon-reload
changed_when: true
listen: Reload systemd user units
# A changed environment file or unit file changes the generated unit, so
# Yamtrack is recreated. The database is a bind mount and the rest of /yamtrack
# is image content, so a recreate loses nothing — but it is a few seconds of
# downtime for the one site.
- name: Restart Yamtrack
ansible.builtin.systemd_service:
name: "{{ unit_name }}"
scope: user
state: restarted
daemon_reload: true
listen: Restart Yamtrack
+121
View File
@@ -0,0 +1,121 @@
---
# Yamtrack as a rootless Podman Quadlet unit.
#
# Everything here runs as {{ podman_user }} — the connection user. No become, no
# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
# user manager is reachable through the standard environment.
# /yamtrack/db is the image's state directory: db.sqlite3 plus its -wal/-shm
# sidecars, and nothing else worth keeping — staticfiles/ is built into the
# image and the code is read-only.
- name: Ensure the Yamtrack data directory exists
ansible.builtin.file:
path: "{{ data_dir }}"
state: directory
owner: "{{ podman_user }}"
group: "{{ podman_user }}"
mode: '0755'
# The app's secrets and cache coordinates are consumed by systemd
# (EnvironmentFile= in the Quadlet unit), not by Ansible, so this has to be
# written before the unit starts. Mode 0600: it holds SECRET.
- name: Write the environment file
ansible.builtin.template:
src: yamtrack.env.j2
dest: "{{ env_file }}"
owner: "{{ podman_user }}"
group: "{{ podman_user }}"
mode: '0600'
notify: Restart Yamtrack
- name: Install the Yamtrack Quadlet unit
ansible.builtin.template:
src: yamtrack.container.j2
dest: "{{ quadlet_dir }}/{{ container_name }}.container"
owner: "{{ podman_user }}"
group: "{{ podman_user }}"
mode: '0644'
notify:
- Reload systemd user units
- Restart Yamtrack
- name: Start Yamtrack
block:
- name: Start Yamtrack
ansible.builtin.systemd_service:
name: "{{ unit_name }}"
scope: user
state: started
daemon_reload: true
rescue:
# systemd reports "control process exited with error code" and nothing
# else, which is useless for the usual failures (an unattached network, an
# unreachable cache, a bad volume). Pull the real reason out.
- name: Collect Yamtrack diagnostics
ansible.builtin.command: "{{ item }}"
loop:
- "systemctl --user status {{ unit_name }} --no-pager"
- "journalctl --user -u {{ unit_name }} --no-pager -n 40"
- "podman logs --tail 40 {{ container_name }}"
register: yamtrack_diag
changed_when: false
failed_when: false
- name: Show Yamtrack diagnostics
ansible.builtin.debug:
msg: "{{ yamtrack_diag.results | map(attribute='stdout') | list }}"
- name: Fail
ansible.builtin.fail:
msg: "Yamtrack did not start — see the diagnostics above."
# Quadlet honours [Install] WantedBy= when it generates the unit, so
# `systemctl --user enable` is neither needed nor possible on a generated unit.
#
# `is-active` says the container is running; it does not say the app can serve.
# The entrypoint runs migrations before supervisord starts nginx, gunicorn,
# celery and beat, and `Restart=always` means even a crash-looping container is
# reported active — so the checks below are the ones that matter.
- name: Verify Yamtrack is running
ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
register: yamtrack_active
changed_when: false
failed_when: yamtrack_active.stdout != 'active'
# `valkey` is the host the environment file actually asks for; it resolves from
# ContainerName= on the shared unit, which valkey/roles/attach joined to this
# app's network.
- name: Confirm Yamtrack can resolve its cache
ansible.builtin.command: >-
podman exec {{ container_name }} getent hosts {{ valkey_container }}
register: yamtrack_cache_lookup
changed_when: false
failed_when: false
- name: Report cache resolution
ansible.builtin.debug:
msg: >-
{{ valkey_container }} resolves as
{{ yamtrack_cache_lookup.stdout | default('(no answer)') }}
# `is-active` says the container is running; the image's own HEALTHCHECK is what
# says the app works. It wgets http://127.0.0.1:8000/health/, and that view
# checks all four things this deployment can get wrong — the cache, the database,
# Redis and Celery — so a green result here is the app being genuinely ready.
#
# Running the image's check rather than one written here means depending on
# nothing the play has to know: no curl or wget wrapper of our own, and no
# hardcoded path that a later image tag could move. `podman exec <c> wget -qO-
# http://127.0.0.1:{{ container_port }}/health/` prints the per-check breakdown
# when this fails; see readme.md, "Verification".
#
# The start period (30s) plus the entrypoint's migrations are why this allows
# three minutes rather than one.
- name: Wait for Yamtrack to pass its own health check
ansible.builtin.command: "podman healthcheck run {{ container_name }}"
register: yamtrack_health
until: yamtrack_health.rc == 0
retries: 36
delay: 5
changed_when: false
failed_when: yamtrack_health.rc != 0
@@ -0,0 +1,56 @@
# Yamtrack — rootless Podman Quadlet unit.
# Managed by Ansible (yamtrack/roles/yamtrack). Do not edit by hand.
#
# This unit joins {{ network_name }}.network, which is created by
# caddy/roles/register (the same include that attaches Caddy to it). The shared
# valkey container is joined to that network by valkey/roles/attach, which is
# what makes the host in REDIS_URL resolve.
[Unit]
Description=Yamtrack
Documentation=https://github.com/FuzzyGrim/Yamtrack
After=network-online.target
Wants=network-online.target
[Container]
ContainerName={{ container_name }}
Image={{ image }}
# keep-id maps *container root* onto {{ podman_user }} on the host, so
# {{ data_dir }} stays owned by {{ podman_user }} and never by a subuid, and a
# plain tar backup keeps working without `podman unshare` or `sudo`.
#
# uid 0 is not incidental. `--userns=keep-id` also decides which user the
# container runs as — the *mapped* one — and the image's entrypoint needs real
# root for its first steps: `groupmod` and `usermod` rewrite /etc/group and
# /etc/passwd, which are owned by container-root in the image. With
# `keep-id:uid=1000` the entrypoint runs as `abc`, cannot lock /etc/group, and
# exits 10 before supervisord is ever reached. The entrypoint's PUID/PGID are
# pinned to the same 0 in the environment file, so `abc` resolves to container
# root and everything the app writes lands on {{ podman_user }}.
#
# See readme.md, "Who runs as whom".
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
# SECRET, TZ, URLS and REDIS_URL live here rather than as Environment= lines
# above: this unit is world-readable in ~/.config, and SECRET signs sessions and
# password-reset tokens. Read by systemd, not by Ansible, so the file has to
# exist before this unit starts.
EnvironmentFile={{ env_file }}
Network={{ network_name }}.network
# The image's state directory: db.sqlite3 and its -wal/-shm sidecars. The rest
# of /yamtrack is image content (staticfiles/ is built in, the code is
# read-only), so this is the whole of what has to survive.
Volume={{ data_dir }}:/yamtrack/db
[Service]
Restart=always
# The entrypoint runs `manage.py migrate --noinput` before supervisord starts
# nginx, gunicorn, celery and celery beat, so allow for a slow first start on a
# large database.
TimeoutStartSec=300
[Install]
WantedBy=default.target
@@ -0,0 +1,37 @@
# Environment for {{ container_name }}.container.
# Managed by Ansible (yamtrack/roles/yamtrack). Do not edit by hand.
#
# Mode 0600 on purpose: the Quadlet unit that reads this file is world-readable,
# and SECRET signs sessions and password-reset tokens. Note that `podman inspect`
# still prints the resulting environment to anyone who can run podman as this
# user.
#
# The first four are the variables the Docker-era container was given, so this
# deployment behaves identically. ALLOWED_HOSTS is deliberately absent: its
# default is `*`, which is what the old container relied on. PUID/PGID are new —
# the Docker-era container ran as real root, and under rootless keep-id the
# entrypoint needs to be told which uid is the mapped one. See readme.md, "Who
# runs as whom".
# The Django SECRET_KEY. Carried over verbatim so existing sessions stay valid.
SECRET={{ yamtrack_secret }}
# TIME_ZONE for the app and for celery beat's schedules.
TZ={{ yamtrack_tz }}
# CSRF trusted origins. The old container set `URLS=https://{{ host }}`.
URLS={{ urls }}
# The shared Valkey unit, attached to this app's network by
# yamtrack/roles/valkey. Yamtrack requires Redis: it is the cache backend and
# the Celery broker, with no fallback.
REDIS_URL={{ redis_url }}
# The entrypoint runs `groupmod`/`usermod` to make the image's `abc` account
# match these, then chowns the data directory to it. Its own defaults are 1000,
# which under this unit's keep-id:uid=0 is a *subuid* on the host — the database
# would come out owned by e.g. 165536, invisible to a plain tar backup. Pinning
# both to 0 makes `abc` resolve to container root, which keep-id maps onto
# podman_user.
PUID={{ app_uid }}
PGID={{ app_gid }}
+113
View File
@@ -0,0 +1,113 @@
---
# ---------------------------------------------------------------------------
# Rootless Podman / Quadlet
# ---------------------------------------------------------------------------
# Duplicated on purpose from caddy/vars/default.yml: a play cannot include that
# file wholesale, because it also sets `host_directory` — to the Caddy data
# directory — and every playbook defines that for itself. Keep these three in
# sync with caddy/vars/default.yml. (Its fourth, podman_unprivileged_port_start,
# belongs to the podman tasks, which this play does not run: Yamtrack listens on
# 8000 inside the container and publishes nothing host-side.)
#
# podman_user is derived from the connection: this play carries no become, so
# the Quadlet units belong to whoever Ansible connects as. Requires gather_facts
# (set in playbook.yml).
podman_user: "{{ ansible_facts['user_id'] }}"
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
# ---------------------------------------------------------------------------
# Yamtrack
# ---------------------------------------------------------------------------
host: track.tanshu.com
container_name: yamtrack
unit_name: "{{ container_name }}.service"
# Fully qualified on purpose: rootless Podman resolves short names against
# unqualified-search-registries, and there is no TTY to answer a prompt when
# the systemd unit starts. ghcr.io was already qualified; the tag is explicit
# for the same reason. Nothing is pinned — see readme.md, "Notes and gotchas".
image: ghcr.io/fuzzygrim/yamtrack:latest
# The Quadlet file is <network_name>.network. caddy/roles/register pins the
# Podman network name to <network_name>_net (so: yamtrack.network, yamtrack_net,
# which is the name the Docker-era deployment used).
network_name: "{{ container_name }}"
host_directory: "{{ podman_data_dir }}/{{ container_name }}"
# The image's bind mount: /yamtrack/db holds db.sqlite3 and its WAL sidecars.
# Nothing else on /yamtrack is state — staticfiles/ is baked into the image and
# the code is read-only — so this is the only directory worth keeping.
data_dir: "{{ host_directory }}/data"
# The port inside the container that Caddy proxies to: the image's own nginx
# listens there and forwards to gunicorn. Reproduces the Docker-era Caddyfile
# entry (`reverse_proxy yamtrack:8000`) exactly.
container_port: 8000
# SECRET, TZ and REDIS_URL live here rather than as Environment= lines in the
# unit file: that file is world-readable in ~/.config. Read by systemd through
# EnvironmentFile=, not by Ansible, so it has to be written before the unit
# starts.
env_file: "{{ host_directory }}/{{ container_name }}.env"
# The uid this unit maps onto podman_user, and the PUID/PGID the image's
# entrypoint is given. It is 0 — container root — and that is deliberate:
#
# 1. `--userns=keep-id` does two things: it maps one container uid onto the
# connecting user, and it runs the container *as that uid*. The entrypoint's
# first act is `groupmod`/`usermod`, which rewrite /etc/group and
# /etc/passwd — files owned by container-root in the image. With
# `keep-id:uid=1000` those run as `abc` and die on "cannot lock /etc/group"
# (the unit exits 10) before supervisord is ever reached. Root is required.
# 2. Mapping container root onto podman_user is what keeps
# ~/podman-data/yamtrack owned by podman_user rather than by a subuid. It is
# the same choice caddy, bitwarden and tandoor make.
#
# It is also rendered as PUID/PGID in the environment file: the entrypoint's own
# default is 1000, which under this mapping is a subuid on the host, so its
# `chown -R abc:abc db` would strand the database on an unmapped uid. See
# readme.md, "Who runs as whom".
app_uid: 0
app_gid: 0
yamtrack_tz: "Asia/Kolkata"
# The Django SECRET_KEY, as the image calls it (`SECRET`). Carried over verbatim
# from the Docker-era playbook so existing sessions and password-reset tokens
# stay valid — rotating it logs every user out. Committed in plaintext as
# before; flagged as a follow-up, not changed here.
yamtrack_secret: "c14c5970eb728fbf8b2325fd3004069d"
# What the old container passed as URLS, which Yamtrack turns into its CSRF
# trusted origins. ALLOWED_HOSTS is deliberately not set: its default is `*`,
# which is what the Docker-era container relied on.
urls: "https://{{ host }}"
# ---------------------------------------------------------------------------
# Valkey (the shared cache / Celery broker)
# ---------------------------------------------------------------------------
# Yamtrack hard-requires Redis: its cache backend is django_redis with no
# fallback, the Celery worker and beat need a broker, and several metadata
# providers talk to Redis directly. It is a *separate* playbook, deployed before
# this one: yamtrack/roles/valkey only attaches its network to that shared unit.
#
# The Docker-era container was given `redis://valkey:6379`, and valkey is what
# the container is called, so the URL is unchanged.
valkey_container: valkey
valkey_unit: "{{ valkey_container }}.service"
redis_url: "redis://{{ valkey_container }}:6379"
# ---------------------------------------------------------------------------
# Caddy
# ---------------------------------------------------------------------------
# The shared Caddy unit and its Caddyfile, both owned by caddy/playbook.yml.
caddy_name: caddy
caddyfile_path: "{{ podman_data_dir }}/{{ caddy_name }}/conf/Caddyfile"
snippet_block: |
{{ host }} {
reverse_proxy {{ container_name }}:{{ container_port }}
}