Add the Podman migration planning docs

Three documents written before the container migration, all referenced from the service readmes, so they belong in the tree: - podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's architecture, privileges and namespaces; the four things that actually break (anonymous volumes, per-container uids, tooling that assumes Docker, image naming); and the per-container uid matrix that the UserNS=keep-id lines in the Quadlet units come from. - podman-migration.md -- the plan: target architecture, the ownership/chown pass, the Quadlet unit shape, and the low-port and lingering prerequisites. - backup-plan.md -- per-service irreplaceable state, the consistency problems (live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1 architecture. caddy, postgres, tandoor and bitwarden each link into these from their readme, so leaving them untracked left those links dead.
2026-10-16 08:50:53 +05:30
1 parent f431d94f06
commit 8136c2ddaf
@@ -0,0 +1,318 @@
+# Podman compatibility audit — monoco container fleet
+
+Status: **planning only.** Read-only inspection; nothing on monoco was modified.
+Scope: all 26 running containers, audited for rootless Podman + Quadlet.
+Companion docs: [podman-migration.md](podman-migration.md), [backup-plan.md](backup-plan.md)
+
+---
+
+## 1. Verdict
+
+**Nothing in this fleet blocks a rootless Podman migration.** There is no privileged
+container, no capability add, no device, no host networking, no custom seccomp, no
+Docker-socket mount, and no Compose file. Every image is **arm64**, so there is no
+emulation.
+
+The breakages are all in the *surrounding tooling* (backup scripts, cron, Ansible
+modules) and in *ownership/volume handling* — not in the images. Your instinct that
+your own `registry.tanshu.com` images will be fine is correct; I checked them
+specifically and they are among the cleanest in the fleet.
+
+Four things need real attention, and one of them is not obvious:
+
+| # | Issue | Severity |
+|---|---|---|
+| 1 | `hiwikijs` and `valkey` use **anonymous Docker volumes** — copying `/var/lib/*` misses them entirely | **data loss** |
+| 2 | Every data tree needs re-owning, and **each app's uid differs** (0 / 33 / 70 / 999 / 1000 / 1003) | blocks startup |
+| 3 | `--link` in backup scripts, and root's crontab doing `docker exec` | breaks backups + cron |
+| 4 | Locally-built `sms-wa-bridge` has no registry — short-name resolution fails | won't start |
+
+---
+
+## 2. What I verified as clean
+
+Worth stating explicitly, because it is why this migration is tractable:
+
+- **Architecture**: every image is `arm64` / `linux`. No binfmt/qemu needed.
+- **Privileges**: `Privileged=false`, `CapAdd=[]`, `CapDrop=[]`, `Devices=[]` on all 26.
+- **Namespaces**: no `--network host`, no `--pid host`, no `--ipc host`, no `--userns host`.
+- **Security**: no custom `SecurityOpt`, no custom seccomp profile, no `--extra-hosts`.
+- **Docker socket**: not mounted anywhere. No image talks to the Docker API.
+- **Compose**: none. Every container is a plain `docker run` / Ansible `docker_container`.
+- **shm**: all at the 64 MB default; nothing needs a larger `/dev/shm`.
+- **Sysctls**: none set.
+- **Resource limits**: `Memory=0`, `NanoCpus=0` on every container. (The Ansible role
+  for gitea *declares* `cpus: .10` / `memory: 512M`, but they are **not in effect** on
+  the running container — so no cgroup-delegation concerns for rootless.)
+- **Custom DNS**: `Dns=[]` everywhere; aardvark-dns will replace Docker's `127.0.0.11`.
+- **Hostnames**: all default (container ID). Nothing depends on a fixed hostname —
+  `sms-wa-bridge` explicitly cleans its own `Singleton*` locks on start because of this.
+
+---
+
+## 3. The four real problems
+
+### 3.1 Anonymous volumes — the one that loses data
+
+Eight images declare `VOLUME`:
+
+| Image | VOLUME | Mounted? |
+|---|---|---|
+| `vaultwarden/server` | `/data` | ✅ bind-mounted |
+| `gitea/gitea` | `/data` | ✅ bind-mounted |
+| `registry.tanshu.com/{greatbear,hops}` | `/app/pb_data` | ✅ bind-mounted |
+| `nextcloud` | `/var/www/html` | ✅ bind-mounted |
+| `postgres:17-alpine` | `/var/lib/postgresql/data` | ✅ bind-mounted |
+| **`ghcr.io/requarks/wiki`** | `/wiki/data/content` | ❌ **anonymous volume** |
+| **`valkey/valkey:alpine`** | `/data` | ❌ **anonymous volume** |
+
+The two anonymous ones live in `/var/lib/docker/volumes/<hash>/_data`, **not** under
+`/var/lib/<service>`. A copy of the data directories will silently omit them.
+
+`hiwikijs` is the serious one: `/var/lib/hiwikijs` exists but is an **empty directory**
+from Dec 2022, and `~/bin/hiwikijs-backup.sh` archives exactly that empty directory and
+exits successfully. So Wiki.js currently has **no backup at all** — this predates the
+migration and should be fixed regardless of it.
+
+Fix: export both to real directories (e.g. `~/podman-data/hiwikijs/content`,
+`~/podman-data/valkey`) and declare them as Quadlet `Volume=` bind mounts. `valkey` is a
+rebuildable cache; `hiwikijs` is not.
+
+### 3.2 Ownership — one uid per container, and they all differ
+
+Under rootless Podman the user namespace maps only **one** uid:gid to your host user.
+The rule is `UserNS=keep-id:uid=<app-uid>,gid=<app-gid>`, where `<app-uid>` is the uid
+the application actually writes as — **not** the image's `USER` and **not** `docker exec id`.
+
+Measured (see §4 for the full matrix):
+
+| App uid | Containers |
+|---|---|
+| 0 | caddy, brewman ×5, greatbear, hops, affine, audiobookshelf, bitwarden, tandoor, sms-wa-bridge, vince |
+| 33 | nextcloud |
+| 70 | postgres |
+| 999 | valkey |
+| 1000 | mozimo, mozimo-staging, hiwikijs, menu-app ×3, yamtrack, trillium |
+| 1003 | gitea |
+
+Trap: `docker exec <c> id -u` reports the **image's** configured user, not the running
+process. It says `0` for `postgres` (which actually writes as **70**), for `nextcloud`
+(**33**), and for `gitea` (**1003**) — because those entrypoints drop privileges after
+starting. Derive the uid from **the ownership of what the app writes**, which is what
+the table above does.
+
+Also note `tanshu` is **uid 1001**, not 1000 — most of the uid-1000 data is currently
+owned by the `ubuntu` user. "Chown it to me" is therefore not a no-op anywhere.
+
+The reassuring part: for the entrypoints that start as root and drop to an app user
+(`postgres`, `nextcloud`, `gitea`, `trillium`, `yamtrack`), the root phase maps to a
+subuid and its internal `chown` still lands on your host user. So a single
+`keep-id:uid=<app-uid>` plus `chown -R tanshu:tanshu` should work — **but verify per
+service**, because a file written during the root phase and never chowned will be
+subuid-owned and unreadable by the dropped-privilege process.
+
+### 3.3 Tooling that assumes Docker
+
+- **`--link postgres:db`** in `db-backup.sh`, `bitwarden-`, `hiwikijs-`, `kakawa-`,
+  `nextcloud-`, `tandoor-backup.sh` and their restores. No Podman equivalent.
+- **Root's crontab**: `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
+  Root **cannot** `podman exec` into another user's rootless containers — this must
+  become a systemd **user** timer.
+- **`~/bin/empty-trash.sh`** execs a container named `cloud`, which no longer exists
+  (it is `nextcloud`).
+- **Ansible**: `docker_container` / `docker_image` / `docker_network` →
+  `containers.podman`. Note `podman_network` has **no** `connected:`/`appends:`
+  parameters, which every `*/roles/network/tasks/main.yaml` in this repo relies on.
+  Declare the full `network:` list on each `podman_container` instead.
+- **Everything outside this repo.** You said ~90% is Ansible and the rest is
+  `docker run`. The deploy pipelines for `mozimo`, `brewman`, `menu-admin`, `hops`,
+  `greatbear` and `sms-wa-bridge` live elsewhere — if they still call `docker` against
+  the host, they will recreate Docker containers alongside your Podman ones.
+
+### 3.4 Image naming
+
+Rootless Podman resolves short names against `unqualified-search-registries`. In a
+**Quadlet/systemd context there is no TTY to confirm a short-name prompt**, so
+ambiguous names fail the unit rather than prompting.
+
+- **`sms-wa-bridge`** was built locally with `docker build -t sms-wa-bridge` — no
+  registry. Podman will try `docker.io/library/sms-wa-bridge` and fail. Transfer it with
+  `docker save sms-wa-bridge | podman load` and reference **`localhost/sms-wa-bridge:latest`**.
+- `nextcloud`, `caddy`, `postgres`, `valkey/valkey`, `gitea/gitea`, `vaultwarden/server`,
+  `vabene1111/recipes`, `triliumnext/notes` are all short or implicit-`docker.io` names.
+  **Fully qualify all of them** (`docker.io/library/nextcloud:latest` etc.) so resolution
+  never depends on host config.
+- `registry.tanshu.com/*` and `ghcr.io/*` are already qualified. ✅
+
+---
+
+## 4. Per-container matrix
+
+`App uid` is what the application writes as. `Anon vol` = uses an anonymous Docker volume.
+
+| Container | Image | App uid | Anon vol | Verdict | Notes |
+|---|---|---|---|---|---|
+| `caddy` | `caddy:latest` | 0 | – | ⚠️ | Needs `ip_unprivileged_port_start≤80`; publishes 80, 443, **443/udp (HTTP/3)** — verify UDP forwarding |
+| `postgres` | `postgres:17-alpine` | **70** | – | ⚠️ | PGDATA is `pgdata/` under the mount, owned uid 70. Drop the `0.0.0.0:5432` publish |
+| `valkey` | `valkey/valkey:alpine` | **999** | **YES** | ⚠️ | Export the anonymous volume |
+| `nextcloud` | `nextcloud:latest` | **33** | – | ✅ | `VOLUME /var/www/html` is bind-mounted; needs `POSTGRES_HOST=db` / `REDIS_HOST=redis` aliases |
+| `gitea` | `gitea/gitea:latest` | **1003** | – | ⚠️ | `/var/lib/gitea/ssh` holds **SSH host keys** — copy them or clients see host-key warnings |
+| `mozimo` | `registry.tanshu.com/mozimo:4a3863a` | 1000 | – | ✅ | Image `USER=node`; entrypoint starts PocketBase + SSR together |
+| `mozimo-staging` | `registry.tanshu.com/mozimo:3929d3f` | 1000 | – | ✅ | |
+| `greatbear` | `registry.tanshu.com/tanshu/greatbear` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
+| `hops` | `registry.tanshu.com/hops:latest` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
+| `menu-app.greatbear.in` | `registry.tanshu.com/menu-admin` | 1000 | – | ⚠️ | **Container names contain dots** — name the Quadlet units `menu-greatbear.container` etc. and set `ContainerName=` explicitly so the Caddyfile keeps resolving |
+| `menu-app.hngmohali.com` | ″ | 1000 | – | ⚠️ | ″ |
+| `menu-app.hopsngrains.com` | ″ | 1000 | – | ⚠️ | ″ |
+| `hiwikijs` | `ghcr.io/requarks/wiki:latest` | 1000 | **YES** | ❌ | Content is in an anonymous volume; `/var/lib/hiwikijs` is an empty decoy |
+| `affine` | `ghcr.io/toeverything/affine-graphql:stable` | 0 | – | ✅ | Needs postgres + valkey; indexer disabled (`AFFINE_INDEXER_ENABLED=false`), so no pgvector dependency |
+| `audiobookshelf` | `ghcr.io/advplyr/audiobookshelf` | 0 | – | ✅ | `tini` entrypoint; 15 GB is media, state is 22 MB |
+| `bitwarden` | `vaultwarden/server:latest` | 0 | – | ⚠️ | Authoritative store is **Postgres**; the `db.sqlite3` on disk is stale — do not restore it |
+| `yamtrack` | `ghcr.io/fuzzygrim/yamtrack` | 1000 | – | ✅ | SQLite + WAL; needs a consistent copy |
+| `trillium` | `triliumnext/notes:latest` | 1000 | – | ✅ | `document.db` + WAL; has its own `backup/` dir |
+| `tandoor` | `vabene1111/recipes:latest` | 0 | – | ✅ | Needs `POSTGRES_HOST=db` alias |
+| `sms-wa-bridge` | `sms-wa-bridge` (local) | 0 | – | ⚠️ | Chromium **is fine** (see below). Must be `localhost/sms-wa-bridge:latest` |
+| `vince` | `ghcr.io/vinceanalytics/vince` | 0 | – | ⚠️ | **Distroless** — no shell, no `id`, so no exec-based checks. Container was run with `serve`, image `CMD` is null → set `Exec=serve` |
+| `brewman-acc/exp/hinchco/hops/mhl` | `registry.tanshu.com/brewman:latest` | 0 | – | ✅ | 5 containers, identical config, one DB each |
+
+### The Chromium question, answered
+
+`sms-wa-bridge` is the classic rootless-Podman landmine: Chromium normally needs
+`CLONE_NEWUSER` for its sandbox, which is blocked inside a container, and it can also
+die on a 64 MB `/dev/shm`. I checked `server.js` and it already handles both:
+
+```js
+puppeteer: {
+  headless: true,
+  executablePath: process.env.CHROMIUM_PATH || undefined,
+  args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"]
+}
+```
+
+`--no-sandbox` removes the userns dependency and `--disable-dev-shm-usage` removes the
+shm dependency. **No action needed** — this container will behave identically.
+
+---
+
+## 5. Your migration thesis — validated, with three corrections
+
+> *"if we copy the data directories, change ownership and deploy on new server, we should be good to go"*
+
+**Broadly yes** — and with downtime free, this is a much safer plan than an in-place
+cutover, because you can stop everything cleanly and copy quiesced data. Three
+corrections:
+
+**Correction 1 — "the data directories" is not all the state.** Adding to `/var/lib/*`:
+- the two **anonymous volumes** (§3.1) — `hiwikijs` is genuinely at risk;
+- **`/var/lib/gitea/ssh`** (SSH host keys) — easy to miss, and clients will complain;
+- **`/var/lib/caddy/data`** — the ACME account key and ~20 certificates. Lose it and you
+  re-issue everything against Let's Encrypt rate limits;
+- **the environment variables** — that is where the credentials live (Shopify client
+  secret, Sentry token, JWT private keys, PocketBase passwords, DB passwords);
+- **`/etc/crontab` entries and `~/bin/`** — the operational glue.
+
+**Correction 2 — "change ownership" is per-container, not global.** Chowning everything
+to `tanshu` is right, but it only works if each container is also given the matching
+`UserNS=keep-id:uid=<app-uid>`. Get the pairing wrong and the app starts and then fails
+to write — the failure mode is a permissions error at runtime, not a clean startup
+failure. The §4 table gives the pairing.
+
+**Correction 3 — Postgres wants a clean stop, and a dump is still worth it.** Copying
+`PGDATA` after a clean shutdown is valid *at the same major version* (17.5 here). But a
+logical dump is cheap and gives you a version-independent fallback. Do both.
+
+Since downtime is not a constraint, the copying step should be:
+
+```
+1. Stop all containers cleanly.
+2. Copy: /var/lib/<svc>/, /var/lib/gitea/ssh, the two anonymous volumes,
+   /var/lib/caddy/data, ~/bin/, env vars, docker inspect JSON.
+3. Verify checksums on the copy.
+4. chown -R tanshu:tanshu on the destination, per §3.2.
+5. Load images (docker save | podman load, or podman pull + podman login).
+6. Write Quadlet units, set UserNS / Network / Volume.
+7. systemctl --user daemon-reload && start the data tier, then apps, then Caddy.
+```
+
+Because you stop cleanly, the SQLite WAL problem from the backup plan largely
+disappears — no live WAL, no torn copies. That is the single biggest simplification
+this approach buys you.
+
+---
+
+## 6. Low ports and lingering
+
+You already know both, confirming for completeness:
+
+**Ports.** Rootless cannot bind < 1024. Caddy binds 80/443 **inside** its own network
+namespace (where container-root has `CAP_NET_BIND_SERVICE`, which Podman grants by
+default), but the **host-side publish** needs the sysctl:
+
+```
+sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
+echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
+```
+
+Verify after reboot — this is a favourite thing to forget on a fresh instance.
+
+**Lingering.** Without it, Quadlet **user** services stop when the login session ends,
+which on a server is immediately, and they do not start at boot:
+
+```
+sudo loginctl enable-linger tanshu
+```
+
+Also confirm `/etc/subuid` and `/etc/subgid` have a range for the new user — monoco has
+`tanshu:165536:65536`, and a fresh instance will assign a different base. Do not
+hardcode `165536` anywhere.
+
+---
+
+## 7. Cutover notes specific to a new instance
+
+**DNS and certificates are the thing to plan around.** All ~20 hostnames resolve to
+`13.202.6.2`. On a new instance with a new IP:
+
+- Caddy cannot obtain certificates until DNS points at the new host (HTTP-01), and the
+  existing certs in `/var/lib/caddy/data` are only valid to *serve* for those names once
+  traffic arrives.
+- **Check whether `13.202.6.2` is an Elastic IP.** If it is, reassign it to the new
+  instance and DNS, certificates, and clients are all untouched — by far the cleanest
+  cutover. Worth confirming before designing around a DNS change.
+
+**Logs do not migrate.** All 26 containers use the `json-file` driver; Podman uses
+journald. If the existing logs matter, export them before decommissioning. Otherwise
+`podman logs` / `journalctl --user -u <svc>` replaces `docker logs`.
+
+**Image transfer.** `docker save <img> | podman load` avoids re-downloading (and avoids
+registry auth for the local image). For `registry.tanshu.com`, run `podman login` on the
+new host so `podman auto-update`/pulls work.
+
+**The OS question is still open.** Per [podman-migration.md §3.4](podman-migration.md),
+a **fresh 26.04 instance with stock Podman 5.7 needs no source builds at all**. Since you
+are standing up a new instance anyway, this is now essentially free — and it makes the
+"build Podman 6 for pesto" question entirely separable from the migration.
+
+---
+
+## 8. Per-service verification after cutover
+
+For each container, in this order:
+
+1. **Starts and stays up** — `systemctl --user status <svc>`
+2. **Writes as the right uid** — create/update something through the app, then confirm
+   on the host that the new file is owned by `tanshu`. This is the check that catches a
+   wrong `keep-id`.
+3. **Resolves its dependencies** — `podman exec <c> getent hosts postgres` (and `db` /
+   `redis` aliases where used)
+4. **Healthcheck reports healthy** — `podman inspect --format '{{.State.Health.Status}}'`
+5. Special cases:
+   - `caddy`: real client IPs in the access log (this is where the rootless `rootlessport`
+     limitation shows up — see §3.4 of the migration plan); `https://<vhost>` responds;
+     HTTP/3 (`curl --http3`) if you care
+   - `sms-wa-bridge`: WhatsApp re-links (or restores) and a test message sends
+   - `gitea`: `git clone` over **both** HTTPS and SSH (port 2222), no host-key warning
+   - `postgres`: every client app connects; nothing listening on host `5432`
+   - `nextcloud`: upload a file (exercises the uid-33 mapping and `REDIS_HOST`)
+   - `vince`: reachable on `stats.mozimo.in` (distroless, so no shell to diagnose with)
+6. **Reboot the host** and confirm everything returns unattended