playbooks/docs/podman-compat-audit.md

# Podman compatibility audit — monoco container fleet

Status: **planning only.** Read-only inspection; nothing on monoco was modified.
Scope: all 26 running containers, audited for rootless Podman + Quadlet.
Companion docs: [podman-migration.md](podman-migration.md), [backup-plan.md](backup-plan.md)

---

## 1. Verdict

**Nothing in this fleet blocks a rootless Podman migration.** There is no privileged
container, no capability add, no device, no host networking, no custom seccomp, no
Docker-socket mount, and no Compose file. Every image is **arm64**, so there is no
emulation.

The breakages are all in the *surrounding tooling* (backup scripts, cron, Ansible
modules) and in *ownership/volume handling* — not in the images. Your instinct that
your own `registry.tanshu.com` images will be fine is correct; I checked them
specifically and they are among the cleanest in the fleet.

Four things need real attention, and one of them is not obvious:

| # | Issue | Severity |
|---|---|---|
| 1 | `hiwikijs` and `valkey` use **anonymous Docker volumes** — copying `/var/lib/*` misses them entirely | **data loss** |
| 2 | Every data tree needs re-owning, and **each app's uid differs** (0 / 33 / 70 / 999 / 1000 / 1003) | blocks startup |
| 3 | `--link` in backup scripts, and root's crontab doing `docker exec` | breaks backups + cron |
| 4 | Locally-built `sms-wa-bridge` has no registry — short-name resolution fails | won't start |

---

## 2. What I verified as clean

Worth stating explicitly, because it is why this migration is tractable:

- **Architecture**: every image is `arm64` / `linux`. No binfmt/qemu needed.
- **Privileges**: `Privileged=false`, `CapAdd=[]`, `CapDrop=[]`, `Devices=[]` on all 26.
- **Namespaces**: no `--network host`, no `--pid host`, no `--ipc host`, no `--userns host`.
- **Security**: no custom `SecurityOpt`, no custom seccomp profile, no `--extra-hosts`.
- **Docker socket**: not mounted anywhere. No image talks to the Docker API.
- **Compose**: none. Every container is a plain `docker run` / Ansible `docker_container`.
- **shm**: all at the 64 MB default; nothing needs a larger `/dev/shm`.
- **Sysctls**: none set.
- **Resource limits**: `Memory=0`, `NanoCpus=0` on every container. (The Ansible role
  for gitea *declares* `cpus: .10` / `memory: 512M`, but they are **not in effect** on
  the running container — so no cgroup-delegation concerns for rootless.)
- **Custom DNS**: `Dns=[]` everywhere; aardvark-dns will replace Docker's `127.0.0.11`.
- **Hostnames**: all default (container ID). Nothing depends on a fixed hostname —
  `sms-wa-bridge` explicitly cleans its own `Singleton*` locks on start because of this.

---

## 3. The four real problems

### 3.1 Anonymous volumes — the one that loses data

Eight images declare `VOLUME`:

| Image | VOLUME | Mounted? |
|---|---|---|
| `vaultwarden/server` | `/data` | ✅ bind-mounted |
| `gitea/gitea` | `/data` | ✅ bind-mounted |
| `registry.tanshu.com/{greatbear,hops}` | `/app/pb_data` | ✅ bind-mounted |
| `nextcloud` | `/var/www/html` | ✅ bind-mounted |
| `postgres:17-alpine` | `/var/lib/postgresql/data` | ✅ bind-mounted |
| **`ghcr.io/requarks/wiki`** | `/wiki/data/content` | ❌ **anonymous volume** |
| **`valkey/valkey:alpine`** | `/data` | ❌ **anonymous volume** |

The two anonymous ones live in `/var/lib/docker/volumes/<hash>/_data`, **not** under
`/var/lib/<service>`. A copy of the data directories will silently omit them.

`hiwikijs` is the serious one: `/var/lib/hiwikijs` exists but is an **empty directory**
from Dec 2022, and `~/bin/hiwikijs-backup.sh` archives exactly that empty directory and
exits successfully. So Wiki.js currently has **no backup at all** — this predates the
migration and should be fixed regardless of it.

Fix: export both to real directories (e.g. `~/podman-data/hiwikijs/content`,
`~/podman-data/valkey`) and declare them as Quadlet `Volume=` bind mounts. `valkey` is a
rebuildable cache; `hiwikijs` is not.

### 3.2 Ownership — one uid per container, and they all differ

Under rootless Podman the user namespace maps only **one** uid:gid to your host user.
The rule is `UserNS=keep-id:uid=<app-uid>,gid=<app-gid>`, where `<app-uid>` is the uid
the application actually writes as — **not** the image's `USER` and **not** `docker exec id`.

Measured (see §4 for the full matrix):

| App uid | Containers |
|---|---|
| 0 | caddy, brewman ×5, greatbear, hops, affine, audiobookshelf, bitwarden, tandoor, sms-wa-bridge, vince |
| 33 | nextcloud |
| 70 | postgres |
| 999 | valkey |
| 1000 | mozimo, mozimo-staging, hiwikijs, menu-app ×3, yamtrack, trillium |
| 1003 | gitea |

Trap: `docker exec <c> id -u` reports the **image's** configured user, not the running
process. It says `0` for `postgres` (which actually writes as **70**), for `nextcloud`
(**33**), and for `gitea` (**1003**) — because those entrypoints drop privileges after
starting. Derive the uid from **the ownership of what the app writes**, which is what
the table above does.

Also note `tanshu` is **uid 1001**, not 1000 — most of the uid-1000 data is currently
owned by the `ubuntu` user. "Chown it to me" is therefore not a no-op anywhere.

The reassuring part: for the entrypoints that start as root and drop to an app user
(`postgres`, `nextcloud`, `gitea`, `trillium`, `yamtrack`), the root phase maps to a
subuid and its internal `chown` still lands on your host user. So a single
`keep-id:uid=<app-uid>` plus `chown -R tanshu:tanshu` should work — **but verify per
service**, because a file written during the root phase and never chowned will be
subuid-owned and unreadable by the dropped-privilege process.

### 3.3 Tooling that assumes Docker

- **`--link postgres:db`** in `db-backup.sh`, `bitwarden-`, `hiwikijs-`, `kakawa-`,
  `nextcloud-`, `tandoor-backup.sh` and their restores. No Podman equivalent.
- **Root's crontab**: `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
  Root **cannot** `podman exec` into another user's rootless containers — this must
  become a systemd **user** timer.
- **`~/bin/empty-trash.sh`** execs a container named `cloud`, which no longer exists
  (it is `nextcloud`).
- **Ansible**: `docker_container` / `docker_image` / `docker_network` →
  `containers.podman`. Note `podman_network` has **no** `connected:`/`appends:`
  parameters, which every `*/roles/network/tasks/main.yaml` in this repo relies on.
  Declare the full `network:` list on each `podman_container` instead.
- **Everything outside this repo.** You said ~90% is Ansible and the rest is
  `docker run`. The deploy pipelines for `mozimo`, `brewman`, `menu-admin`, `hops`,
  `greatbear` and `sms-wa-bridge` live elsewhere — if they still call `docker` against
  the host, they will recreate Docker containers alongside your Podman ones.

### 3.4 Image naming

Rootless Podman resolves short names against `unqualified-search-registries`. In a
**Quadlet/systemd context there is no TTY to confirm a short-name prompt**, so
ambiguous names fail the unit rather than prompting.

- **`sms-wa-bridge`** was built locally with `docker build -t sms-wa-bridge` — no
  registry. Podman will try `docker.io/library/sms-wa-bridge` and fail. Transfer it with
  `docker save sms-wa-bridge | podman load` and reference **`localhost/sms-wa-bridge:latest`**.
- `nextcloud`, `caddy`, `postgres`, `valkey/valkey`, `gitea/gitea`, `vaultwarden/server`,
  `vabene1111/recipes`, `triliumnext/notes` are all short or implicit-`docker.io` names.
  **Fully qualify all of them** (`docker.io/library/nextcloud:latest` etc.) so resolution
  never depends on host config.
- `registry.tanshu.com/*` and `ghcr.io/*` are already qualified. ✅

---

## 4. Per-container matrix

`App uid` is what the application writes as. `Anon vol` = uses an anonymous Docker volume.

| Container | Image | App uid | Anon vol | Verdict | Notes |
|---|---|---|---|---|---|
| `caddy` | `caddy:latest` | 0 | – | ⚠️ | Needs `ip_unprivileged_port_start≤80`; publishes 80, 443, **443/udp (HTTP/3)** — verify UDP forwarding |
| `postgres` | `postgres:17-alpine` | **70** | – | ⚠️ | PGDATA is `pgdata/` under the mount, owned uid 70. Drop the `0.0.0.0:5432` publish |
| `valkey` | `valkey/valkey:alpine` | **999** | **YES** | ⚠️ | Export the anonymous volume |
| `nextcloud` | `nextcloud:latest` | **33** | – | ✅ | `VOLUME /var/www/html` is bind-mounted; needs `POSTGRES_HOST=db` / `REDIS_HOST=redis` aliases |
| `gitea` | `gitea/gitea:latest` | **1003** | – | ⚠️ | `/var/lib/gitea/ssh` holds **SSH host keys** — copy them or clients see host-key warnings |
| `mozimo` | `registry.tanshu.com/mozimo:4a3863a` | 1000 | – | ✅ | Image `USER=node`; entrypoint starts PocketBase + SSR together |
| `mozimo-staging` | `registry.tanshu.com/mozimo:3929d3f` | 1000 | – | ✅ | |
| `greatbear` | `registry.tanshu.com/tanshu/greatbear` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
| `hops` | `registry.tanshu.com/hops:latest` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
| `menu-app.greatbear.in` | `registry.tanshu.com/menu-admin` | 1000 | – | ⚠️ | **Container names contain dots** — name the Quadlet units `menu-greatbear.container` etc. and set `ContainerName=` explicitly so the Caddyfile keeps resolving |
| `menu-app.hngmohali.com` | ″ | 1000 | – | ⚠️ | ″ |
| `menu-app.hopsngrains.com` | ″ | 1000 | – | ⚠️ | ″ |
| `hiwikijs` | `ghcr.io/requarks/wiki:latest` | 1000 | **YES** | ❌ | Content is in an anonymous volume; `/var/lib/hiwikijs` is an empty decoy |
| `affine` | `ghcr.io/toeverything/affine-graphql:stable` | 0 | – | ✅ | Needs postgres + valkey; indexer disabled (`AFFINE_INDEXER_ENABLED=false`), so no pgvector dependency |
| `audiobookshelf` | `ghcr.io/advplyr/audiobookshelf` | 0 | – | ✅ | `tini` entrypoint; 15 GB is media, state is 22 MB |
| `bitwarden` | `vaultwarden/server:latest` | 0 | – | ⚠️ | Authoritative store is **Postgres**; the `db.sqlite3` on disk is stale — do not restore it |
| `yamtrack` | `ghcr.io/fuzzygrim/yamtrack` | 1000 | – | ✅ | SQLite + WAL; needs a consistent copy |
| `trillium` | `triliumnext/notes:latest` | 1000 | – | ✅ | `document.db` + WAL; has its own `backup/` dir |
| `tandoor` | `vabene1111/recipes:latest` | 0 | – | ✅ | Needs `POSTGRES_HOST=db` alias |
| `sms-wa-bridge` | `sms-wa-bridge` (local) | 0 | – | ⚠️ | Chromium **is fine** (see below). Must be `localhost/sms-wa-bridge:latest` |
| `vince` | `ghcr.io/vinceanalytics/vince` | 0 | – | ⚠️ | **Distroless** — no shell, no `id`, so no exec-based checks. Container was run with `serve`, image `CMD` is null → set `Exec=serve` |
| `brewman-acc/exp/hinchco/hops/mhl` | `registry.tanshu.com/brewman:latest` | 0 | – | ✅ | 5 containers, identical config, one DB each |

### The Chromium question, answered

`sms-wa-bridge` is the classic rootless-Podman landmine: Chromium normally needs
`CLONE_NEWUSER` for its sandbox, which is blocked inside a container, and it can also
die on a 64 MB `/dev/shm`. I checked `server.js` and it already handles both:

```js
puppeteer: {
  headless: true,
  executablePath: process.env.CHROMIUM_PATH || undefined,
  args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"]
}
```

`--no-sandbox` removes the userns dependency and `--disable-dev-shm-usage` removes the
shm dependency. **No action needed** — this container will behave identically.

---

## 5. Your migration thesis — validated, with three corrections

> *"if we copy the data directories, change ownership and deploy on new server, we should be good to go"*

**Broadly yes** — and with downtime free, this is a much safer plan than an in-place
cutover, because you can stop everything cleanly and copy quiesced data. Three
corrections:

**Correction 1 — "the data directories" is not all the state.** Adding to `/var/lib/*`:
- the two **anonymous volumes** (§3.1) — `hiwikijs` is genuinely at risk;
- **`/var/lib/gitea/ssh`** (SSH host keys) — easy to miss, and clients will complain;
- **`/var/lib/caddy/data`** — the ACME account key and ~20 certificates. Lose it and you
  re-issue everything against Let's Encrypt rate limits;
- **the environment variables** — that is where the credentials live (Shopify client
  secret, Sentry token, JWT private keys, PocketBase passwords, DB passwords);
- **`/etc/crontab` entries and `~/bin/`** — the operational glue.

**Correction 2 — "change ownership" is per-container, not global.** Chowning everything
to `tanshu` is right, but it only works if each container is also given the matching
`UserNS=keep-id:uid=<app-uid>`. Get the pairing wrong and the app starts and then fails
to write — the failure mode is a permissions error at runtime, not a clean startup
failure. The §4 table gives the pairing.

**Correction 3 — Postgres wants a clean stop, and a dump is still worth it.** Copying
`PGDATA` after a clean shutdown is valid *at the same major version* (17.5 here). But a
logical dump is cheap and gives you a version-independent fallback. Do both.

Since downtime is not a constraint, the copying step should be:

```
1. Stop all containers cleanly.
2. Copy: /var/lib/<svc>/, /var/lib/gitea/ssh, the two anonymous volumes,
   /var/lib/caddy/data, ~/bin/, env vars, docker inspect JSON.
3. Verify checksums on the copy.
4. chown -R tanshu:tanshu on the destination, per §3.2.
5. Load images (docker save | podman load, or podman pull + podman login).
6. Write Quadlet units, set UserNS / Network / Volume.
7. systemctl --user daemon-reload && start the data tier, then apps, then Caddy.
```

Because you stop cleanly, the SQLite WAL problem from the backup plan largely
disappears — no live WAL, no torn copies. That is the single biggest simplification
this approach buys you.

---

## 6. Low ports and lingering

You already know both, confirming for completeness:

**Ports.** Rootless cannot bind < 1024. Caddy binds 80/443 **inside** its own network
namespace (where container-root has `CAP_NET_BIND_SERVICE`, which Podman grants by
default), but the **host-side publish** needs the sysctl:

```
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
```

Verify after reboot — this is a favourite thing to forget on a fresh instance.

**Lingering.** Without it, Quadlet **user** services stop when the login session ends,
which on a server is immediately, and they do not start at boot:

```
sudo loginctl enable-linger tanshu
```

Also confirm `/etc/subuid` and `/etc/subgid` have a range for the new user — monoco has
`tanshu:165536:65536`, and a fresh instance will assign a different base. Do not
hardcode `165536` anywhere.

---

## 7. Cutover notes specific to a new instance

**DNS and certificates are the thing to plan around.** All ~20 hostnames resolve to
`13.202.6.2`. On a new instance with a new IP:

- Caddy cannot obtain certificates until DNS points at the new host (HTTP-01), and the
  existing certs in `/var/lib/caddy/data` are only valid to *serve* for those names once
  traffic arrives.
- **Check whether `13.202.6.2` is an Elastic IP.** If it is, reassign it to the new
  instance and DNS, certificates, and clients are all untouched — by far the cleanest
  cutover. Worth confirming before designing around a DNS change.

**Logs do not migrate.** All 26 containers use the `json-file` driver; Podman uses
journald. If the existing logs matter, export them before decommissioning. Otherwise
`podman logs` / `journalctl --user -u <svc>` replaces `docker logs`.

**Image transfer.** `docker save <img> | podman load` avoids re-downloading (and avoids
registry auth for the local image). For `registry.tanshu.com`, run `podman login` on the
new host so `podman auto-update`/pulls work.

**The OS question is still open.** Per [podman-migration.md §3.4](podman-migration.md),
a **fresh 26.04 instance with stock Podman 5.7 needs no source builds at all**. Since you
are standing up a new instance anyway, this is now essentially free — and it makes the
"build Podman 6 for pesto" question entirely separable from the migration.

---

## 8. Per-service verification after cutover

For each container, in this order:

1. **Starts and stays up** — `systemctl --user status <svc>`
2. **Writes as the right uid** — create/update something through the app, then confirm
   on the host that the new file is owned by `tanshu`. This is the check that catches a
   wrong `keep-id`.
3. **Resolves its dependencies** — `podman exec <c> getent hosts postgres` (and `db` /
   `redis` aliases where used)
4. **Healthcheck reports healthy** — `podman inspect --format '{{.State.Health.Status}}'`
5. Special cases:
   - `caddy`: real client IPs in the access log (this is where the rootless `rootlessport`
     limitation shows up — see §3.4 of the migration plan); `https://<vhost>` responds;
     HTTP/3 (`curl --http3`) if you care
   - `sms-wa-bridge`: WhatsApp re-links (or restores) and a test message sends
   - `gitea`: `git clone` over **both** HTTPS and SSH (port 2222), no host-key warning
   - `postgres`: every client app connects; nothing listening on host `5432`
   - `nextcloud`: upload a file (exercises the uid-33 mapping and `REDIS_HOST`)
   - `vince`: reachable on `stats.mozimo.in` (distroless, so no shell to diagnose with)
6. **Reboot the host** and confirm everything returns unattended