playbooks/docs/podman-compat-audit.md

Podman compatibility audit — monoco container fleet

Status: planning only. Read-only inspection; nothing on monoco was modified. Scope: all 26 running containers, audited for rootless Podman + Quadlet. Companion docs: podman-migration.md, backup-plan.md

---

1. Verdict

Nothing in this fleet blocks a rootless Podman migration. There is no privileged container, no capability add, no device, no host networking, no custom seccomp, no Docker-socket mount, and no Compose file. Every image is arm64, so there is no emulation.

The breakages are all in the surrounding tooling (backup scripts, cron, Ansible modules) and in ownership/volume handling — not in the images. Your instinct that your own registry.tanshu.com images will be fine is correct; I checked them specifically and they are among the cleanest in the fleet.

Four things need real attention, and one of them is not obvious:

#	Issue	Severity
1	hiwikijs and valkey use anonymous Docker volumes — copying /var/lib/* misses them entirely	data loss
2	Every data tree needs re-owning, and each app's uid differs (0 / 33 / 70 / 999 / 1000 / 1003)	blocks startup
3	--link in backup scripts, and root's crontab doing docker exec	breaks backups + cron
4	Locally-built sms-wa-bridge has no registry — short-name resolution fails	won't start

---

2. What I verified as clean

Worth stating explicitly, because it is why this migration is tractable:

• Architecture: every image is arm64 / linux. No binfmt/qemu needed.
• Privileges: Privileged=false, CapAdd=[], CapDrop=[], Devices=[] on all 26.
• Namespaces: no --network host, no --pid host, no --ipc host, no --userns host.
• Security: no custom SecurityOpt, no custom seccomp profile, no --extra-hosts.
• Docker socket: not mounted anywhere. No image talks to the Docker API.
• Compose: none. Every container is a plain docker run / Ansible docker_container.
• shm: all at the 64 MB default; nothing needs a larger /dev/shm.
• Sysctls: none set.
• Resource limits: Memory=0, NanoCpus=0 on every container. (The Ansible role for gitea declares cpus: .10 / memory: 512M, but they are not in effect on the running container — so no cgroup-delegation concerns for rootless.)
• Custom DNS: Dns=[] everywhere; aardvark-dns will replace Docker's 127.0.0.11.
• Hostnames: all default (container ID). Nothing depends on a fixed hostname — sms-wa-bridge explicitly cleans its own Singleton* locks on start because of this.

---

3. The four real problems

3.1 Anonymous volumes — the one that loses data

Eight images declare VOLUME:

Image	VOLUME	Mounted?
vaultwarden/server	/data	✅ bind-mounted
gitea/gitea	/data	✅ bind-mounted
registry.tanshu.com/{greatbear,hops}	/app/pb_data	✅ bind-mounted
nextcloud	/var/www/html	✅ bind-mounted
postgres:17-alpine	/var/lib/postgresql/data	✅ bind-mounted
ghcr.io/requarks/wiki	/wiki/data/content	❌ anonymous volume
valkey/valkey:alpine	/data	❌ anonymous volume

The two anonymous ones live in /var/lib/docker/volumes/<hash>/_data, not under /var/lib/<service>. A copy of the data directories will silently omit them.

hiwikijs is the serious one: /var/lib/hiwikijs exists but is an empty directory from Dec 2022, and ~/bin/hiwikijs-backup.sh archives exactly that empty directory and exits successfully. So Wiki.js currently has no backup at all — this predates the migration and should be fixed regardless of it.

Fix: export both to real directories (e.g. ~/podman-data/hiwikijs/content, ~/podman-data/valkey) and declare them as Quadlet Volume= bind mounts. valkey is a rebuildable cache; hiwikijs is not.

3.2 Ownership — one uid per container, and they all differ

Under rootless Podman the user namespace maps only one uid:gid to your host user. The rule is UserNS=keep-id:uid=<app-uid>,gid=<app-gid>, where <app-uid> is the uid the application actually writes as — not the image's USER and not docker exec id.

Measured (see §4 for the full matrix):

App uid	Containers
0	caddy, brewman ×5, greatbear, hops, affine, audiobookshelf, bitwarden, tandoor, sms-wa-bridge, vince
33	nextcloud
70	postgres
999	valkey
1000	mozimo, mozimo-staging, hiwikijs, menu-app ×3, yamtrack, trillium
1003	gitea

Trap: docker exec <c> id -u reports the image's configured user, not the running process. It says 0 for postgres (which actually writes as 70), for nextcloud (33), and for gitea (1003) — because those entrypoints drop privileges after starting. Derive the uid from the ownership of what the app writes, which is what the table above does.

Also note tanshu is uid 1001, not 1000 — most of the uid-1000 data is currently owned by the ubuntu user. "Chown it to me" is therefore not a no-op anywhere.

The reassuring part: for the entrypoints that start as root and drop to an app user (postgres, nextcloud, gitea, trillium, yamtrack), the root phase maps to a subuid and its internal chown still lands on your host user. So a single keep-id:uid=<app-uid> plus chown -R tanshu:tanshu should work — but verify per service, because a file written during the root phase and never chowned will be subuid-owned and unreadable by the dropped-privilege process.

3.3 Tooling that assumes Docker

• --link postgres:db in db-backup.sh, bitwarden-, hiwikijs-, kakawa-, nextcloud-, tandoor-backup.sh and their restores. No Podman equivalent.
• Root's crontab: */5 * * * * docker exec -u www-data nextcloud php cron.php. Root cannot podman exec into another user's rootless containers — this must become a systemd user timer.
• ~/bin/empty-trash.sh execs a container named cloud, which no longer exists (it is nextcloud).
• Ansible: docker_container / docker_image / docker_network → containers.podman. Note podman_network has no connected:/appends: parameters, which every */roles/network/tasks/main.yaml in this repo relies on. Declare the full network: list on each podman_container instead.
• Everything outside this repo. You said ~90% is Ansible and the rest is docker run. The deploy pipelines for mozimo, brewman, menu-admin, hops, greatbear and sms-wa-bridge live elsewhere — if they still call docker against the host, they will recreate Docker containers alongside your Podman ones.

3.4 Image naming

Rootless Podman resolves short names against unqualified-search-registries. In a Quadlet/systemd context there is no TTY to confirm a short-name prompt, so ambiguous names fail the unit rather than prompting.

• sms-wa-bridge was built locally with docker build -t sms-wa-bridge — no registry. Podman will try docker.io/library/sms-wa-bridge and fail. Transfer it with docker save sms-wa-bridge | podman load and reference localhost/sms-wa-bridge:latest.
• nextcloud, caddy, postgres, valkey/valkey, gitea/gitea, vaultwarden/server, vabene1111/recipes, triliumnext/notes are all short or implicit-docker.io names. Fully qualify all of them (docker.io/library/nextcloud:latest etc.) so resolution never depends on host config.
• registry.tanshu.com/* and ghcr.io/* are already qualified. ✅

---

4. Per-container matrix

App uid is what the application writes as. Anon vol = uses an anonymous Docker volume.

Container	Image	App uid	Anon vol	Verdict	Notes
caddy	caddy:latest	0	–	⚠️	Needs ip_unprivileged_port_start≤80; publishes 80, 443, 443/udp (HTTP/3) — verify UDP forwarding
postgres	postgres:17-alpine	70	–	⚠️	PGDATA is pgdata/ under the mount, owned uid 70. Drop the 0.0.0.0:5432 publish
valkey	valkey/valkey:alpine	999	YES	⚠️	Export the anonymous volume
nextcloud	nextcloud:latest	33	–	✅	VOLUME /var/www/html is bind-mounted; needs POSTGRES_HOST=db / REDIS_HOST=redis aliases
gitea	gitea/gitea:latest	1003	–	⚠️	/var/lib/gitea/ssh holds SSH host keys — copy them or clients see host-key warnings
mozimo	registry.tanshu.com/mozimo:4a3863a	1000	–	✅	Image USER=node; entrypoint starts PocketBase + SSR together
mozimo-staging	registry.tanshu.com/mozimo:3929d3f	1000	–	✅
greatbear	registry.tanshu.com/tanshu/greatbear	0	–	✅	VOLUME /app/pb_data bind-mounted
hops	registry.tanshu.com/hops:latest	0	–	✅	VOLUME /app/pb_data bind-mounted
menu-app.greatbear.in	registry.tanshu.com/menu-admin	1000	–	⚠️	Container names contain dots — name the Quadlet units menu-greatbear.container etc. and set ContainerName= explicitly so the Caddyfile keeps resolving
menu-app.hngmohali.com	″	1000	–	⚠️	″
menu-app.hopsngrains.com	″	1000	–	⚠️	″
hiwikijs	ghcr.io/requarks/wiki:latest	1000	YES	❌	Content is in an anonymous volume; /var/lib/hiwikijs is an empty decoy
affine	ghcr.io/toeverything/affine-graphql:stable	0	–	✅	Needs postgres + valkey; indexer disabled (AFFINE_INDEXER_ENABLED=false), so no pgvector dependency
audiobookshelf	ghcr.io/advplyr/audiobookshelf	0	–	✅	tini entrypoint; 15 GB is media, state is 22 MB
bitwarden	vaultwarden/server:latest	0	–	⚠️	Authoritative store is Postgres; the db.sqlite3 on disk is stale — do not restore it
yamtrack	ghcr.io/fuzzygrim/yamtrack	1000	–	✅	SQLite + WAL; needs a consistent copy
trillium	triliumnext/notes:latest	1000	–	✅	document.db + WAL; has its own backup/ dir
tandoor	vabene1111/recipes:latest	0	–	✅	Needs POSTGRES_HOST=db alias
sms-wa-bridge	sms-wa-bridge (local)	0	–	⚠️	Chromium is fine (see below). Must be localhost/sms-wa-bridge:latest
vince	ghcr.io/vinceanalytics/vince	0	–	⚠️	Distroless — no shell, no id, so no exec-based checks. Container was run with serve, image CMD is null → set Exec=serve
brewman-acc/exp/hinchco/hops/mhl	registry.tanshu.com/brewman:latest	0	–	✅	5 containers, identical config, one DB each

The Chromium question, answered

sms-wa-bridge is the classic rootless-Podman landmine: Chromium normally needs CLONE_NEWUSER for its sandbox, which is blocked inside a container, and it can also die on a 64 MB /dev/shm. I checked server.js and it already handles both:

puppeteer: {
  headless: true,
  executablePath: process.env.CHROMIUM_PATH || undefined,
  args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"]
}

--no-sandbox removes the userns dependency and --disable-dev-shm-usage removes the shm dependency. No action needed — this container will behave identically.

---

5. Your migration thesis — validated, with three corrections

"if we copy the data directories, change ownership and deploy on new server, we should be good to go"

Broadly yes — and with downtime free, this is a much safer plan than an in-place cutover, because you can stop everything cleanly and copy quiesced data. Three corrections:

Correction 1 — "the data directories" is not all the state. Adding to /var/lib/*:

• the two anonymous volumes (§3.1) — hiwikijs is genuinely at risk;
• /var/lib/gitea/ssh (SSH host keys) — easy to miss, and clients will complain;
• /var/lib/caddy/data — the ACME account key and ~20 certificates. Lose it and you re-issue everything against Let's Encrypt rate limits;
• the environment variables — that is where the credentials live (Shopify client secret, Sentry token, JWT private keys, PocketBase passwords, DB passwords);
• /etc/crontab entries and ~/bin/ — the operational glue.

Correction 2 — "change ownership" is per-container, not global. Chowning everything to tanshu is right, but it only works if each container is also given the matching UserNS=keep-id:uid=<app-uid>. Get the pairing wrong and the app starts and then fails to write — the failure mode is a permissions error at runtime, not a clean startup failure. The §4 table gives the pairing.

Correction 3 — Postgres wants a clean stop, and a dump is still worth it. Copying PGDATA after a clean shutdown is valid at the same major version (17.5 here). But a logical dump is cheap and gives you a version-independent fallback. Do both.

Since downtime is not a constraint, the copying step should be:

1. Stop all containers cleanly.
2. Copy: /var/lib/<svc>/, /var/lib/gitea/ssh, the two anonymous volumes,
   /var/lib/caddy/data, ~/bin/, env vars, docker inspect JSON.
3. Verify checksums on the copy.
4. chown -R tanshu:tanshu on the destination, per §3.2.
5. Load images (docker save | podman load, or podman pull + podman login).
6. Write Quadlet units, set UserNS / Network / Volume.
7. systemctl --user daemon-reload && start the data tier, then apps, then Caddy.

Because you stop cleanly, the SQLite WAL problem from the backup plan largely disappears — no live WAL, no torn copies. That is the single biggest simplification this approach buys you.

---

6. Low ports and lingering

You already know both, confirming for completeness:

Ports. Rootless cannot bind < 1024. Caddy binds 80/443 inside its own network namespace (where container-root has CAP_NET_BIND_SERVICE, which Podman grants by default), but the host-side publish needs the sysctl:

sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf

Verify after reboot — this is a favourite thing to forget on a fresh instance.

Lingering. Without it, Quadlet user services stop when the login session ends, which on a server is immediately, and they do not start at boot:

sudo loginctl enable-linger tanshu

Also confirm /etc/subuid and /etc/subgid have a range for the new user — monoco has tanshu:165536:65536, and a fresh instance will assign a different base. Do not hardcode 165536 anywhere.

---

7. Cutover notes specific to a new instance

DNS and certificates are the thing to plan around. All ~20 hostnames resolve to 13.202.6.2. On a new instance with a new IP:

• Caddy cannot obtain certificates until DNS points at the new host (HTTP-01), and the existing certs in /var/lib/caddy/data are only valid to serve for those names once traffic arrives.
• Check whether 13.202.6.2 is an Elastic IP. If it is, reassign it to the new instance and DNS, certificates, and clients are all untouched — by far the cleanest cutover. Worth confirming before designing around a DNS change.

Logs do not migrate. All 26 containers use the json-file driver; Podman uses journald. If the existing logs matter, export them before decommissioning. Otherwise podman logs / journalctl --user -u <svc> replaces docker logs.

Image transfer. docker save <img> | podman load avoids re-downloading (and avoids registry auth for the local image). For registry.tanshu.com, run podman login on the new host so podman auto-update/pulls work.

The OS question is still open. Per podman-migration.md §3.4, a fresh 26.04 instance with stock Podman 5.7 needs no source builds at all. Since you are standing up a new instance anyway, this is now essentially free — and it makes the "build Podman 6 for pesto" question entirely separable from the migration.

---

8. Per-service verification after cutover

For each container, in this order:

1. Starts and stays up — systemctl --user status <svc>
2. Writes as the right uid — create/update something through the app, then confirm on the host that the new file is owned by tanshu. This is the check that catches a wrong keep-id.
3. Resolves its dependencies — podman exec <c> getent hosts postgres (and db / redis aliases where used)
4. Healthcheck reports healthy — podman inspect --format '{{.State.Health.Status}}'
5. Special cases:
   • caddy: real client IPs in the access log (this is where the rootless rootlessport limitation shows up — see §3.4 of the migration plan); https://<vhost> responds; HTTP/3 (curl --http3) if you care
   • sms-wa-bridge: WhatsApp re-links (or restores) and a test message sends
   • gitea: git clone over both HTTPS and SSH (port 2222), no host-key warning
   • postgres: every client app connects; nothing listening on host 5432
   • nextcloud: upload a file (exercises the uid-33 mapping and REDIS_HOST)
   • vince: reachable on stats.mozimo.in (distroless, so no shell to diagnose with)
6. Reboot the host and confirm everything returns unattended