playbooks/docs/podman-migration.md

Migrating monoco from Docker to rootless Podman 6 + Quadlet

Status: plan, not yet executed Revised for: rootless Podman 6.x, data in ~/podman-data/, per-app networks, systemd Quadlet units in ~/.config/containers/systemd/ Audited: live monoco — Ubuntu 24.04.5 aarch64, 2 vCPU Neoverse-N1, 7.6 GB RAM, Docker 29.8.1, 26 containers

This replaces an earlier rootful draft. The rootful sections are gone; §2 explains what the rootless choice costs, so the tradeoff is on the record rather than a surprise later.

---

1. Decisions locked in

Decision	Value
Runtime	Rootless Podman 6.x, built from source
Units	Quadlet .container / .network files in ~/.config/containers/systemd/
Data	~/podman-data/<service>/ (was /var/lib/<service>/)
Networks	One per logical app, as today — not consolidated
Client IPs	Preserved via pesto (rootless_port_forwarder = "pasta")
Caddy	Rootless, multi-homed onto every app network

---

2. What this costs — read once, then proceed

You asked whether Podman 6 can be built from source. Yes. But three things should be on the record before we start, because they are not obvious from the request:

1. Building 6.x solves one problem and creates a permanent maintenance obligation. The only thing the 6.x build buys over stock 4.9 is real client IPs at Caddy (via pesto). Everything else in this plan works on stock Podman. In exchange you own five source-built projects across three toolchains, with no distro security updates, on a host serving ~20 sites. Pesto itself is upstream-described as not default, with no graceful upgrade path. §3.4 shows how upgrading to Ubuntu 26.04 removes the source builds entirely — read that before committing to §3.

2. Building 6.x does not reduce the rootless work at all. The expensive part of rootless is re-owning ~19 GB of data into a user namespace (§4). That cost is identical on 4.9, 5.x, or 6.x. If the goal is "off Docker soon", the version is orthogonal.

3. monoco is a 2 vCPU / 7.6 GB ARM64 box with ~2.5 GB RAM free. Do not compile on it (§3.2). Rust builds of netavark will thrash a machine that is already running 26 containers.

None of this argues against doing it — rootless is a legitimate goal and pesto is a real fix. It argues for building elsewhere and phasing carefully.

---

3. Building Podman 6.x + Netavark 2 + pesto from source

3.1 Component and version matrix

Podman 6.0.0 (released 2026-06-24, latest patch 6.0.1) is not packaged for Ubuntu 24.04 by anyone. I checked devel:kubic:libcontainers:unstable (has netavark 1.12.2, aardvark-dns 1.13.1, conmon, fuse-overlayfs — no podman), isv:containers, and devel:kubic:libcontainers:testing. Debian unstable is still on podman 5.8.1 with a bug open requesting 6.0.1. So source is genuinely the only path.

Component	Version needed	Ubuntu 24.04 has	Gap
podman	6.0.x	4.9.3	build
— Go toolchain	1.25.6 (go.mod)	go 1.22	bootstrap Go
netavark	2.x (latest 2.1.0) — hard requirement	1.4.0	build
aardvark-dns	2.x	1.4.0	build
— Rust toolchain	1.88 (rust-version)	rustc 1.75	bootstrap rustup
passt (ships pesto)	≥ 0^20260507.g1afd4ed	0^20240220	build
container-libs common	v0.68.0	—	build
buildah / skopeo	1.44.0 / 1.23	optional	only if you build/push images on monoco
libgpgme-dev, libbtrfs-dev, libseccomp-dev, libassuan-dev	any	✅ present	none
build-essential, pkg-config	any	❌ gcc not installed	apt install

Podman 6 breaking changes that matter here, all of which monoco satisfies:

• iptables support removed → nftables required. Host has nft 1.0.9 ✅
• CNI removed → netavark. Fine.
• slirp4netns removed → pasta required.
• cgroups v1 removed → stat -fc %T /sys/fs/cgroup/ = cgroup2fs ✅
• BoltDB removed → SQLite.
• Netavark and Aardvark must be 2.0.0 — 1.x will not work.

3.2 Build off-box

monoco has 2 vCPU, 7.6 GB RAM, ~2.5 GB available, and 26 production containers. Rust compilation of netavark + aardvark-dns is the worst case for that profile.

Build on a separate aarch64 machine or in CI, then ship artifacts. All three projects are portable:

• Podman / container-libs: Go — CGO_ENABLED=0 cross-compiles trivially; for GOARCH=arm64 from any host.
• netavark / aardvark-dns: Rust — cargo build --release --target aarch64-unknown-linux-gnu.
• passt/pesto: plain C make.

Preferred output: four .deb packages (podman, netavark, aardvark-dns, passt) so apt tracks them and you can rebuild on security updates. A tarball + install script is acceptable but you lose upgrade tracking.

Build reference:

• Podman: go.podman.io/podman/v6 (the import path moved to a CNCF org in 6.0)
• netavark: make in containers/netavark, installs netavark + config
• aardvark-dns: containers/aardvark-dns
• passt: passt.top, installs passt, pasta, and pesto

3.3 Install order on monoco

1. apt install build-essential pkg-config libgpgme-dev libbtrfs-dev libseccomp-dev libassuan-dev
2. apt install passt         # baseline; then replace with your built passt+pesto
3. install netavark 2.x + aardvark-dns 2.x   (to /usr/libexec/podman/)
4. install podman 6.x                        (to /usr/bin/podman)
5. verify: podman info --debug | grep -E 'networkBackend|version'

Keep Ubuntu's podman 4.9 package uninstalled — do not let apt install it as a dependency and shadow your build. Pin with apt-mark hold if something pulls it in.

3.4 What about upgrading to Ubuntu 26.04 LTS first?

Short answer: 26.04 is a large improvement and worth doing, but it does not give you pesto — so it does not remove the source builds if client IPs are the goal.

26.04 "Resolute" ships (vs 24.04 today):

Package	24.04 (now)	26.04 LTS
podman	4.9.3	5.7.0
passt	0.0~git20240220	0.0~git20260120
netavark	1.4.0	1.16.1
aardvark-dns	1.4.0	1.16.0
crun	—	1.21
buildah	—	1.42.1

Two independent blockers, either of which alone is fatal:

1. rootless_port_forwarder is Podman 6.0+. The implementing PR (containers/podman#28478) merged 2026-05-21 into main and shipped in 6.0.0 on 2026-06-24. 26.04 carries 5.7.0, which predates it. The Podman 6 announcement lists pesto as a 6.0 feature, and the option is documented in container-libs main as rootlessport (default) | pasta.
2. pesto needs passt ≥ 0^20260507. 26.04 ships 0^20260120 — about three and a half months too old.

Also note Podman 6 hard-requires netavark/aardvark-dns 2.0.0, and 26.04 has 1.16 — so moving to 6.x means building those too, on any base.

The useful reframe: pesto is a separable, optional improvement, not a migration prerequisite. Losing real client IPs degrades Caddy access logs, X-Forwarded-For downstream, Nextcloud/Vaultwarden IP-based brute-force protection, and Vince's server-side analytics — but it does not break the reverse proxy, and nothing goes down. That means the migration should not wait on it.

Recommended sequencing (revising §6 of the earlier draft):

#	Step	Notes
1	Full backup + verified restores	see docs/backup-plan.md
2	OS upgrade 24.04 → 26.04, Docker still running	supported LTS→LTS path; needs its own window
3	Migrate Docker → Podman on stock 5.7, ~/.config/containers/systemd/	no source builds at all
4	Later, separately: decide whether client IPs justify Podman 6 + pesto	orthogonal; Quadlet units unchanged

Doing the OS upgrade first matters: migrating to a self-built Podman 6 on 24.04 and then upgrading the OS leaves hand-built binaries fighting the distro packages at exactly the moment you least want that.

The tradeoff, stated plainly (see §2 for the rootless case):

	Source builds	Data re-owning	Real client IPs
Rootless, 26.04, stock 5.7	none	required	❌
Rootless, 26.04, built 6.0 + pesto	4 projects	required	✅ (experimental)
Rootful, 26.04, stock 5.7	none	not required	✅

---

4. The uid mapping problem — the biggest piece of actual work

This is where rootless diverges hardest from Docker, and it is worth doing carefully.

4.1 The trap: tanshu is not uid 1000

id tanshu   → uid=1001(tanshu) gid=1002(tanshu)

Most container data on this host is owned by uid 1000, which is the ubuntu user, not you. So "the data is already owned by my user" is false everywhere. Anything you don't explicitly re-own will be invisible inside a rootless container.

Measured ownership of the actual bind-mount sources:

Container	Runs as (in-container)	Data source	Current host owner
mozimo, mozimo-staging	1000 (node)	/var/lib/mozimo/pb_data	1000:1000
hiwikijs	1000 (node)	(anon volume)	1000:1000
menu-app.* (×3)	1000:1000	/var/lib/menu.*	1000:1000
vince	1000	/var/lib/mozimo/vince_data	1000:1000
yamtrack	1000	/var/lib/yamtrack/data	1000:1000
trillium	1000 (node)	/var/lib/trillium	1000:1000
nextcloud	33 (www-data)	/var/lib/nextcloud	33:33
postgres	70	/var/lib/postgresql/data/pgdata	70:0
valkey	999	(anon volume)	999:1000
gitea	1001 / 1003 (git)	/var/lib/gitea, /home/git/.ssh	1001:1002, 1003:1003
caddy, audiobookshelf, affine, bitwarden, sms-wa-bridge, greatbear, hops, tandoor, brewman-*	0	various	0:0

The postgres entry is the important one to sanity-check: docker exec reports uid 0 (that's the image's configured user) but PGDATA is owned by 70, which is the uid the server actually runs as. Derive the app uid from what the app writes, not from docker exec id — the entrypoint in postgres, nextcloud, gitea, tandoor and trillium drops privileges after starting.

4.2 The rule

Only one container uid:gid can be mapped to your host user. Use it for whichever uid the app actually writes as:

UserNS=keep-id:uid=<app-uid>,gid=<app-gid>

Service	Directive
mozimo, mozimo-staging, hiwikijs, menu-app.*, vince, yamtrack, trillium	UserNS=keep-id:uid=1000,gid=1000
nextcloud	UserNS=keep-id:uid=33,gid=33
postgres	UserNS=keep-id:uid=70,gid=70
valkey	UserNS=keep-id:uid=999,gid=999
gitea	UserNS=keep-id:uid=1001,gid=1002 ⚠︎ verify
caddy, audiobookshelf, affine, bitwarden, sms-wa-bridge, greatbear, hops, tandoor, brewman-*	UserNS=keep-id:uid=0,gid=0

Then chown every data tree to tanshu:tanshu (1001:1002). That is the whole trick: the app's uid inside the container is your uid on the host, so files stay readable and writable by you, and tar/pg_dump backups keep working without podman unshare or sudo.

Do not use the default rootless mapping (container 0 → tanshu) for the non-root apps — files would come out owned by subuid 165536+, invisible to your backup scripts. keep-id is what keeps the host side clean.

⚠︎ gitea writes as two different uids (1001:1002 for /data, 1003:1003 for /data/git/.ssh), and keep-id maps only one. Resolve this before cutover: either relax the .ssh ownership, or mount /home/git/.ssh from a path owned by 1001.

4.3 The chown pass

Data moves from /var/lib/X to ~/podman-data/X. /home/tanshu and /var/lib are both on / (/dev/root), so mv is an instant rename, not a copy — verify with findmnt -T /var/lib/<svc> that none of them is a separate mount first.

# after stopping Docker, per service:
mv /var/lib/<svc> ~/podman-data/<svc>
chown -R tanshu:tanshu ~/podman-data/<svc>

chown -R is metadata-only, but audiobookshelf is 15 GB of many small files, so budget a few minutes for that one. Total data is ~19 GB:

audiobookshelf 15G | nextcloud 1.2G | postgres 995M | gitea 747M
sms-wa-bridge 397M | mozimo 147M | mozimo-staging 137M | trillium 106M
greatbear 101M | tandoor 98M | menu.* 3×70M | hops 19M | caddy 736K | affine 916K

Plus two anonymous volumes that must be exported to real directories (they are invisible to your backup scripts today):

Container	Volume	→
hiwikijs	9c1e3fde… (owned 1000:1000)	~/podman-data/hiwikijs/content
valkey	6089e6fc… (owned 999:1000)	~/podman-data/valkey

docker run --rm -v 9c1e3fde…:/from -v ~/podman-data/hiwikijs:/to alpine \
  sh -c 'cd /from && cp -a . /to/content'

---

5. Target architecture

5.1 Layout

~/.config/containers/systemd/
  caddy.container
  postgres.container
  nextcloud.container
  nextcloud.network
  gitea.network
  menu-hopsngrains.network
  … one .network per logical app

~/podman-data/
  caddy/{conf,data,config,site}     ← caddyfile_path becomes this
  postgres/  nextcloud/  gitea/  audiobookshelf/  … 
  caddy/conf/Caddyfile              ← regenerated by the Ansible caddy roles

5.2 Network naming — a Quadlet quirk to internalise

A Quadlet file $NAME.network produces a Podman network actually called systemd-$NAME. So nextcloud.network → network systemd-nextcloud.

That is cosmetic for DNS (containers resolve each other by container name and alias), but it matters for:

• podman network ls output — do not be confused by the prefix.
• Any podman network connect you run by hand.
• The .network file is only read if some .container references it — networks are not created standalone.

A minimal network file is enough; DNS is on by default:

# ~/.config/containers/systemd/nextcloud.network
[Network]

5.3 A service unit

# ~/.config/containers/systemd/nextcloud.container
[Unit]
Description=Nextcloud
After=network-online.target

[Container]
ContainerName=nextcloud
Image=docker.io/library/nextcloud:latest
Network=nextcloud.network
UserNS=keep-id:uid=33,gid=33
Volume=%h/podman-data/nextcloud:/var/www/html
Volume=/etc/localtime:/etc/localtime:ro

[Service]
Restart=always
TimeoutStartSec=300

[Install]
WantedBy=default.target

%h is the systemd specifier for your home directory and works in Quadlet.

5.4 Postgres and valkey: multi-homed with aliases

NetworkAlias= exists in Quadlet 6.0, but it applies the alias to all networks the container joins. Your aliases are per-network (db on only 3 of 13), so use the Network= alias suffix instead — Quadlet passes it straight through to podman:

# ~/.config/containers/systemd/postgres.container
[Container]
ContainerName=postgres
Image=docker.io/library/postgres:17-alpine
UserNS=keep-id:uid=70,gid=70
Network=nextcloud.network:alias=db
Network=tandoor.network:alias=db
Network=hiwikijs.network:alias=db
Network=affine.network
Network=bitwarden.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Volume=%h/podman-data/postgres:/var/lib/postgresql/data

This is the direct analogue of today's 13-network Postgres hub. valkey is the same pattern with Network=nextcloud.network:alias=redis plus affine, yamtrack.

Pin postgres:17-alpine. The running container reports image postgres:17-alpine but its image ID is the one now tagged postgres:alpine; the server is 17.5.

5.5 Caddy: multi-homed, exactly as today

# ~/.config/containers/systemd/caddy.container
[Unit]
After=network-online.target postgres.service

[Container]
ContainerName=caddy
Image=docker.io/library/caddy:latest
UserNS=keep-id:uid=0,gid=0
PublishPort=80:80
PublishPort=443:443
PublishPort=443:443/udp
Volume=%h/podman-data/caddy/conf:/etc/caddy
Volume=%h/podman-data/caddy/site:/srv
Volume=%h/podman-data/caddy/data:/data
Volume=%h/podman-data/caddy/config:/config
Network=nextcloud.network
Network=gitea.network
Network=mozimo.network
Network=trillium.network
Network=tandoor.network
Network=hiwikijs.network
Network=bitwarden.network
Network=yamtrack.network
Network=affine.network
Network=audiobookshelf.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Network=menu-hopsngrains.network
Network=menu-greatbear.network
Network=menu-hngmohali.network
Network=greatbear.network
Network=hops.network
Network=sms-wa-bridge.network

[Service]
Restart=always

[Install]
WantedBy=default.target

The Caddyfile itself needs no changes — every upstream is still a container name. Only caddyfile_path in the Ansible vars moves to /home/tanshu/podman-data/caddy/conf/Caddyfile.

Note Network=foo.network in Caddy's unit makes Quadlet order the network units first, so the "Caddy must be connected after the app exists" fragility of the current Docker setup disappears entirely.

Drop these three — they are stale (Caddy holds interfaces for networks with no service behind them): iot_net (the Caddyfile's iot.tanshu.com → grafana:3000 is dead), n8n_net, kakawa_net. Remove that Caddyfile block too.

---

6. Enabling pesto — do it before the first container starts

Deferred by default. See §3.4: this whole section only applies if you decide the client-IP regression justifies building Podman 6 + passt. On Ubuntu 26.04 with stock Podman 5.7, this option does not exist, and the migration proceeds without it. If you do enable pesto, do it before any container starts — the switch has no graceful upgrade path.

Pesto is what preserves real client IPs. Per the Podman blog, switching the forwarder on a running system has no graceful upgrade path, so enable it before bringing anything up.

# ~/.config/containers/containers.conf   (user-level, rootless)
[network]
rootless_port_forwarder = "pasta"

Traffic path becomes: client → host:port → pasta → rootless netns → netavark bridge → container. Pasta forwards without rewriting the source address, so Caddy sees the real client.

Two prerequisites:

1. passt with pesto — your source build from §3. Verify: which pesto.
2. Low ports. Rootless cannot bind < 1024. Caddy needs 80 and 443:

   sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
   # persist:
   echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
   

   This is a root-level change — unavoidable for rootless on privileged ports.

Good news on IPv6: pasta's main behavioural difference from rootlessport is that it does not transparently forward IPv6 host traffic to IPv4, so services need explicit IPv6 binding. I checked: monoco has no global IPv6 address and no default IPv6 route, and mozimo.in, git.tanshu.com, cloud.hopsngrains.com, vault.tanshu.com resolve to IPv4 only (no AAAA). So this limitation does not affect you.

Enable lingering, or nothing survives logout or reboot:

sudo loginctl enable-linger tanshu

Currently Linger=no. Without this, Quadlet user services stop when your last session ends — which on a server means immediately.

---

7. Cutover sequence

Pre-flight (no downtime):

1. Capture ground truth: docker inspect for every container/network/volume → JSON.
2. Resolve the gitea dual-uid question (§4.2) and confirm each app's true uid.
3. Export the two anonymous volumes (§4.3).
4. Test restores, not just backups — at least the Postgres dumps.
5. Back up /var/lib/caddy/data explicitly. It holds the ACME account key and all issued certificates. It is a bind mount so it carries over, but losing it means re-issuing ~20 certs against Let's Encrypt rate limits.
6. Build and install the Podman 6 stack (§3). podman login registry.tanshu.com.
7. Pre-pull every image. For locally-built sms-wa-bridge:latest (no registry prefix) avoid a re-download: docker save sms-wa-bridge:latest | podman load.
8. Write and review all Quadlet units; systemctl --user daemon-reload and check systemctl --user list-unit-files | grep -E 'caddy|nextcloud'.
9. Take a provider snapshot. Cheapest, fastest rollback you will ever buy.
10. Rehearse on a clone of the host, including a reboot.

Window (expect 10–20 minutes; 26 containers plus a 15 GB chown):

# 1. Stop the edge first so no request is half-served
docker stop caddy
docker stop $(docker ps -q)

# 2. Freeze state; nothing may write to /var/lib during the move
sudo systemctl stop docker.socket docker.service containerd.service
sudo systemctl disable docker.service

# 3. Move + re-own data (instant rename, then a metadata walk)
mv /var/lib/<svc> ~/podman-data/<svc> && chown -R tanshu:tanshu ~/podman-data/<svc>

# 4. Bring up the data tier first
systemctl --user start postgres.service valkey.service

# 5. Verify DNS inside the namespace BEFORE starting Caddy
podman exec nextcloud getent hosts postgres
podman exec nextcloud getent hosts redis

# 6. App tier, then Caddy last
systemctl --user start <each app>
systemctl --user start caddy.service

# 7. Reload + smoke test
podman exec caddy caddy reload

Do not docker rm anything. Stopped containers are the fastest rollback and cost only disk. Keep them a week.

---

8. What must change in this repo and on the host

(a) Legacy --link in the backup scripts — a guaranteed break. db-backup.sh, nextcloud-backup.sh, hiwikijs-, tandoor-, kakawa-, bitwarden-backup.sh and their -restore.sh counterparts all do docker run --rm --link postgres:db … pg_dump -h db. --link is Docker-only and is absent from podman run. Under rootless there is no default bridge network to fall back on either.

Best fix — drop the helper container and dump in place (no networking, no --user juggling, no shared network):

podman exec -e PGPASSWORD="..." postgres \
  pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"

Also pin the base image: the scripts mix postgres:alpine and postgres:17-alpine.

Bonus: once data is owned by tanshu and reachable without sudo, the sudo tar and chown $(logname) gymnastics in nextcloud-backup.sh can go away.

(b) Root's crontab cannot reach rootless containers.

*/5 * * * * docker exec -u www-data nextcloud php cron.php

Root cannot podman exec into another user's rootless containers. Move this to a systemd user timer alongside the Quadlet units — which is the idiomatic approach anyway. Same for ~/bin/empty-trash.sh, which also targets a container named cloud that no longer exists (it is nextcloud).

(c) Ansible modules. docker_container / docker_image / docker_network become containers.podman's podman_container / podman_image / podman_network, which shell out to the podman CLI and need no Python SDK — the pip: name=docker task disappears. Note podman_network lacks docker_network's connected:/appends: parameters, which the */roles/network/tasks/main.yaml files rely on. Prefer declaring the full network: list on each podman_container and deleting those network roles — that is precisely what Quadlet does, so the two stay in sync.

Also update caddyfile_path everywhere (now /home/tanshu/podman-data/caddy/conf/Caddyfile).

(d) ~/.zshrc:128 and any other docker invocations in ~/bin, /etc/cron*, and the external deploy pipelines for mozimo, brewman, menu-admin, hops, greatbear, sms-wa-bridge. Those live outside this repo. If they are not updated, the next deploy will recreate Docker containers next to your Podman ones.

(e) postgres currently publishes 5432 on 0.0.0.0 and UFW is inactive. Fix while you are in here — it needs no published port at all, only expose/network access.

---

9. Rollback

1. docker start — old containers exist, stopped but not removed; bind mounts untouched. Reverting is systemctl start docker + docker start $(docker ps -aq). Valid as long as you do not docker rm. ⚠︎ This only works if no app performed an irreversible migration on first start under Podman (Postgres major upgrade, Nextcloud schema migration, Trillium migration). Check that risk per service, and prefer the snapshot over trusting this.
2. Provider snapshot — instant whole-host revert. Covers /var/lib damage and certificate loss.
3. Backups — once §8(a) is fixed and restores are verified.

Note that the chown pass is itself hard to unwind precisely: reverting to Docker requires the original ownership (0:0, 33:33, 70:0, 1000:1000). Record the table in §4.1 before you start, and re-chown on rollback.

---

10. Verification checklist

Linkage

• podman exec caddy getent hosts <name> resolves for all 20 upstreams
• getent hosts db and getent hosts redis resolve from the apps that use them
• Every container is on the same set of networks it had under Docker

Rootless plumbing

• loginctl show-user tanshu | grep Linger → Linger=yes
• which pesto returns a path; rootless_port_forwarder is set before first start
• Caddy sees real client IPs — check podman logs caddy for a genuine remote IP
• X-Forwarded-For reaches Nextcloud/Gitea correctly

HTTP / TLS

• curl -sSI https://<vhost> 200/redirect for all 20 site blocks
• Certificates are the existing ones (compare serial against the pre-migration snapshot) — nothing re-issued
• WebSocket / streaming paths (Gitea, Nextcloud, RustDesk)

Applications

• Each app can read and write — catches the uid mapping (§4.2)
• Postgres accepts connections from every client
• postgres is no longer published on 0.0.0.0:5432

Operations

• Nextcloud cron fires via the new systemd user timer
• A full backup cycle of every ~/bin/*-backup.sh completes and restores
• Reboot the host; all containers return with correct policies
• systemctl --user status caddy is clean; podman exec caddy caddy reload works
• Disk is acceptable after Docker images are pruned (32 GB reclaimable)

---

11. Risk register

Risk	Likelihood	Impact	Mitigation
--link backup scripts break silently	Certain	Backups stop unnoticed	Fix §8(a) before the window; verify a full cycle
Data invisible due to uid mismatch	High	Apps fail to start or write	keep-id per §4.2 + chown -R tanshu; test each app's write path
Compiling on monoco OOMs production	High if done on-box	Outage during build	Build off-box (§3.2)
Containers do not return after reboot	High if linger skipped	Long unnoticed outage	enable-linger + explicit reboot test
pesto is experimental / breaks	Medium	Wrong client IPs, or Caddy unreachable	Snapshot first; know how to fall back to rootlessport
Switching forwarder with containers running	Medium	Corrupt port-forward state	Enable pesto before first start (§6)
Low-port sysctl not set	Medium	Caddy cannot bind 80/443	§6; verify after reboot
Let's Encrypt rate limit	Low	Certs unavailable for a week	Preserve /var/lib/caddy/data; verify certs load, don't re-issue
External deploy pipelines recreate Docker containers	High	Duplicate apps, port conflicts	Audit and update CI (§8(d))
Self-built stack drifts from security fixes	Certain over time	Unpatched engine on 20 sites	Rebuild pipeline + calendar reminder; or reconsider §2
Two runtimes writing the same data	Medium	Database corruption	Hard-stop Docker before starting Podman

---

12. Open questions

1. Are we upgrading to Ubuntu 26.04 as part of this? §3.4 argues yes, and that it should happen before the Podman migration. This decision changes the rest of the plan: on 26.04 there are no source builds, and §3 in its entirety becomes optional/deferred.
2. Where do mozimo, brewman, menu-admin, hops, greatbear, sms-wa-bridge deploy from? If those pipelines run docker against monoco they must change in lockstep. Should they move into this repo?
3. Is a temporary client-IP regression acceptable? Recommended: yes — migrate on stock Podman, accept gateway IPs at Caddy, and treat Podman 6 + pesto as a separate later project. The Quadlet units do not change, so it is a drop-in upgrade.
4. postgres's 0.0.0.0:5432 exposure — fold into this work or handle separately?

Resolved during the audit:

• gitea's dual uid — the container sets USER_UID=1003/USER_GID=1003, and git is uid 1003 on the host (/home/git/.ssh is 1003:1003). The real data lives in /var/lib/gitea/gitea/ owned git:git. Use UserNS=keep-id:uid=1003,gid=1003.
• Build host / packaging — moot if §3.4 is taken; revisit only if the pesto project proceeds.