tanshu/playbooks
1
0 Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add the Podman migration planning docs

Browse Source 
Three documents written before the container migration, all referenced from the
service readmes, so they belong in the tree:

- podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's
  architecture, privileges and namespaces; the four things that actually break
  (anonymous volumes, per-container uids, tooling that assumes Docker, image
  naming); and the per-container uid matrix that the UserNS=keep-id lines in the
  Quadlet units come from.
- podman-migration.md -- the plan: target architecture, the ownership/chown
  pass, the Quadlet unit shape, and the low-port and lingering prerequisites.
- backup-plan.md -- per-service irreplaceable state, the consistency problems
  (live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1
  architecture.

caddy, postgres, tandoor and bitwarden each link into these from their readme,
so leaving them untracked left those links dead.
This commit is contained in:
tanshu committed 2026-10-16 08:50:53 +05:30
1 parent f431d94f06
commit 8136c2ddaf
3 files changed
+1466
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/podman-migration.md
+679
View File
@@ -0,0 +1,679 @@
# Migrating `monoco` from Docker to rootless Podman 6 + Quadlet
Status: **plan, not yet executed**
Revised for: rootless Podman 6.x, data in `~/podman-data/`, per-app networks, systemd Quadlet units in `~/.config/containers/systemd/`
Audited: live `monoco` — Ubuntu 24.04.5 **aarch64**, 2 vCPU Neoverse-N1, 7.6 GB RAM, Docker 29.8.1, 26 containers
> This replaces an earlier rootful draft. The rootful sections are gone; §2 explains
> what the rootless choice costs, so the tradeoff is on the record rather than a surprise later.
---
## 1. Decisions locked in
| Decision | Value |
|---|---|
| Runtime | Rootless Podman **6.x**, built from source |
| Units | Quadlet `.container` / `.network` files in `~/.config/containers/systemd/` |
| Data | `~/podman-data/<service>/` (was `/var/lib/<service>/`) |
| Networks | One per logical app, as today — **not** consolidated |
| Client IPs | Preserved via **pesto** (`rootless_port_forwarder = "pasta"`) |
| Caddy | Rootless, multi-homed onto every app network |
---
## 2. What this costs — read once, then proceed
You asked whether Podman 6 can be built from source. **Yes.** But three things should
be on the record before we start, because they are not obvious from the request:
**1. Building 6.x solves one problem and creates a permanent maintenance obligation.**
The *only* thing the 6.x build buys over stock 4.9 is real client IPs at Caddy (via
pesto). Everything else in this plan works on stock Podman. In exchange you own
five source-built projects across three toolchains, with no distro security updates,
on a host serving ~20 sites. Pesto itself is upstream-described as *not default*, with
*no graceful upgrade path*. **§3.4 shows how upgrading to Ubuntu 26.04 removes the
source builds entirely — read that before committing to §3.**
**2. Building 6.x does not reduce the rootless work at all.** The expensive part of
rootless is re-owning ~19 GB of data into a user namespace (§4). That cost is
identical on 4.9, 5.x, or 6.x. If the goal is "off Docker soon", the version is
orthogonal.
**3. monoco is a 2 vCPU / 7.6 GB ARM64 box with ~2.5 GB RAM free.** Do not compile
on it (§3.2). Rust builds of netavark will thrash a machine that is already running
26 containers.
None of this argues against doing it — rootless is a legitimate goal and pesto is a
real fix. It argues for building elsewhere and phasing carefully.
---
## 3. Building Podman 6.x + Netavark 2 + pesto from source
### 3.1 Component and version matrix
Podman 6.0.0 (released 2026-06-24, latest patch 6.0.1) is not packaged for Ubuntu
24.04 by anyone. I checked `devel:kubic:libcontainers:unstable` (has netavark 1.12.2,
aardvark-dns 1.13.1, conmon, fuse-overlayfs — **no podman**), `isv:containers`, and
`devel:kubic:libcontainers:testing`. Debian unstable is still on podman 5.8.1 with a
bug open requesting 6.0.1. So source is genuinely the only path.
| Component | Version needed | Ubuntu 24.04 has | Gap |
|---|---|---|---|
| **podman** | 6.0.x | 4.9.3 | **build** |
| — Go toolchain | **1.25.6** (`go.mod`) | go 1.22 | **bootstrap Go** |
| **netavark** | **2.x** (latest 2.1.0) — hard requirement | 1.4.0 | **build** |
| **aardvark-dns** | **2.x** | 1.4.0 | **build** |
| — Rust toolchain | **1.88** (`rust-version`) | rustc 1.75 | **bootstrap rustup** |
| **passt** (ships `pesto`) | **≥ 0^20260507.g1afd4ed** | 0^20240220 | **build** |
| **container-libs** common | v0.68.0 | — | **build** |
| buildah / skopeo | 1.44.0 / 1.23 | optional | only if you build/push images on monoco |
| libgpgme-dev, libbtrfs-dev, libseccomp-dev, libassuan-dev | any | ✅ present | none |
| build-essential, pkg-config | any | ❌ gcc not installed | `apt install` |
Podman 6 breaking changes that matter here, all of which monoco satisfies:
- **iptables support removed** → nftables required. Host has `nft` 1.0.9 ✅
- **CNI removed** → netavark. Fine.
- **slirp4netns removed** → pasta required.
- **cgroups v1 removed** → `stat -fc %T /sys/fs/cgroup/` = `cgroup2fs` ✅
- **BoltDB removed** → SQLite.
- Netavark and Aardvark **must be 2.0.0** — 1.x will not work.
### 3.2 Build off-box
monoco has 2 vCPU, 7.6 GB RAM, ~2.5 GB available, and 26 production containers.
Rust compilation of netavark + aardvark-dns is the worst case for that profile.
**Build on a separate aarch64 machine or in CI, then ship artifacts.** All three
projects are portable:
- **Podman / container-libs**: Go — `CGO_ENABLED=0` cross-compiles trivially; for
`GOARCH=arm64` from any host.
- **netavark / aardvark-dns**: Rust — `cargo build --release --target aarch64-unknown-linux-gnu`.
- **passt/pesto**: plain C `make`.
Preferred output: four `.deb` packages (podman, netavark, aardvark-dns, passt) so
`apt` tracks them and you can rebuild on security updates. A tarball + install script
is acceptable but you lose upgrade tracking.
Build reference:
- Podman: `go.podman.io/podman/v6` (the import path moved to a CNCF org in 6.0)
- netavark: `make` in `containers/netavark`, installs `netavark` + config
- aardvark-dns: `containers/aardvark-dns`
- passt: `passt.top`, installs `passt`, `pasta`, and **`pesto`**
### 3.3 Install order on monoco
```
1. apt install build-essential pkg-config libgpgme-dev libbtrfs-dev libseccomp-dev libassuan-dev
2. apt install passt # baseline; then replace with your built passt+pesto
3. install netavark 2.x + aardvark-dns 2.x (to /usr/libexec/podman/)
4. install podman 6.x (to /usr/bin/podman)
5. verify: podman info --debug | grep -E 'networkBackend|version'
```
Keep Ubuntu's `podman` 4.9 package **uninstalled** — do not let apt install it as a
dependency and shadow your build. Pin with `apt-mark hold` if something pulls it in.
### 3.4 What about upgrading to Ubuntu 26.04 LTS first?
Short answer: **26.04 is a large improvement and worth doing, but it does not give you
pesto — so it does not remove the source builds if client IPs are the goal.**
26.04 "Resolute" ships (vs 24.04 today):
| Package | 24.04 (now) | 26.04 LTS |
|---|---|---|
| podman | 4.9.3 | **5.7.0** |
| passt | 0.0~git20240220 | **0.0~git20260120** |
| netavark | 1.4.0 | **1.16.1** |
| aardvark-dns | 1.4.0 | **1.16.0** |
| crun | — | 1.21 |
| buildah | — | 1.42.1 |
Two independent blockers, either of which alone is fatal:
1. **`rootless_port_forwarder` is Podman 6.0+.** The implementing PR
(`containers/podman#28478`) merged **2026-05-21** into `main` and shipped in
**6.0.0 on 2026-06-24**. 26.04 carries 5.7.0, which predates it. The Podman 6
announcement lists pesto as a 6.0 feature, and the option is documented in
container-libs `main` as `rootlessport` (default) | `pasta`.
2. **`pesto` needs passt ≥ `0^20260507`.** 26.04 ships `0^20260120` — about three and
a half months too old.
Also note Podman 6 **hard-requires netavark/aardvark-dns 2.0.0**, and 26.04 has 1.16 —
so moving to 6.x means building those too, on any base.
**The useful reframe:** pesto is a *separable, optional* improvement, not a migration
prerequisite. Losing real client IPs degrades Caddy access logs, `X-Forwarded-For`
downstream, Nextcloud/Vaultwarden IP-based brute-force protection, and Vince's
server-side analytics — but it does not break the reverse proxy, and nothing goes down.
That means the migration should not wait on it.
**Recommended sequencing** (revising §6 of the earlier draft):
| # | Step | Notes |
|---|---|---|
| 1 | Full backup + **verified restores** | see `docs/backup-plan.md` |
| 2 | **OS upgrade 24.04 → 26.04**, Docker still running | supported LTS→LTS path; needs its own window |
| 3 | Migrate Docker → Podman on **stock 5.7**, `~/.config/containers/systemd/` | **no source builds at all** |
| 4 | *Later, separately:* decide whether client IPs justify Podman 6 + pesto | orthogonal; Quadlet units unchanged |
Doing the OS upgrade **first** matters: migrating to a self-built Podman 6 on 24.04 and
*then* upgrading the OS leaves hand-built binaries fighting the distro packages at
exactly the moment you least want that.
The tradeoff, stated plainly (see §2 for the rootless case):
| | Source builds | Data re-owning | Real client IPs |
|---|---|---|---|
| Rootless, 26.04, stock 5.7 | none | required | ❌ |
| Rootless, 26.04, built 6.0 + pesto | 4 projects | required | ✅ (experimental) |
| Rootful, 26.04, stock 5.7 | none | **not required** | ✅ |
---
## 4. The uid mapping problem — the biggest piece of actual work
This is where rootless diverges hardest from Docker, and it is worth doing carefully.
### 4.1 The trap: `tanshu` is not uid 1000
```
id tanshu → uid=1001(tanshu) gid=1002(tanshu)
```
Most container data on this host is owned by **uid 1000**, which is the **`ubuntu`**
user, not you. So "the data is already owned by my user" is false everywhere. Anything
you don't explicitly re-own will be invisible inside a rootless container.
Measured ownership of the actual bind-mount sources:
| Container | Runs as (in-container) | Data source | Current host owner |
|---|---|---|---|
| `mozimo`, `mozimo-staging` | 1000 (`node`) | `/var/lib/mozimo/pb_data` | 1000:1000 |
| `hiwikijs` | 1000 (`node`) | *(anon volume)* | 1000:1000 |
| `menu-app.*` (×3) | 1000:1000 | `/var/lib/menu.*` | 1000:1000 |
| `vince` | 1000 | `/var/lib/mozimo/vince_data` | 1000:1000 |
| `yamtrack` | 1000 | `/var/lib/yamtrack/data` | 1000:1000 |
| `trillium` | 1000 (`node`) | `/var/lib/trillium` | 1000:1000 |
| `nextcloud` | 33 (`www-data`) | `/var/lib/nextcloud` | 33:33 |
| `postgres` | **70** | `/var/lib/postgresql/data/pgdata` | 70:0 |
| `valkey` | **999** | *(anon volume)* | 999:1000 |
| `gitea` | 1001 / 1003 (`git`) | `/var/lib/gitea`, `/home/git/.ssh` | 1001:1002, 1003:1003 |
| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | 0 | various | 0:0 |
> The `postgres` entry is the important one to sanity-check: `docker exec` reports uid 0
> (that's the image's configured user) but PGDATA is owned by **70**, which is the uid the
> server actually runs as. **Derive the app uid from what the app writes, not from
> `docker exec id`** — the entrypoint in `postgres`, `nextcloud`, `gitea`, `tandoor`
> and `trillium` drops privileges after starting.
### 4.2 The rule
Only **one** container uid:gid can be mapped to your host user. Use it for whichever
uid the app actually writes as:
```
UserNS=keep-id:uid=<app-uid>,gid=<app-gid>
```
| Service | Directive |
|---|---|
| `mozimo`, `mozimo-staging`, `hiwikijs`, `menu-app.*`, `vince`, `yamtrack`, `trillium` | `UserNS=keep-id:uid=1000,gid=1000` |
| `nextcloud` | `UserNS=keep-id:uid=33,gid=33` |
| `postgres` | `UserNS=keep-id:uid=70,gid=70` |
| `valkey` | `UserNS=keep-id:uid=999,gid=999` |
| `gitea` | `UserNS=keep-id:uid=1001,gid=1002` ⚠︎ verify |
| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | `UserNS=keep-id:uid=0,gid=0` |
**Then chown every data tree to `tanshu:tanshu` (1001:1002).** That is the whole
trick: the app's uid inside the container *is* your uid on the host, so files stay
readable and writable by you, and `tar`/`pg_dump` backups keep working without
`podman unshare` or `sudo`.
Do **not** use the default rootless mapping (container 0 → tanshu) for the non-root
apps — files would come out owned by subuid `165536+`, invisible to your backup
scripts. `keep-id` is what keeps the host side clean.
⚠︎ `gitea` writes as two different uids (1001:1002 for `/data`, 1003:1003 for
`/data/git/.ssh`), and `keep-id` maps only one. Resolve this before cutover: either
relax the `.ssh` ownership, or mount `/home/git/.ssh` from a path owned by 1001.
### 4.3 The chown pass
Data moves from `/var/lib/X` to `~/podman-data/X`. `/home/tanshu` and `/var/lib` are
both on `/` (`/dev/root`), so **`mv` is an instant rename, not a copy** — verify with
`findmnt -T /var/lib/<svc>` that none of them is a separate mount first.
```
# after stopping Docker, per service:
mv /var/lib/<svc> ~/podman-data/<svc>
chown -R tanshu:tanshu ~/podman-data/<svc>
```
`chown -R` is metadata-only, but audiobookshelf is 15 GB of many small files, so
budget a few minutes for that one. Total data is ~19 GB:
```
audiobookshelf 15G | nextcloud 1.2G | postgres 995M | gitea 747M
sms-wa-bridge 397M | mozimo 147M | mozimo-staging 137M | trillium 106M
greatbear 101M | tandoor 98M | menu.* 3×70M | hops 19M | caddy 736K | affine 916K
```
Plus two anonymous volumes that must be exported to real directories (they are
invisible to your backup scripts today):
| Container | Volume | → |
|---|---|---|
| `hiwikijs` | `9c1e3fde…` (owned 1000:1000) | `~/podman-data/hiwikijs/content` |
| `valkey` | `6089e6fc…` (owned 999:1000) | `~/podman-data/valkey` |
```
docker run --rm -v 9c1e3fde…:/from -v ~/podman-data/hiwikijs:/to alpine \
sh -c 'cd /from && cp -a . /to/content'
```
---
## 5. Target architecture
### 5.1 Layout
```
~/.config/containers/systemd/
caddy.container
postgres.container
nextcloud.container
nextcloud.network
gitea.network
menu-hopsngrains.network
… one .network per logical app
~/podman-data/
caddy/{conf,data,config,site} ← caddyfile_path becomes this
postgres/ nextcloud/ gitea/ audiobookshelf/ …
caddy/conf/Caddyfile ← regenerated by the Ansible caddy roles
```
### 5.2 Network naming — a Quadlet quirk to internalise
A Quadlet file `$NAME.network` produces a Podman network actually called
**`systemd-$NAME`**. So `nextcloud.network` → network `systemd-nextcloud`.
That is cosmetic for DNS (containers resolve each other by container name and alias),
but it matters for:
- `podman network ls` output — do not be confused by the prefix.
- Any `podman network connect` you run by hand.
- The `.network` file is **only read if some `.container` references it** — networks
are not created standalone.
A minimal network file is enough; DNS is on by default:
```ini
# ~/.config/containers/systemd/nextcloud.network
[Network]
```
### 5.3 A service unit
```ini
# ~/.config/containers/systemd/nextcloud.container
[Unit]
Description=Nextcloud
After=network-online.target
[Container]
ContainerName=nextcloud
Image=docker.io/library/nextcloud:latest
Network=nextcloud.network
UserNS=keep-id:uid=33,gid=33
Volume=%h/podman-data/nextcloud:/var/www/html
Volume=/etc/localtime:/etc/localtime:ro
[Service]
Restart=always
TimeoutStartSec=300
[Install]
WantedBy=default.target
```
`%h` is the systemd specifier for your home directory and works in Quadlet.
### 5.4 Postgres and valkey: multi-homed with aliases
`NetworkAlias=` exists in Quadlet 6.0, but it applies the alias to **all** networks
the container joins. Your aliases are per-network (`db` on only 3 of 13), so use the
`Network=` alias suffix instead — Quadlet passes it straight through to podman:
```ini
# ~/.config/containers/systemd/postgres.container
[Container]
ContainerName=postgres
Image=docker.io/library/postgres:17-alpine
UserNS=keep-id:uid=70,gid=70
Network=nextcloud.network:alias=db
Network=tandoor.network:alias=db
Network=hiwikijs.network:alias=db
Network=affine.network
Network=bitwarden.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Volume=%h/podman-data/postgres:/var/lib/postgresql/data
```
This is the direct analogue of today's 13-network Postgres hub. `valkey` is the same
pattern with `Network=nextcloud.network:alias=redis` plus `affine`, `yamtrack`.
**Pin `postgres:17-alpine`.** The running container reports image `postgres:17-alpine`
but its image ID is the one now tagged `postgres:alpine`; the server is 17.5.
### 5.5 Caddy: multi-homed, exactly as today
```ini
# ~/.config/containers/systemd/caddy.container
[Unit]
After=network-online.target postgres.service
[Container]
ContainerName=caddy
Image=docker.io/library/caddy:latest
UserNS=keep-id:uid=0,gid=0
PublishPort=80:80
PublishPort=443:443
PublishPort=443:443/udp
Volume=%h/podman-data/caddy/conf:/etc/caddy
Volume=%h/podman-data/caddy/site:/srv
Volume=%h/podman-data/caddy/data:/data
Volume=%h/podman-data/caddy/config:/config
Network=nextcloud.network
Network=gitea.network
Network=mozimo.network
Network=trillium.network
Network=tandoor.network
Network=hiwikijs.network
Network=bitwarden.network
Network=yamtrack.network
Network=affine.network
Network=audiobookshelf.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Network=menu-hopsngrains.network
Network=menu-greatbear.network
Network=menu-hngmohali.network
Network=greatbear.network
Network=hops.network
Network=sms-wa-bridge.network
[Service]
Restart=always
[Install]
WantedBy=default.target
```
The Caddyfile itself needs **no changes** — every upstream is still a container name.
Only `caddyfile_path` in the Ansible vars moves to
`/home/tanshu/podman-data/caddy/conf/Caddyfile`.
Note `Network=foo.network` in Caddy's unit makes Quadlet order the network units
first, so the "Caddy must be connected after the app exists" fragility of the current
Docker setup disappears entirely.
Drop these three — they are stale (Caddy holds interfaces for networks with no
service behind them): `iot_net` (the Caddyfile's `iot.tanshu.com → grafana:3000` is
dead), `n8n_net`, `kakawa_net`. Remove that Caddyfile block too.
---
## 6. Enabling pesto — do it before the first container starts
> **Deferred by default.** See §3.4: this whole section only applies if you decide the
> client-IP regression justifies building Podman 6 + passt. On Ubuntu 26.04 with stock
> Podman 5.7, this option does not exist, and the migration proceeds without it. If you
> *do* enable pesto, do it before any container starts — the switch has no graceful
> upgrade path.
Pesto is what preserves real client IPs. Per the Podman blog, switching the forwarder
on a running system has **no graceful upgrade path**, so enable it *before* bringing
anything up.
```ini
# ~/.config/containers/containers.conf (user-level, rootless)
[network]
rootless_port_forwarder = "pasta"
```
Traffic path becomes: `client → host:port → pasta → rootless netns → netavark bridge → container`.
Pasta forwards without rewriting the source address, so Caddy sees the real client.
Two prerequisites:
1. **`passt` with `pesto`** — your source build from §3. Verify: `which pesto`.
2. **Low ports.** Rootless cannot bind < 1024. Caddy needs 80 and 443:
```
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
# persist:
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
```
This is a root-level change — unavoidable for rootless on privileged ports.
**Good news on IPv6:** pasta's main behavioural difference from `rootlessport` is that
it does not transparently forward IPv6 host traffic to IPv4, so services need explicit
IPv6 binding. I checked: monoco has **no global IPv6 address and no default IPv6
route**, and `mozimo.in`, `git.tanshu.com`, `cloud.hopsngrains.com`, `vault.tanshu.com`
resolve to IPv4 only (no AAAA). **So this limitation does not affect you.**
**Enable lingering**, or nothing survives logout or reboot:
```
sudo loginctl enable-linger tanshu
```
Currently `Linger=no`. Without this, Quadlet user services stop when your last session
ends — which on a server means immediately.
---
## 7. Cutover sequence
Pre-flight (no downtime):
1. Capture ground truth: `docker inspect` for every container/network/volume → JSON.
2. Resolve the `gitea` dual-uid question (§4.2) and confirm each app's true uid.
3. Export the two anonymous volumes (§4.3).
4. **Test restores**, not just backups — at least the Postgres dumps.
5. Back up `/var/lib/caddy/data` explicitly. It holds the ACME account key and all
issued certificates. It is a bind mount so it carries over, but losing it means
re-issuing ~20 certs against Let's Encrypt rate limits.
6. Build and install the Podman 6 stack (§3). `podman login registry.tanshu.com`.
7. Pre-pull every image. For locally-built `sms-wa-bridge:latest` (no registry prefix)
avoid a re-download: `docker save sms-wa-bridge:latest | podman load`.
8. Write and review all Quadlet units; `systemctl --user daemon-reload` and check
`systemctl --user list-unit-files | grep -E 'caddy|nextcloud'`.
9. **Take a provider snapshot.** Cheapest, fastest rollback you will ever buy.
10. Rehearse on a clone of the host, including a reboot.
Window (expect 10–20 minutes; 26 containers plus a 15 GB `chown`):
```
# 1. Stop the edge first so no request is half-served
docker stop caddy
docker stop $(docker ps -q)
# 2. Freeze state; nothing may write to /var/lib during the move
sudo systemctl stop docker.socket docker.service containerd.service
sudo systemctl disable docker.service
# 3. Move + re-own data (instant rename, then a metadata walk)
mv /var/lib/<svc> ~/podman-data/<svc> && chown -R tanshu:tanshu ~/podman-data/<svc>
# 4. Bring up the data tier first
systemctl --user start postgres.service valkey.service
# 5. Verify DNS inside the namespace BEFORE starting Caddy
podman exec nextcloud getent hosts postgres
podman exec nextcloud getent hosts redis
# 6. App tier, then Caddy last
systemctl --user start <each app>
systemctl --user start caddy.service
# 7. Reload + smoke test
podman exec caddy caddy reload
```
**Do not `docker rm` anything.** Stopped containers are the fastest rollback and cost
only disk. Keep them a week.
---
## 8. What must change in this repo and on the host
**(a) Legacy `--link` in the backup scripts — a guaranteed break.**
`db-backup.sh`, `nextcloud-backup.sh`, `hiwikijs-`, `tandoor-`, `kakawa-`,
`bitwarden-backup.sh` and their `-restore.sh` counterparts all do
`docker run --rm --link postgres:db … pg_dump -h db`. `--link` is Docker-only and is
absent from `podman run`. Under rootless there is no default `bridge` network to fall
back on either.
Best fix — drop the helper container and dump in place (no networking, no `--user`
juggling, no shared network):
```bash
podman exec -e PGPASSWORD="..." postgres \
pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"
```
Also pin the base image: the scripts mix `postgres:alpine` and `postgres:17-alpine`.
Bonus: once data is owned by `tanshu` and reachable without `sudo`, the `sudo tar` and
`chown $(logname)` gymnastics in `nextcloud-backup.sh` can go away.
**(b) Root's crontab cannot reach rootless containers.**
```
*/5 * * * * docker exec -u www-data nextcloud php cron.php
```
Root cannot `podman exec` into another user's rootless containers. Move this to a
**systemd user timer** alongside the Quadlet units — which is the idiomatic approach
anyway. Same for `~/bin/empty-trash.sh`, which also targets a container named `cloud`
that no longer exists (it is `nextcloud`).
**(c) Ansible modules.** `docker_container` / `docker_image` / `docker_network` become
`containers.podman`'s `podman_container` / `podman_image` / `podman_network`, which
shell out to the `podman` CLI and need **no Python SDK** — the `pip: name=docker` task
disappears. Note `podman_network` lacks `docker_network`'s `connected:`/`appends:`
parameters, which the `*/roles/network/tasks/main.yaml` files rely on. Prefer declaring
the full `network:` list on each `podman_container` and deleting those network roles —
that is precisely what Quadlet does, so the two stay in sync.
Also update `caddyfile_path` everywhere (now `/home/tanshu/podman-data/caddy/conf/Caddyfile`).
**(d) `~/.zshrc:128`** and any other `docker ` invocations in `~/bin`, `/etc/cron*`,
and the **external deploy pipelines** for `mozimo`, `brewman`, `menu-admin`, `hops`,
`greatbear`, `sms-wa-bridge`. Those live outside this repo. If they are not updated,
the next deploy will recreate Docker containers next to your Podman ones.
**(e) `postgres` currently publishes `5432` on `0.0.0.0`** and UFW is inactive. Fix
while you are in here — it needs no published port at all, only `expose`/network access.
---
## 9. Rollback
1. **`docker start`** — old containers exist, stopped but not removed; bind mounts
untouched. Reverting is `systemctl start docker` + `docker start $(docker ps -aq)`.
Valid as long as you do not `docker rm`. ⚠︎ This only works if no app performed an
irreversible migration on first start under Podman (Postgres major upgrade,
Nextcloud schema migration, Trillium migration). Check that risk per service, and
prefer the snapshot over trusting this.
2. **Provider snapshot** — instant whole-host revert. Covers `/var/lib` damage and
certificate loss.
3. **Backups** — once §8(a) is fixed and restores are verified.
Note that the `chown` pass is itself hard to unwind precisely: reverting to Docker
requires the original ownership (0:0, 33:33, 70:0, 1000:1000). Record the table in
§4.1 before you start, and re-`chown` on rollback.
---
## 10. Verification checklist
**Linkage**
- [ ] `podman exec caddy getent hosts <name>` resolves for all 20 upstreams
- [ ] `getent hosts db` and `getent hosts redis` resolve from the apps that use them
- [ ] Every container is on the same set of networks it had under Docker
**Rootless plumbing**
- [ ] `loginctl show-user tanshu | grep Linger` → `Linger=yes`
- [ ] `which pesto` returns a path; `rootless_port_forwarder` is set **before** first start
- [ ] Caddy sees **real client IPs** — check `podman logs caddy` for a genuine remote IP
- [ ] `X-Forwarded-For` reaches Nextcloud/Gitea correctly
**HTTP / TLS**
- [ ] `curl -sSI https://<vhost>` 200/redirect for all 20 site blocks
- [ ] Certificates are the **existing** ones (compare serial against the pre-migration
snapshot) — nothing re-issued
- [ ] WebSocket / streaming paths (Gitea, Nextcloud, RustDesk)
**Applications**
- [ ] Each app can read **and write** — catches the uid mapping (§4.2)
- [ ] Postgres accepts connections from every client
- [ ] `postgres` is no longer published on `0.0.0.0:5432`
**Operations**
- [ ] Nextcloud cron fires via the new systemd user timer
- [ ] A full backup cycle of every `~/bin/*-backup.sh` completes and restores
- [ ] **Reboot the host**; all containers return with correct policies
- [ ] `systemctl --user status caddy` is clean; `podman exec caddy caddy reload` works
- [ ] Disk is acceptable after Docker images are pruned (32 GB reclaimable)
---
## 11. Risk register
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| `--link` backup scripts break silently | **Certain** | Backups stop unnoticed | Fix §8(a) before the window; verify a full cycle |
| Data invisible due to uid mismatch | **High** | Apps fail to start or write | `keep-id` per §4.2 + `chown -R tanshu`; test each app's write path |
| Compiling on monoco OOMs production | **High if done on-box** | Outage during build | Build off-box (§3.2) |
| Containers do not return after reboot | High if `linger` skipped | Long unnoticed outage | `enable-linger` + explicit reboot test |
| pesto is experimental / breaks | Medium | Wrong client IPs, or Caddy unreachable | Snapshot first; know how to fall back to `rootlessport` |
| Switching forwarder with containers running | Medium | Corrupt port-forward state | Enable pesto **before** first start (§6) |
| Low-port sysctl not set | Medium | Caddy cannot bind 80/443 | §6; verify after reboot |
| Let's Encrypt rate limit | Low | Certs unavailable for a week | Preserve `/var/lib/caddy/data`; verify certs load, don't re-issue |
| External deploy pipelines recreate Docker containers | **High** | Duplicate apps, port conflicts | Audit and update CI (§8(d)) |
| Self-built stack drifts from security fixes | **Certain over time** | Unpatched engine on 20 sites | Rebuild pipeline + calendar reminder; or reconsider §2 |
| Two runtimes writing the same data | Medium | Database corruption | Hard-stop Docker before starting Podman |
---
## 12. Open questions
1. **Are we upgrading to Ubuntu 26.04 as part of this?** §3.4 argues yes, and that it
should happen **before** the Podman migration. This decision changes the rest of the
plan: on 26.04 there are **no source builds**, and §3 in its entirety becomes
optional/deferred.
2. **Where do `mozimo`, `brewman`, `menu-admin`, `hops`, `greatbear`, `sms-wa-bridge`
deploy from?** If those pipelines run `docker` against monoco they must change in
lockstep. Should they move into this repo?
3. **Is a temporary client-IP regression acceptable?** Recommended: yes — migrate on
stock Podman, accept gateway IPs at Caddy, and treat Podman 6 + pesto as a separate
later project. The Quadlet units do not change, so it is a drop-in upgrade.
4. **`postgres`'s `0.0.0.0:5432` exposure** — fold into this work or handle separately?
Resolved during the audit:
- ~~**`gitea`'s dual uid**~~ — the container sets `USER_UID=1003`/`USER_GID=1003`, and
`git` is uid 1003 on the host (`/home/git/.ssh` is `1003:1003`). The real data lives
in `/var/lib/gitea/gitea/` owned `git:git`. Use **`UserNS=keep-id:uid=1003,gid=1003`**.
- ~~**Build host / packaging**~~ — moot if §3.4 is taken; revisit only if the pesto
project proceeds.