tanshu/playbooks
1
0 Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add the Podman migration planning docs

Browse Source 
Three documents written before the container migration, all referenced from the
service readmes, so they belong in the tree:

- podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's
  architecture, privileges and namespaces; the four things that actually break
  (anonymous volumes, per-container uids, tooling that assumes Docker, image
  naming); and the per-container uid matrix that the UserNS=keep-id lines in the
  Quadlet units come from.
- podman-migration.md -- the plan: target architecture, the ownership/chown
  pass, the Quadlet unit shape, and the low-port and lingering prerequisites.
- backup-plan.md -- per-service irreplaceable state, the consistency problems
  (live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1
  architecture.

caddy, postgres, tandoor and bitwarden each link into these from their readme,
so leaving them untracked left those links dead.
This commit is contained in:
tanshu committed 2026-10-16 08:50:53 +05:30
1 parent f431d94f06
commit 8136c2ddaf
3 files changed
+1466
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/backup-plan.md
+469
View File
@@ -0,0 +1,469 @@
# Full backup plan for `monoco`
Status: **plan, not yet executed**
Audited: live host, read-only. No changes made.
Host: EC2 `i-08edfd71ce301b8ba`, `ap-south-1c`, Ubuntu 24.04.5 aarch64
---
## 1. Objective
A complete, **verified** backup of monoco that:
1. Can restore the entire host (disaster recovery), and
2. Can restore any single service in isolation (the important case day-to-day), and
3. Is a hard prerequisite for the Docker → Podman migration, because that migration
changes file ownership on every data directory.
"Verified" is doing real work in that sentence. §10 defines what verification means,
and it is not optional.
---
## 2. What is actually on the box
| | |
|---|---|
| Provider | **EC2 confirmed** (IMDSv2), `i-08edfd71ce301b8ba`, `ap-south-1c` |
| Storage | **Single 80 GB EBS root volume** (`nvme0n1p1`), ext4, **no LVM**, no separate data volume |
| Free space | **30 GB** of 77 GB |
| Containers | 26, all plain `docker run`, all state on host bind mounts |
| Total app data | **~19 GB** |
| **Irreplaceable data** | **~2.4 GB** |
That last line is the most important number in this document. The gap between 19 GB
and 2.4 GB is almost entirely one thing:
```
/var/lib/audiobookshelf/audiobooks 15.0 GB 696 files. Re-downloadable media.
/var/lib/audiobookshelf/config 2.6 MB ← absdatabase.sqlite, irreplaceable
/var/lib/audiobookshelf/metadata 19 MB ← irreplaceable
```
So the real state of Audiobookshelf is **22 MB, not 15 GB**. Treating it as 15 GB is
what makes a "full backup" feel impossible; it isn't. Decide explicitly whether the
media is in scope (§8) — but do not let it drive the design.
### 2.1 What is irreplaceable, by service
| Service | Total | Back up | Skip (regenerable / from image) |
|---|---|---|---|
| `postgres` | 995 MB | 13 databases, ~890 MB raw | — |
| `gitea` | 747 MB | `git/` 302 MB + `gitea/` 445 MB **incl. gitea.db** | — |
| `nextcloud` | 1.2 GB | `data/` 324 MB, `config/` (config.php!), `custom_apps/` 5.8 MB | `apps/` 522 MB, `core/` 159 MB, `dist/` 83 MB, `3rdparty/` 55 MB, `lib/` 19 MB |
| `mozimo` | 147 MB | `pb_data/` (incl. `auxiliary.db` 18.6 MB + WAL) | `types.d.ts` |
| `mozimo-staging` | 137 MB | `pb_data/` | |
| `greatbear` | 101 MB | `pb_data/` | |
| `tandoor` | 98 MB | `mediafiles/` | `staticfiles/`, `externalfiles/` |
| `menu-app.*` ×3 | 210 MB | `/data` each | |
| `trillium` | 106 MB | `document.db` 3.8 MB + `backup/` 12 MB | `log/` 91 MB |
| `sms-wa-bridge` | 397 MB | `rules.json` (tiny) | `wwebjs_auth/` 307 MB — re-linkable via QR |
| `hops` | 19 MB | `pb_data/` | |
| `audiobookshelf` | 15 GB | `config/` + `metadata/` = **22 MB** | `audiobooks/` 15 GB |
| `bitwarden` | 4.2 MB | **Postgres `bitwarden` DB** | `data/db.sqlite3` — **stale**, see §3.3 |
| `yamtrack` | 2.3 MB | `data/db.sqlite3` | |
| `affine` | 916 KB | `config/` + `storage/` | |
| `caddy` | 736 KB | **all of it** — ACME account + certs | |
| `hiwikijs` | ? | **anonymous volume** — see §13.1 | `/var/lib/hiwikijs` is an empty decoy |
| `valkey` | ? | — | cache, rebuildable |
**~2.4 GB** once media and the WhatsApp session are excluded.
---
## 3. The consistency problem — read this before writing any backup
This is the part that silently produces useless backups. Every one of these is being
written to *right now* while the container runs.
### 3.1 SQLite in WAL mode
Live WAL sidecars are present on disk, which proves the databases are hot:
```
/var/lib/mozimo/pb_data/auxiliary.db 18,632,704 + -wal 1,116,552 + -shm
/var/lib/mozimo/pb_data/data.db 204,800 + -wal 78,312 + -shm
/var/lib/gitea/gitea/gitea.db 8,147,968 (modified 06:07 today)
/var/lib/audiobookshelf/config/absdatabase.sqlite 2,576,384
/var/lib/trillium/document.db 3,800,000 + -shm
/var/lib/yamtrack/data/db.sqlite3 + -wal + -shm
```
**A plain `tar` of a live SQLite database can produce a backup that will not open.**
The main file and the WAL are copied at different instants, so the result can be torn.
This affects `gitea-backup.sh` today.
Three acceptable methods, in order of preference:
1. **Use the application's own dump command** — always best, because the app
understands its own invariants:
- Gitea: `docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini`
→ one zip containing the DB, repos, and config.
- PocketBase (`mozimo`, `mozimo-staging`, `greatbear`, `hops`): use the built-in
backup, or stop the container for the copy.
2. **`sqlite3 <db> ".backup <out>"`** — safe on a live database, needs `sqlite3`
(present on the host: `~/.sqlite_history` exists).
3. **Stop the container, copy the directory, start it.** Simple, provable, costs
seconds of downtime per service. For a pre-migration backup this is the right
default, because the containers are being stopped anyway.
### 3.2 Postgres
13 databases, ~890 MB raw. A hot `tar` of `/var/lib/postgresql/data` is **not** a
valid backup — you need a logical dump or a proper physical base backup.
```
affine 11 MB | bitwarden 10 MB | brewman_acc 174 MB | brewman_exp 16 MB
brewman_hinchco 41 MB | brewman_hops 122 MB | brewman_mhl 132 MB | hiwikijs 16 MB
medusa-criollo 21 MB | nextcloud 319 MB | postgres 7.5 MB | tandoor 16 MB
zitadel 9.3 MB
```
Note two orphans with no running container — decide whether they are in scope:
**`medusa-criollo`** (21 MB, no container references it) and **`zitadel`** (9.3 MB,
container deleted). Everything else maps to a live service.
Method: `pg_dump -Fc` per database (parallel-safe, individually restorable) **plus**
`pg_dumpall --roles-only` for roles/globals, which per-database dumps do not capture.
### 3.3 The bitwarden trap
`bitwarden` has `DATABASE_URL=postgresql://…@postgres:5432/bitwarden`, and a 10 MB
`bitwarden` database exists in Postgres. **Postgres is authoritative.**
But `/var/lib/bitwarden/data/db.sqlite3` still exists with `-wal`/`-shm` sidecars, and
`~/bin/bitwarden-backup.sh` dumps the Postgres DB while `bitwarden-upload.sh` uploads a
`~/Data/warden/production.sqlite3` that **does not exist** (and `~/bin/gdrive` is not
installed).
So the current bitwarden backup story is three-quarters broken, and there is a stale
SQLite file that would look like a plausible restore source and silently lose months of
data. **Back up the Postgres database. Explicitly mark or delete the stale SQLite.**
### 3.4 Nextcloud
Files can be copied live; the **database cannot**. Dump the DB and the files together
so they correspond. Nextcloud's own guidance is maintenance mode during the backup:
`occ maintenance:mode --on`, dump, copy, `--off`. `config/config.php` must be included —
without it the instance is not restorable.
---
## 4. Backup architecture
Standard 3-2-1, using what this host actually has:
| Layer | What | Why | Where |
|---|---|---|---|
| **1. EBS snapshot** | Whole 80 GB volume | Instant whole-system restore; the only thing that captures the OS, Docker images, and 15 GB of media in one shot | AWS, S3-backed, inherently offsite |
| **2. Logical backup (restic)** | The ~2.4 GB of real state | Provably restorable, per-service, deduplicated, **encrypted** (matters — see §5) | S3 bucket in `ap-south-1` |
| **3. Local staging** | Transient only | Working space during the run | `/var/backups/` — **not** a backup |
Why both 1 and 2: an EBS snapshot is *crash-consistent*, not application-consistent —
it is equivalent to pulling the power cord. Restoring it means Postgres and every
SQLite database go through crash recovery. It is excellent for "the instance died" and
poor for "I need last night's database cleanly". The logical layer is the one that
gives you a *verified* restore.
**Region note:** an EBS snapshot in `ap-south-1` does not survive the loss of that
region or that AWS account. Enable **cross-region snapshot copy** (or at minimum
replicate the restic repo to a second bucket) before calling this disaster recovery.
---
## 5. Phase 0 — freeze the definitions (no downtime, minutes)
Do this first: it is tiny, has zero impact, and it is the part that is easiest to
forget and most annoying to reconstruct.
- **Every container's full config** — `docker inspect` for all containers, networks,
volumes → JSON. This is the source of truth for the migration.
- **Secrets and configuration.** The env vars hold live credentials: Shopify client
secret, Sentry auth token, PocketBase superuser passwords, JWT private keys,
`MIDDLEWARE_SECRET_KEY`, DB passwords, `AUTH_SECRET`, Vince admin password. Capture
them all — and note they are why the backup repository **must be encrypted at rest**.
- **Caddy** — the Caddyfile *and* `/var/lib/caddy/data` (ACME account key + ~20 issued
certificates). Losing the account key means re-issuing everything against Let's
Encrypt rate limits.
- **Deployment scripts** — everything in `~/bin/`, both crontabs, `~/.zshrc`.
- **Package and image inventory** — `dpkg --get-selections`, `docker images --digests`,
so the host can be rebuilt from scratch if it must be.
- **DNS records** for the ~20 hostnames pointing at `13.202.6.2`.
None of this needs the services stopped.
---
## 6. Phase 1 — EBS snapshot (no downtime)
```
aws ec2 create-snapshot \
--volume-id <vol-id from nvme0n1> \
--description "monoco pre-podman-migration $(date -Iseconds)"
```
Then:
- Tag it with the date, purpose, and the fact that it **predates the Podman migration**.
- Wait for `State: completed` (not just `pending`). A pending snapshot is not a backup.
- **Copy it to a second region** if it is meant to survive regional loss.
This is your fastest rollback if the migration goes badly wrong. `awscli` is not
installed — install it or take the snapshot from the console.
---
## 7. Phase 2 — the logical backup (the one that matters)
Container-native, so it works identically before and after the Podman migration —
write it this way and you do not have to rewrite it during the cutover.
### 7.1 Postgres (~890 MB → est. 200–300 MB compressed)
```
# roles and globals first — per-DB dumps do NOT include these
docker exec postgres pg_dumpall -U postgres --roles-only | gzip > roles.sql.gz
# one restorable file per database
for db in affine bitwarden brewman_acc brewman_exp brewman_hinchco \
brewman_hops brewman_mhl hiwikijs nextcloud tandoor; do
docker exec postgres pg_dump -U postgres -Fc "$db" > "$db.dump"
done
```
Deliberately excluded: `postgres` (system), and the `medusa-criollo` / `zitadel`
orphans — unless you decide otherwise (§3.2).
### 7.2 Gitea — use its own dump
```
docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini -f /data/dumps
```
One archive with `gitea.db`, the repositories, and config, produced consistently.
A `tar` of `/var/lib/gitea` will *probably* work and will occasionally produce a
corrupt `gitea.db` (§3.1).
### 7.3 SQLite services — stop, copy, start
Simplest provably-correct method, and these are fast to restart:
```
for c in mozimo mozimo-staging greatbear hops trillium yamtrack audiobookshelf; do
docker stop "$c"
done
# tar the relevant paths (§2.1 — skip audiobooks/, log/, staticfiles/)
for c in ...; do docker start "$c"; done
```
Alternatives if downtime is unacceptable: `sqlite3 "$db" ".backup"` per database, or
PocketBase's built-in backup for the four `pb_data` services.
### 7.4 Nextcloud
```
docker exec -u www-data nextcloud php occ maintenance:mode --on
docker exec postgres pg_dump -U postgres -Fc nextcloud > nextcloud.dump
tar -czf nextcloud-files.tar.gz \
-C /var/lib/nextcloud data config custom_apps
docker exec -u www-data nextcloud php occ maintenance:mode --off
```
### 7.5 Caddy, affine, sms-wa-bridge, menu apps, tandoor media
Plain file copies — none of these need a quiesce:
`/var/lib/caddy` (all 736 KB), `/var/lib/affine/{config,storage}`,
`/var/lib/sms-wa-bridge/rules.json`, each `/var/lib/menu.*`,
`/var/lib/tandoor/mediafiles`.
### 7.6 The anonymous volumes
`hiwikijs` and `valkey` are **not** under `/var/lib` — they live in
`/var/lib/docker/volumes/<hash>/_data` and are invisible to every existing script.
Export them to real directories before the migration (the migration plan requires it
anyway):
```
docker run --rm -v 9c1e3fde…:/from -v /var/backups/vol:/to alpine \
sh -c 'cp -a /from/. /to/hiwikijs/'
```
`hiwikijs` matters; `valkey` is a rebuildable cache.
### 7.7 Package it
Install **restic** (static aarch64 binary — nothing is installed today), initialise an
encrypted repo on S3, and back up the staging directory:
```
restic -r s3:s3.ap-south-1.amazonaws.com/monoco-backup init
restic -r s3:… backup /var/backups/monoco-$(date +%F) --tag pre-podman-migration
restic -r s3:… check
```
restic gives you encryption, deduplication (so daily runs are cheap), and snapshots you
can mount and restore individual files from. No credentials are configured on this host
today, so this needs an IAM role on the instance or an access key placed deliberately.
---
## 8. Phase 3 — bulk media (decision required)
The 15 GB of audiobooks and the 307 MB WhatsApp session.
- **If the audiobooks exist elsewhere** (purchased library, cloud, seedbox), exclude
them. They are already covered by the EBS snapshot for the "instance died" case, and
including them inflates every backup run for no recovery benefit.
- **If they exist only here**, they must be in scope — but do not put them in restic
daily. Handle them as a separate, infrequent, one-way copy.
This decision changes the backup from 2.4 GB to 18 GB. Make it explicitly.
---
## 9. Where the backup physically goes
There is **30 GB free** on a single volume. A backup written there is not a backup: it
shares the failure domain with the thing it protects, and a 19 GB archive would fill
the disk and take production down.
| Destination | Status |
|---|---|
| S3 `ap-south-1` (restic) | **Recommended.** Same region = fast and cheap; add cross-region for DR |
| EBS snapshot | Recommended, whole-system |
| Google Drive via `~/bin/gdrive` | **Broken** — binary not installed, `~/Data/warden` does not exist |
| Another server | No SSH hosts configured on the box; `registry.tanshu.com` (3.109.171.183) is a candidate if it has space |
| Local disk | Staging only |
---
## 10. Verification — acceptance criteria
A backup that has never been restored is a hypothesis. Before the migration proceeds:
- [ ] `restic check --read-data-subset=5%` passes
- [ ] **Every Postgres dump restores** into a scratch database, and a row count on a
known table matches production
- [ ] `roles.sql.gz` restores (roles/globals are the classic omission)
- [ ] The Gitea dump extracts and `gitea.db` opens with `sqlite3 … "PRAGMA integrity_check"`
- [ ] One PocketBase `data.db` restores and opens
- [ ] `trillium/document.db` and `yamtrack/db.sqlite3` open
- [ ] Nextcloud restores **with** `config/config.php`, in a scratch instance
- [ ] `/var/lib/caddy/data` contains the ACME account key and the certificate set
- [ ] The EBS snapshot reaches `completed`, and is **booted as a test instance** with
services confirmed running
- [ ] A manifest of checksums exists for the archive set
- [ ] **Original uid:gid ownership of every `/var/lib` path is recorded** — required to
undo the migration's `chown` (§14)
---
## 11. Ongoing backups — currently there are none
This is the biggest single finding: **nothing is scheduled.**
- The only cron entry on the host is `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
- `tanshu`, `ubuntu`, and `git` have no crontab.
- No systemd timer mentions backup (only `dpkg-db-backup.timer`).
- No restic, borg, rclone, duplicity, or aws CLI is installed.
- The five archives in `~/` are **manual one-offs** — Jun 2025, Feb 2026, Sep 2026,
Oct 2026 — for bitwarden, brewman_hops (SQL only), and nextcloud, plus obsolete
`mongodb.tar` and `overleaf.tar`.
So backups depend on someone remembering. Recommended replacement — a `restic` run on a
systemd timer, with all the container-native dump steps from §7 baked into a script,
retention of **7 daily / 4 weekly / 6 monthly**, and **failure alerting**. A backup that
fails silently every night for three months is indistinguishable from no backup, which
is exactly the state today.
---
## 12. Retention
| Layer | Retention |
|---|---|
| Logical (restic) | 7 daily, 4 weekly, 6 monthly |
| Weekly full | 3 months |
| Monthly full | 12 months |
| EBS snapshots | Keep the pre-migration one indefinitely; daily snapshots 7–14 days |
---
## 13. Bugs in the existing backup scripts
Worth fixing regardless of the migration, because these are actively misleading:
**13.1 `hiwikijs-backup.sh` backs up nothing.** It runs
`tar -C /var/lib hiwikijs`, but `/var/lib/hiwikijs` is an **empty directory**
(`drwxr-xr-x www-data`, dated Dec 2022). The real Wiki.js content is in the anonymous
volume `9c1e3fde…`. The script exits successfully and produces a valid archive of
nothing. Anyone trusting it has no Wiki.js backup at all.
**13.2 `gitea-backup.sh` hot-copies a live SQLite database** (§3.1). It will usually
work, which is the worst property a backup can have. Replace with `gitea dump`.
**13.3 `bitwarden-backup.sh` + `bitwarden-upload.sh` are inconsistent with each other
and with reality** (§3.3) — one dumps Postgres, the other uploads a SQLite file that
does not exist, and neither runs on a schedule.
**13.4 `iot-backup.sh` and `kakawa-backup.sh` target services that no longer exist**
(the `iot_net`/`kakawa_net` Docker networks are still attached to Caddy, but the
containers are gone).
**13.5 All DB scripts use `--link postgres:db`**, which does not exist in Podman and
will break at cutover. Under rootless Podman the `--user $(id -u)` and `sudo chown`
gymnastics are also unnecessary once the data is owned by you.
**13.6 No coverage at all** for: `mozimo`, `mozimo-staging`, `greatbear`, `hops`,
the three `menu-app.*` containers, `affine`, `vince`, `audiobookshelf`, `valkey`, or
`/var/lib/caddy` (the certificates).
---
## 14. Pre-migration specifics
Beyond a normal full backup, the migration needs these captured, because it changes
ownership and layout:
1. **Original `uid:gid` and mode of every bind-mount source** — the migration's
`chown -R tanshu:tanshu` is not trivially reversible without this table.
2. **The two anonymous volumes exported** (§7.6) — otherwise `hiwikijs` state is lost
permanently when Docker is removed.
3. **`docker inspect` JSON for all 26 containers** — the only complete record of the
runtime configuration.
4. **A restore rehearsal** onto a scratch host or the booted EBS snapshot, following the
migration plan's rollback path.
---
## 15. Risks
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Backups silently produce corrupt SQLite | **High** | Data loss discovered at restore time | §3.1 methods; integrity-check every DB in verification |
| `hiwikijs` backed up as an empty directory | **Certain today** | Permanent wiki loss | Export the anonymous volume (§7.6) |
| Postgres roles/globals omitted | Medium | Restore fails on permissions | `pg_dumpall --roles-only` (§7.1) |
| Stale bitwarden SQLite restored instead of Postgres | Medium | Months of vault data lost | §3.3 — mark or delete it |
| Let's Encrypt account key lost | Low | ~20 certs blocked by rate limits | Include all of `/var/lib/caddy` |
| Backup written to the same volume | Medium | Both lost together | S3/Snapshot only (§9) |
| 30 GB free filled by staging | Medium | Production outage during backup | Stage only the ~2.4 GB of real state; media excluded |
| Snapshot left in `pending` | Low | Restore from an incomplete snapshot | Wait for `completed`; boot-test it |
| Same-region snapshot as sole DR | Medium | Region/account loss unrecoverable | Cross-region copy |
| No alerting on backup failure | **High** | Months of silent failure | Monitoring on the timer (§11) |
---
## 16. Open questions
1. **Is the 15 GB of audiobook media in scope?** (§8) This decides 2.4 GB vs 18 GB.
2. **Where do backups go?** S3 in `ap-south-1` is the natural fit — is there an IAM role
on the instance, or should a dedicated bucket + credentials be set up?
3. **Are `medusa-criollo` and `zitadel` databases dead?** If so, exclude them and drop
them.
4. **The 1.3 MB `bitwarden-backup_202602261629.tar.gz`** — is the SQLite in it the
pre-Postgres state? If so it is a historical artifact worth keeping deliberately,
and the live `db.sqlite3` should be deleted so nobody restores the wrong thing.
5. **Is `registry.tanshu.com` a viable second backup target** (independent host, but
same region and presumably same provider account)?
6. **Should the ongoing backup be restic, or plain `pg_dump` + `tar` on a timer?**
restic is better but adds a component; the container-native dumps are needed either
way.
docs/podman-compat-audit.md
+318
View File
@@ -0,0 +1,318 @@
# Podman compatibility audit — monoco container fleet
Status: **planning only.** Read-only inspection; nothing on monoco was modified.
Scope: all 26 running containers, audited for rootless Podman + Quadlet.
Companion docs: [podman-migration.md](podman-migration.md), [backup-plan.md](backup-plan.md)
---
## 1. Verdict
**Nothing in this fleet blocks a rootless Podman migration.** There is no privileged
container, no capability add, no device, no host networking, no custom seccomp, no
Docker-socket mount, and no Compose file. Every image is **arm64**, so there is no
emulation.
The breakages are all in the *surrounding tooling* (backup scripts, cron, Ansible
modules) and in *ownership/volume handling* — not in the images. Your instinct that
your own `registry.tanshu.com` images will be fine is correct; I checked them
specifically and they are among the cleanest in the fleet.
Four things need real attention, and one of them is not obvious:
| # | Issue | Severity |
|---|---|---|
| 1 | `hiwikijs` and `valkey` use **anonymous Docker volumes** — copying `/var/lib/*` misses them entirely | **data loss** |
| 2 | Every data tree needs re-owning, and **each app's uid differs** (0 / 33 / 70 / 999 / 1000 / 1003) | blocks startup |
| 3 | `--link` in backup scripts, and root's crontab doing `docker exec` | breaks backups + cron |
| 4 | Locally-built `sms-wa-bridge` has no registry — short-name resolution fails | won't start |
---
## 2. What I verified as clean
Worth stating explicitly, because it is why this migration is tractable:
- **Architecture**: every image is `arm64` / `linux`. No binfmt/qemu needed.
- **Privileges**: `Privileged=false`, `CapAdd=[]`, `CapDrop=[]`, `Devices=[]` on all 26.
- **Namespaces**: no `--network host`, no `--pid host`, no `--ipc host`, no `--userns host`.
- **Security**: no custom `SecurityOpt`, no custom seccomp profile, no `--extra-hosts`.
- **Docker socket**: not mounted anywhere. No image talks to the Docker API.
- **Compose**: none. Every container is a plain `docker run` / Ansible `docker_container`.
- **shm**: all at the 64 MB default; nothing needs a larger `/dev/shm`.
- **Sysctls**: none set.
- **Resource limits**: `Memory=0`, `NanoCpus=0` on every container. (The Ansible role
for gitea *declares* `cpus: .10` / `memory: 512M`, but they are **not in effect** on
the running container — so no cgroup-delegation concerns for rootless.)
- **Custom DNS**: `Dns=[]` everywhere; aardvark-dns will replace Docker's `127.0.0.11`.
- **Hostnames**: all default (container ID). Nothing depends on a fixed hostname —
`sms-wa-bridge` explicitly cleans its own `Singleton*` locks on start because of this.
---
## 3. The four real problems
### 3.1 Anonymous volumes — the one that loses data
Eight images declare `VOLUME`:
| Image | VOLUME | Mounted? |
|---|---|---|
| `vaultwarden/server` | `/data` | ✅ bind-mounted |
| `gitea/gitea` | `/data` | ✅ bind-mounted |
| `registry.tanshu.com/{greatbear,hops}` | `/app/pb_data` | ✅ bind-mounted |
| `nextcloud` | `/var/www/html` | ✅ bind-mounted |
| `postgres:17-alpine` | `/var/lib/postgresql/data` | ✅ bind-mounted |
| **`ghcr.io/requarks/wiki`** | `/wiki/data/content` | ❌ **anonymous volume** |
| **`valkey/valkey:alpine`** | `/data` | ❌ **anonymous volume** |
The two anonymous ones live in `/var/lib/docker/volumes/<hash>/_data`, **not** under
`/var/lib/<service>`. A copy of the data directories will silently omit them.
`hiwikijs` is the serious one: `/var/lib/hiwikijs` exists but is an **empty directory**
from Dec 2022, and `~/bin/hiwikijs-backup.sh` archives exactly that empty directory and
exits successfully. So Wiki.js currently has **no backup at all** — this predates the
migration and should be fixed regardless of it.
Fix: export both to real directories (e.g. `~/podman-data/hiwikijs/content`,
`~/podman-data/valkey`) and declare them as Quadlet `Volume=` bind mounts. `valkey` is a
rebuildable cache; `hiwikijs` is not.
### 3.2 Ownership — one uid per container, and they all differ
Under rootless Podman the user namespace maps only **one** uid:gid to your host user.
The rule is `UserNS=keep-id:uid=<app-uid>,gid=<app-gid>`, where `<app-uid>` is the uid
the application actually writes as — **not** the image's `USER` and **not** `docker exec id`.
Measured (see §4 for the full matrix):
| App uid | Containers |
|---|---|
| 0 | caddy, brewman ×5, greatbear, hops, affine, audiobookshelf, bitwarden, tandoor, sms-wa-bridge, vince |
| 33 | nextcloud |
| 70 | postgres |
| 999 | valkey |
| 1000 | mozimo, mozimo-staging, hiwikijs, menu-app ×3, yamtrack, trillium |
| 1003 | gitea |
Trap: `docker exec <c> id -u` reports the **image's** configured user, not the running
process. It says `0` for `postgres` (which actually writes as **70**), for `nextcloud`
(**33**), and for `gitea` (**1003**) — because those entrypoints drop privileges after
starting. Derive the uid from **the ownership of what the app writes**, which is what
the table above does.
Also note `tanshu` is **uid 1001**, not 1000 — most of the uid-1000 data is currently
owned by the `ubuntu` user. "Chown it to me" is therefore not a no-op anywhere.
The reassuring part: for the entrypoints that start as root and drop to an app user
(`postgres`, `nextcloud`, `gitea`, `trillium`, `yamtrack`), the root phase maps to a
subuid and its internal `chown` still lands on your host user. So a single
`keep-id:uid=<app-uid>` plus `chown -R tanshu:tanshu` should work — **but verify per
service**, because a file written during the root phase and never chowned will be
subuid-owned and unreadable by the dropped-privilege process.
### 3.3 Tooling that assumes Docker
- **`--link postgres:db`** in `db-backup.sh`, `bitwarden-`, `hiwikijs-`, `kakawa-`,
`nextcloud-`, `tandoor-backup.sh` and their restores. No Podman equivalent.
- **Root's crontab**: `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
Root **cannot** `podman exec` into another user's rootless containers — this must
become a systemd **user** timer.
- **`~/bin/empty-trash.sh`** execs a container named `cloud`, which no longer exists
(it is `nextcloud`).
- **Ansible**: `docker_container` / `docker_image` / `docker_network` →
`containers.podman`. Note `podman_network` has **no** `connected:`/`appends:`
parameters, which every `*/roles/network/tasks/main.yaml` in this repo relies on.
Declare the full `network:` list on each `podman_container` instead.
- **Everything outside this repo.** You said ~90% is Ansible and the rest is
`docker run`. The deploy pipelines for `mozimo`, `brewman`, `menu-admin`, `hops`,
`greatbear` and `sms-wa-bridge` live elsewhere — if they still call `docker` against
the host, they will recreate Docker containers alongside your Podman ones.
### 3.4 Image naming
Rootless Podman resolves short names against `unqualified-search-registries`. In a
**Quadlet/systemd context there is no TTY to confirm a short-name prompt**, so
ambiguous names fail the unit rather than prompting.
- **`sms-wa-bridge`** was built locally with `docker build -t sms-wa-bridge` — no
registry. Podman will try `docker.io/library/sms-wa-bridge` and fail. Transfer it with
`docker save sms-wa-bridge | podman load` and reference **`localhost/sms-wa-bridge:latest`**.
- `nextcloud`, `caddy`, `postgres`, `valkey/valkey`, `gitea/gitea`, `vaultwarden/server`,
`vabene1111/recipes`, `triliumnext/notes` are all short or implicit-`docker.io` names.
**Fully qualify all of them** (`docker.io/library/nextcloud:latest` etc.) so resolution
never depends on host config.
- `registry.tanshu.com/*` and `ghcr.io/*` are already qualified. ✅
---
## 4. Per-container matrix
`App uid` is what the application writes as. `Anon vol` = uses an anonymous Docker volume.
| Container | Image | App uid | Anon vol | Verdict | Notes |
|---|---|---|---|---|---|
| `caddy` | `caddy:latest` | 0 | – | ⚠️ | Needs `ip_unprivileged_port_start≤80`; publishes 80, 443, **443/udp (HTTP/3)** — verify UDP forwarding |
| `postgres` | `postgres:17-alpine` | **70** | – | ⚠️ | PGDATA is `pgdata/` under the mount, owned uid 70. Drop the `0.0.0.0:5432` publish |
| `valkey` | `valkey/valkey:alpine` | **999** | **YES** | ⚠️ | Export the anonymous volume |
| `nextcloud` | `nextcloud:latest` | **33** | – | ✅ | `VOLUME /var/www/html` is bind-mounted; needs `POSTGRES_HOST=db` / `REDIS_HOST=redis` aliases |
| `gitea` | `gitea/gitea:latest` | **1003** | – | ⚠️ | `/var/lib/gitea/ssh` holds **SSH host keys** — copy them or clients see host-key warnings |
| `mozimo` | `registry.tanshu.com/mozimo:4a3863a` | 1000 | – | ✅ | Image `USER=node`; entrypoint starts PocketBase + SSR together |
| `mozimo-staging` | `registry.tanshu.com/mozimo:3929d3f` | 1000 | – | ✅ | |
| `greatbear` | `registry.tanshu.com/tanshu/greatbear` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
| `hops` | `registry.tanshu.com/hops:latest` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted |
| `menu-app.greatbear.in` | `registry.tanshu.com/menu-admin` | 1000 | – | ⚠️ | **Container names contain dots** — name the Quadlet units `menu-greatbear.container` etc. and set `ContainerName=` explicitly so the Caddyfile keeps resolving |
| `menu-app.hngmohali.com` | ″ | 1000 | – | ⚠️ | ″ |
| `menu-app.hopsngrains.com` | ″ | 1000 | – | ⚠️ | ″ |
| `hiwikijs` | `ghcr.io/requarks/wiki:latest` | 1000 | **YES** | ❌ | Content is in an anonymous volume; `/var/lib/hiwikijs` is an empty decoy |
| `affine` | `ghcr.io/toeverything/affine-graphql:stable` | 0 | – | ✅ | Needs postgres + valkey; indexer disabled (`AFFINE_INDEXER_ENABLED=false`), so no pgvector dependency |
| `audiobookshelf` | `ghcr.io/advplyr/audiobookshelf` | 0 | – | ✅ | `tini` entrypoint; 15 GB is media, state is 22 MB |
| `bitwarden` | `vaultwarden/server:latest` | 0 | – | ⚠️ | Authoritative store is **Postgres**; the `db.sqlite3` on disk is stale — do not restore it |
| `yamtrack` | `ghcr.io/fuzzygrim/yamtrack` | 1000 | – | ✅ | SQLite + WAL; needs a consistent copy |
| `trillium` | `triliumnext/notes:latest` | 1000 | – | ✅ | `document.db` + WAL; has its own `backup/` dir |
| `tandoor` | `vabene1111/recipes:latest` | 0 | – | ✅ | Needs `POSTGRES_HOST=db` alias |
| `sms-wa-bridge` | `sms-wa-bridge` (local) | 0 | – | ⚠️ | Chromium **is fine** (see below). Must be `localhost/sms-wa-bridge:latest` |
| `vince` | `ghcr.io/vinceanalytics/vince` | 0 | – | ⚠️ | **Distroless** — no shell, no `id`, so no exec-based checks. Container was run with `serve`, image `CMD` is null → set `Exec=serve` |
| `brewman-acc/exp/hinchco/hops/mhl` | `registry.tanshu.com/brewman:latest` | 0 | – | ✅ | 5 containers, identical config, one DB each |
### The Chromium question, answered
`sms-wa-bridge` is the classic rootless-Podman landmine: Chromium normally needs
`CLONE_NEWUSER` for its sandbox, which is blocked inside a container, and it can also
die on a 64 MB `/dev/shm`. I checked `server.js` and it already handles both:
```js
puppeteer: {
headless: true,
executablePath: process.env.CHROMIUM_PATH || undefined,
args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"]
}
```
`--no-sandbox` removes the userns dependency and `--disable-dev-shm-usage` removes the
shm dependency. **No action needed** — this container will behave identically.
---
## 5. Your migration thesis — validated, with three corrections
> *"if we copy the data directories, change ownership and deploy on new server, we should be good to go"*
**Broadly yes** — and with downtime free, this is a much safer plan than an in-place
cutover, because you can stop everything cleanly and copy quiesced data. Three
corrections:
**Correction 1 — "the data directories" is not all the state.** Adding to `/var/lib/*`:
- the two **anonymous volumes** (§3.1) — `hiwikijs` is genuinely at risk;
- **`/var/lib/gitea/ssh`** (SSH host keys) — easy to miss, and clients will complain;
- **`/var/lib/caddy/data`** — the ACME account key and ~20 certificates. Lose it and you
re-issue everything against Let's Encrypt rate limits;
- **the environment variables** — that is where the credentials live (Shopify client
secret, Sentry token, JWT private keys, PocketBase passwords, DB passwords);
- **`/etc/crontab` entries and `~/bin/`** — the operational glue.
**Correction 2 — "change ownership" is per-container, not global.** Chowning everything
to `tanshu` is right, but it only works if each container is also given the matching
`UserNS=keep-id:uid=<app-uid>`. Get the pairing wrong and the app starts and then fails
to write — the failure mode is a permissions error at runtime, not a clean startup
failure. The §4 table gives the pairing.
**Correction 3 — Postgres wants a clean stop, and a dump is still worth it.** Copying
`PGDATA` after a clean shutdown is valid *at the same major version* (17.5 here). But a
logical dump is cheap and gives you a version-independent fallback. Do both.
Since downtime is not a constraint, the copying step should be:
```
1. Stop all containers cleanly.
2. Copy: /var/lib/<svc>/, /var/lib/gitea/ssh, the two anonymous volumes,
/var/lib/caddy/data, ~/bin/, env vars, docker inspect JSON.
3. Verify checksums on the copy.
4. chown -R tanshu:tanshu on the destination, per §3.2.
5. Load images (docker save | podman load, or podman pull + podman login).
6. Write Quadlet units, set UserNS / Network / Volume.
7. systemctl --user daemon-reload && start the data tier, then apps, then Caddy.
```
Because you stop cleanly, the SQLite WAL problem from the backup plan largely
disappears — no live WAL, no torn copies. That is the single biggest simplification
this approach buys you.
---
## 6. Low ports and lingering
You already know both, confirming for completeness:
**Ports.** Rootless cannot bind < 1024. Caddy binds 80/443 **inside** its own network
namespace (where container-root has `CAP_NET_BIND_SERVICE`, which Podman grants by
default), but the **host-side publish** needs the sysctl:
```
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
```
Verify after reboot — this is a favourite thing to forget on a fresh instance.
**Lingering.** Without it, Quadlet **user** services stop when the login session ends,
which on a server is immediately, and they do not start at boot:
```
sudo loginctl enable-linger tanshu
```
Also confirm `/etc/subuid` and `/etc/subgid` have a range for the new user — monoco has
`tanshu:165536:65536`, and a fresh instance will assign a different base. Do not
hardcode `165536` anywhere.
---
## 7. Cutover notes specific to a new instance
**DNS and certificates are the thing to plan around.** All ~20 hostnames resolve to
`13.202.6.2`. On a new instance with a new IP:
- Caddy cannot obtain certificates until DNS points at the new host (HTTP-01), and the
existing certs in `/var/lib/caddy/data` are only valid to *serve* for those names once
traffic arrives.
- **Check whether `13.202.6.2` is an Elastic IP.** If it is, reassign it to the new
instance and DNS, certificates, and clients are all untouched — by far the cleanest
cutover. Worth confirming before designing around a DNS change.
**Logs do not migrate.** All 26 containers use the `json-file` driver; Podman uses
journald. If the existing logs matter, export them before decommissioning. Otherwise
`podman logs` / `journalctl --user -u <svc>` replaces `docker logs`.
**Image transfer.** `docker save <img> | podman load` avoids re-downloading (and avoids
registry auth for the local image). For `registry.tanshu.com`, run `podman login` on the
new host so `podman auto-update`/pulls work.
**The OS question is still open.** Per [podman-migration.md §3.4](podman-migration.md),
a **fresh 26.04 instance with stock Podman 5.7 needs no source builds at all**. Since you
are standing up a new instance anyway, this is now essentially free — and it makes the
"build Podman 6 for pesto" question entirely separable from the migration.
---
## 8. Per-service verification after cutover
For each container, in this order:
1. **Starts and stays up** — `systemctl --user status <svc>`
2. **Writes as the right uid** — create/update something through the app, then confirm
on the host that the new file is owned by `tanshu`. This is the check that catches a
wrong `keep-id`.
3. **Resolves its dependencies** — `podman exec <c> getent hosts postgres` (and `db` /
`redis` aliases where used)
4. **Healthcheck reports healthy** — `podman inspect --format '{{.State.Health.Status}}'`
5. Special cases:
- `caddy`: real client IPs in the access log (this is where the rootless `rootlessport`
limitation shows up — see §3.4 of the migration plan); `https://<vhost>` responds;
HTTP/3 (`curl --http3`) if you care
- `sms-wa-bridge`: WhatsApp re-links (or restores) and a test message sends
- `gitea`: `git clone` over **both** HTTPS and SSH (port 2222), no host-key warning
- `postgres`: every client app connects; nothing listening on host `5432`
- `nextcloud`: upload a file (exercises the uid-33 mapping and `REDIS_HOST`)
- `vince`: reachable on `stats.mozimo.in` (distroless, so no shell to diagnose with)
6. **Reboot the host** and confirm everything returns unattended
docs/podman-migration.md
+679
View File
@@ -0,0 +1,679 @@
# Migrating `monoco` from Docker to rootless Podman 6 + Quadlet
Status: **plan, not yet executed**
Revised for: rootless Podman 6.x, data in `~/podman-data/`, per-app networks, systemd Quadlet units in `~/.config/containers/systemd/`
Audited: live `monoco` — Ubuntu 24.04.5 **aarch64**, 2 vCPU Neoverse-N1, 7.6 GB RAM, Docker 29.8.1, 26 containers
> This replaces an earlier rootful draft. The rootful sections are gone; §2 explains
> what the rootless choice costs, so the tradeoff is on the record rather than a surprise later.
---
## 1. Decisions locked in
| Decision | Value |
|---|---|
| Runtime | Rootless Podman **6.x**, built from source |
| Units | Quadlet `.container` / `.network` files in `~/.config/containers/systemd/` |
| Data | `~/podman-data/<service>/` (was `/var/lib/<service>/`) |
| Networks | One per logical app, as today — **not** consolidated |
| Client IPs | Preserved via **pesto** (`rootless_port_forwarder = "pasta"`) |
| Caddy | Rootless, multi-homed onto every app network |
---
## 2. What this costs — read once, then proceed
You asked whether Podman 6 can be built from source. **Yes.** But three things should
be on the record before we start, because they are not obvious from the request:
**1. Building 6.x solves one problem and creates a permanent maintenance obligation.**
The *only* thing the 6.x build buys over stock 4.9 is real client IPs at Caddy (via
pesto). Everything else in this plan works on stock Podman. In exchange you own
five source-built projects across three toolchains, with no distro security updates,
on a host serving ~20 sites. Pesto itself is upstream-described as *not default*, with
*no graceful upgrade path*. **§3.4 shows how upgrading to Ubuntu 26.04 removes the
source builds entirely — read that before committing to §3.**
**2. Building 6.x does not reduce the rootless work at all.** The expensive part of
rootless is re-owning ~19 GB of data into a user namespace (§4). That cost is
identical on 4.9, 5.x, or 6.x. If the goal is "off Docker soon", the version is
orthogonal.
**3. monoco is a 2 vCPU / 7.6 GB ARM64 box with ~2.5 GB RAM free.** Do not compile
on it (§3.2). Rust builds of netavark will thrash a machine that is already running
26 containers.
None of this argues against doing it — rootless is a legitimate goal and pesto is a
real fix. It argues for building elsewhere and phasing carefully.
---
## 3. Building Podman 6.x + Netavark 2 + pesto from source
### 3.1 Component and version matrix
Podman 6.0.0 (released 2026-06-24, latest patch 6.0.1) is not packaged for Ubuntu
24.04 by anyone. I checked `devel:kubic:libcontainers:unstable` (has netavark 1.12.2,
aardvark-dns 1.13.1, conmon, fuse-overlayfs — **no podman**), `isv:containers`, and
`devel:kubic:libcontainers:testing`. Debian unstable is still on podman 5.8.1 with a
bug open requesting 6.0.1. So source is genuinely the only path.
| Component | Version needed | Ubuntu 24.04 has | Gap |
|---|---|---|---|
| **podman** | 6.0.x | 4.9.3 | **build** |
| — Go toolchain | **1.25.6** (`go.mod`) | go 1.22 | **bootstrap Go** |
| **netavark** | **2.x** (latest 2.1.0) — hard requirement | 1.4.0 | **build** |
| **aardvark-dns** | **2.x** | 1.4.0 | **build** |
| — Rust toolchain | **1.88** (`rust-version`) | rustc 1.75 | **bootstrap rustup** |
| **passt** (ships `pesto`) | **≥ 0^20260507.g1afd4ed** | 0^20240220 | **build** |
| **container-libs** common | v0.68.0 | — | **build** |
| buildah / skopeo | 1.44.0 / 1.23 | optional | only if you build/push images on monoco |
| libgpgme-dev, libbtrfs-dev, libseccomp-dev, libassuan-dev | any | ✅ present | none |
| build-essential, pkg-config | any | ❌ gcc not installed | `apt install` |
Podman 6 breaking changes that matter here, all of which monoco satisfies:
- **iptables support removed** → nftables required. Host has `nft` 1.0.9 ✅
- **CNI removed** → netavark. Fine.
- **slirp4netns removed** → pasta required.
- **cgroups v1 removed** → `stat -fc %T /sys/fs/cgroup/` = `cgroup2fs` ✅
- **BoltDB removed** → SQLite.
- Netavark and Aardvark **must be 2.0.0** — 1.x will not work.
### 3.2 Build off-box
monoco has 2 vCPU, 7.6 GB RAM, ~2.5 GB available, and 26 production containers.
Rust compilation of netavark + aardvark-dns is the worst case for that profile.
**Build on a separate aarch64 machine or in CI, then ship artifacts.** All three
projects are portable:
- **Podman / container-libs**: Go — `CGO_ENABLED=0` cross-compiles trivially; for
`GOARCH=arm64` from any host.
- **netavark / aardvark-dns**: Rust — `cargo build --release --target aarch64-unknown-linux-gnu`.
- **passt/pesto**: plain C `make`.
Preferred output: four `.deb` packages (podman, netavark, aardvark-dns, passt) so
`apt` tracks them and you can rebuild on security updates. A tarball + install script
is acceptable but you lose upgrade tracking.
Build reference:
- Podman: `go.podman.io/podman/v6` (the import path moved to a CNCF org in 6.0)
- netavark: `make` in `containers/netavark`, installs `netavark` + config
- aardvark-dns: `containers/aardvark-dns`
- passt: `passt.top`, installs `passt`, `pasta`, and **`pesto`**
### 3.3 Install order on monoco
```
1. apt install build-essential pkg-config libgpgme-dev libbtrfs-dev libseccomp-dev libassuan-dev
2. apt install passt # baseline; then replace with your built passt+pesto
3. install netavark 2.x + aardvark-dns 2.x (to /usr/libexec/podman/)
4. install podman 6.x (to /usr/bin/podman)
5. verify: podman info --debug | grep -E 'networkBackend|version'
```
Keep Ubuntu's `podman` 4.9 package **uninstalled** — do not let apt install it as a
dependency and shadow your build. Pin with `apt-mark hold` if something pulls it in.
### 3.4 What about upgrading to Ubuntu 26.04 LTS first?
Short answer: **26.04 is a large improvement and worth doing, but it does not give you
pesto — so it does not remove the source builds if client IPs are the goal.**
26.04 "Resolute" ships (vs 24.04 today):
| Package | 24.04 (now) | 26.04 LTS |
|---|---|---|
| podman | 4.9.3 | **5.7.0** |
| passt | 0.0~git20240220 | **0.0~git20260120** |
| netavark | 1.4.0 | **1.16.1** |
| aardvark-dns | 1.4.0 | **1.16.0** |
| crun | — | 1.21 |
| buildah | — | 1.42.1 |
Two independent blockers, either of which alone is fatal:
1. **`rootless_port_forwarder` is Podman 6.0+.** The implementing PR
(`containers/podman#28478`) merged **2026-05-21** into `main` and shipped in
**6.0.0 on 2026-06-24**. 26.04 carries 5.7.0, which predates it. The Podman 6
announcement lists pesto as a 6.0 feature, and the option is documented in
container-libs `main` as `rootlessport` (default) | `pasta`.
2. **`pesto` needs passt ≥ `0^20260507`.** 26.04 ships `0^20260120` — about three and
a half months too old.
Also note Podman 6 **hard-requires netavark/aardvark-dns 2.0.0**, and 26.04 has 1.16 —
so moving to 6.x means building those too, on any base.
**The useful reframe:** pesto is a *separable, optional* improvement, not a migration
prerequisite. Losing real client IPs degrades Caddy access logs, `X-Forwarded-For`
downstream, Nextcloud/Vaultwarden IP-based brute-force protection, and Vince's
server-side analytics — but it does not break the reverse proxy, and nothing goes down.
That means the migration should not wait on it.
**Recommended sequencing** (revising §6 of the earlier draft):
| # | Step | Notes |
|---|---|---|
| 1 | Full backup + **verified restores** | see `docs/backup-plan.md` |
| 2 | **OS upgrade 24.04 → 26.04**, Docker still running | supported LTS→LTS path; needs its own window |
| 3 | Migrate Docker → Podman on **stock 5.7**, `~/.config/containers/systemd/` | **no source builds at all** |
| 4 | *Later, separately:* decide whether client IPs justify Podman 6 + pesto | orthogonal; Quadlet units unchanged |
Doing the OS upgrade **first** matters: migrating to a self-built Podman 6 on 24.04 and
*then* upgrading the OS leaves hand-built binaries fighting the distro packages at
exactly the moment you least want that.
The tradeoff, stated plainly (see §2 for the rootless case):
| | Source builds | Data re-owning | Real client IPs |
|---|---|---|---|
| Rootless, 26.04, stock 5.7 | none | required | ❌ |
| Rootless, 26.04, built 6.0 + pesto | 4 projects | required | ✅ (experimental) |
| Rootful, 26.04, stock 5.7 | none | **not required** | ✅ |
---
## 4. The uid mapping problem — the biggest piece of actual work
This is where rootless diverges hardest from Docker, and it is worth doing carefully.
### 4.1 The trap: `tanshu` is not uid 1000
```
id tanshu → uid=1001(tanshu) gid=1002(tanshu)
```
Most container data on this host is owned by **uid 1000**, which is the **`ubuntu`**
user, not you. So "the data is already owned by my user" is false everywhere. Anything
you don't explicitly re-own will be invisible inside a rootless container.
Measured ownership of the actual bind-mount sources:
| Container | Runs as (in-container) | Data source | Current host owner |
|---|---|---|---|
| `mozimo`, `mozimo-staging` | 1000 (`node`) | `/var/lib/mozimo/pb_data` | 1000:1000 |
| `hiwikijs` | 1000 (`node`) | *(anon volume)* | 1000:1000 |
| `menu-app.*` (×3) | 1000:1000 | `/var/lib/menu.*` | 1000:1000 |
| `vince` | 1000 | `/var/lib/mozimo/vince_data` | 1000:1000 |
| `yamtrack` | 1000 | `/var/lib/yamtrack/data` | 1000:1000 |
| `trillium` | 1000 (`node`) | `/var/lib/trillium` | 1000:1000 |
| `nextcloud` | 33 (`www-data`) | `/var/lib/nextcloud` | 33:33 |
| `postgres` | **70** | `/var/lib/postgresql/data/pgdata` | 70:0 |
| `valkey` | **999** | *(anon volume)* | 999:1000 |
| `gitea` | 1001 / 1003 (`git`) | `/var/lib/gitea`, `/home/git/.ssh` | 1001:1002, 1003:1003 |
| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | 0 | various | 0:0 |
> The `postgres` entry is the important one to sanity-check: `docker exec` reports uid 0
> (that's the image's configured user) but PGDATA is owned by **70**, which is the uid the
> server actually runs as. **Derive the app uid from what the app writes, not from
> `docker exec id`** — the entrypoint in `postgres`, `nextcloud`, `gitea`, `tandoor`
> and `trillium` drops privileges after starting.
### 4.2 The rule
Only **one** container uid:gid can be mapped to your host user. Use it for whichever
uid the app actually writes as:
```
UserNS=keep-id:uid=<app-uid>,gid=<app-gid>
```
| Service | Directive |
|---|---|
| `mozimo`, `mozimo-staging`, `hiwikijs`, `menu-app.*`, `vince`, `yamtrack`, `trillium` | `UserNS=keep-id:uid=1000,gid=1000` |
| `nextcloud` | `UserNS=keep-id:uid=33,gid=33` |
| `postgres` | `UserNS=keep-id:uid=70,gid=70` |
| `valkey` | `UserNS=keep-id:uid=999,gid=999` |
| `gitea` | `UserNS=keep-id:uid=1001,gid=1002` ⚠︎ verify |
| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | `UserNS=keep-id:uid=0,gid=0` |
**Then chown every data tree to `tanshu:tanshu` (1001:1002).** That is the whole
trick: the app's uid inside the container *is* your uid on the host, so files stay
readable and writable by you, and `tar`/`pg_dump` backups keep working without
`podman unshare` or `sudo`.
Do **not** use the default rootless mapping (container 0 → tanshu) for the non-root
apps — files would come out owned by subuid `165536+`, invisible to your backup
scripts. `keep-id` is what keeps the host side clean.
⚠︎ `gitea` writes as two different uids (1001:1002 for `/data`, 1003:1003 for
`/data/git/.ssh`), and `keep-id` maps only one. Resolve this before cutover: either
relax the `.ssh` ownership, or mount `/home/git/.ssh` from a path owned by 1001.
### 4.3 The chown pass
Data moves from `/var/lib/X` to `~/podman-data/X`. `/home/tanshu` and `/var/lib` are
both on `/` (`/dev/root`), so **`mv` is an instant rename, not a copy** — verify with
`findmnt -T /var/lib/<svc>` that none of them is a separate mount first.
```
# after stopping Docker, per service:
mv /var/lib/<svc> ~/podman-data/<svc>
chown -R tanshu:tanshu ~/podman-data/<svc>
```
`chown -R` is metadata-only, but audiobookshelf is 15 GB of many small files, so
budget a few minutes for that one. Total data is ~19 GB:
```
audiobookshelf 15G | nextcloud 1.2G | postgres 995M | gitea 747M
sms-wa-bridge 397M | mozimo 147M | mozimo-staging 137M | trillium 106M
greatbear 101M | tandoor 98M | menu.* 3×70M | hops 19M | caddy 736K | affine 916K
```
Plus two anonymous volumes that must be exported to real directories (they are
invisible to your backup scripts today):
| Container | Volume | → |
|---|---|---|
| `hiwikijs` | `9c1e3fde…` (owned 1000:1000) | `~/podman-data/hiwikijs/content` |
| `valkey` | `6089e6fc…` (owned 999:1000) | `~/podman-data/valkey` |
```
docker run --rm -v 9c1e3fde…:/from -v ~/podman-data/hiwikijs:/to alpine \
sh -c 'cd /from && cp -a . /to/content'
```
---
## 5. Target architecture
### 5.1 Layout
```
~/.config/containers/systemd/
caddy.container
postgres.container
nextcloud.container
nextcloud.network
gitea.network
menu-hopsngrains.network
… one .network per logical app
~/podman-data/
caddy/{conf,data,config,site} ← caddyfile_path becomes this
postgres/ nextcloud/ gitea/ audiobookshelf/ …
caddy/conf/Caddyfile ← regenerated by the Ansible caddy roles
```
### 5.2 Network naming — a Quadlet quirk to internalise
A Quadlet file `$NAME.network` produces a Podman network actually called
**`systemd-$NAME`**. So `nextcloud.network` → network `systemd-nextcloud`.
That is cosmetic for DNS (containers resolve each other by container name and alias),
but it matters for:
- `podman network ls` output — do not be confused by the prefix.
- Any `podman network connect` you run by hand.
- The `.network` file is **only read if some `.container` references it** — networks
are not created standalone.
A minimal network file is enough; DNS is on by default:
```ini
# ~/.config/containers/systemd/nextcloud.network
[Network]
```
### 5.3 A service unit
```ini
# ~/.config/containers/systemd/nextcloud.container
[Unit]
Description=Nextcloud
After=network-online.target
[Container]
ContainerName=nextcloud
Image=docker.io/library/nextcloud:latest
Network=nextcloud.network
UserNS=keep-id:uid=33,gid=33
Volume=%h/podman-data/nextcloud:/var/www/html
Volume=/etc/localtime:/etc/localtime:ro
[Service]
Restart=always
TimeoutStartSec=300
[Install]
WantedBy=default.target
```
`%h` is the systemd specifier for your home directory and works in Quadlet.
### 5.4 Postgres and valkey: multi-homed with aliases
`NetworkAlias=` exists in Quadlet 6.0, but it applies the alias to **all** networks
the container joins. Your aliases are per-network (`db` on only 3 of 13), so use the
`Network=` alias suffix instead — Quadlet passes it straight through to podman:
```ini
# ~/.config/containers/systemd/postgres.container
[Container]
ContainerName=postgres
Image=docker.io/library/postgres:17-alpine
UserNS=keep-id:uid=70,gid=70
Network=nextcloud.network:alias=db
Network=tandoor.network:alias=db
Network=hiwikijs.network:alias=db
Network=affine.network
Network=bitwarden.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Volume=%h/podman-data/postgres:/var/lib/postgresql/data
```
This is the direct analogue of today's 13-network Postgres hub. `valkey` is the same
pattern with `Network=nextcloud.network:alias=redis` plus `affine`, `yamtrack`.
**Pin `postgres:17-alpine`.** The running container reports image `postgres:17-alpine`
but its image ID is the one now tagged `postgres:alpine`; the server is 17.5.
### 5.5 Caddy: multi-homed, exactly as today
```ini
# ~/.config/containers/systemd/caddy.container
[Unit]
After=network-online.target postgres.service
[Container]
ContainerName=caddy
Image=docker.io/library/caddy:latest
UserNS=keep-id:uid=0,gid=0
PublishPort=80:80
PublishPort=443:443
PublishPort=443:443/udp
Volume=%h/podman-data/caddy/conf:/etc/caddy
Volume=%h/podman-data/caddy/site:/srv
Volume=%h/podman-data/caddy/data:/data
Volume=%h/podman-data/caddy/config:/config
Network=nextcloud.network
Network=gitea.network
Network=mozimo.network
Network=trillium.network
Network=tandoor.network
Network=hiwikijs.network
Network=bitwarden.network
Network=yamtrack.network
Network=affine.network
Network=audiobookshelf.network
Network=brewman-acc.network
Network=brewman-exp.network
Network=brewman-hinchco.network
Network=brewman-hops.network
Network=brewman-mhl.network
Network=menu-hopsngrains.network
Network=menu-greatbear.network
Network=menu-hngmohali.network
Network=greatbear.network
Network=hops.network
Network=sms-wa-bridge.network
[Service]
Restart=always
[Install]
WantedBy=default.target
```
The Caddyfile itself needs **no changes** — every upstream is still a container name.
Only `caddyfile_path` in the Ansible vars moves to
`/home/tanshu/podman-data/caddy/conf/Caddyfile`.
Note `Network=foo.network` in Caddy's unit makes Quadlet order the network units
first, so the "Caddy must be connected after the app exists" fragility of the current
Docker setup disappears entirely.
Drop these three — they are stale (Caddy holds interfaces for networks with no
service behind them): `iot_net` (the Caddyfile's `iot.tanshu.com → grafana:3000` is
dead), `n8n_net`, `kakawa_net`. Remove that Caddyfile block too.
---
## 6. Enabling pesto — do it before the first container starts
> **Deferred by default.** See §3.4: this whole section only applies if you decide the
> client-IP regression justifies building Podman 6 + passt. On Ubuntu 26.04 with stock
> Podman 5.7, this option does not exist, and the migration proceeds without it. If you
> *do* enable pesto, do it before any container starts — the switch has no graceful
> upgrade path.
Pesto is what preserves real client IPs. Per the Podman blog, switching the forwarder
on a running system has **no graceful upgrade path**, so enable it *before* bringing
anything up.
```ini
# ~/.config/containers/containers.conf (user-level, rootless)
[network]
rootless_port_forwarder = "pasta"
```
Traffic path becomes: `client → host:port → pasta → rootless netns → netavark bridge → container`.
Pasta forwards without rewriting the source address, so Caddy sees the real client.
Two prerequisites:
1. **`passt` with `pesto`** — your source build from §3. Verify: `which pesto`.
2. **Low ports.** Rootless cannot bind < 1024. Caddy needs 80 and 443:
```
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
# persist:
echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf
```
This is a root-level change — unavoidable for rootless on privileged ports.
**Good news on IPv6:** pasta's main behavioural difference from `rootlessport` is that
it does not transparently forward IPv6 host traffic to IPv4, so services need explicit
IPv6 binding. I checked: monoco has **no global IPv6 address and no default IPv6
route**, and `mozimo.in`, `git.tanshu.com`, `cloud.hopsngrains.com`, `vault.tanshu.com`
resolve to IPv4 only (no AAAA). **So this limitation does not affect you.**
**Enable lingering**, or nothing survives logout or reboot:
```
sudo loginctl enable-linger tanshu
```
Currently `Linger=no`. Without this, Quadlet user services stop when your last session
ends — which on a server means immediately.
---
## 7. Cutover sequence
Pre-flight (no downtime):
1. Capture ground truth: `docker inspect` for every container/network/volume → JSON.
2. Resolve the `gitea` dual-uid question (§4.2) and confirm each app's true uid.
3. Export the two anonymous volumes (§4.3).
4. **Test restores**, not just backups — at least the Postgres dumps.
5. Back up `/var/lib/caddy/data` explicitly. It holds the ACME account key and all
issued certificates. It is a bind mount so it carries over, but losing it means
re-issuing ~20 certs against Let's Encrypt rate limits.
6. Build and install the Podman 6 stack (§3). `podman login registry.tanshu.com`.
7. Pre-pull every image. For locally-built `sms-wa-bridge:latest` (no registry prefix)
avoid a re-download: `docker save sms-wa-bridge:latest | podman load`.
8. Write and review all Quadlet units; `systemctl --user daemon-reload` and check
`systemctl --user list-unit-files | grep -E 'caddy|nextcloud'`.
9. **Take a provider snapshot.** Cheapest, fastest rollback you will ever buy.
10. Rehearse on a clone of the host, including a reboot.
Window (expect 10–20 minutes; 26 containers plus a 15 GB `chown`):
```
# 1. Stop the edge first so no request is half-served
docker stop caddy
docker stop $(docker ps -q)
# 2. Freeze state; nothing may write to /var/lib during the move
sudo systemctl stop docker.socket docker.service containerd.service
sudo systemctl disable docker.service
# 3. Move + re-own data (instant rename, then a metadata walk)
mv /var/lib/<svc> ~/podman-data/<svc> && chown -R tanshu:tanshu ~/podman-data/<svc>
# 4. Bring up the data tier first
systemctl --user start postgres.service valkey.service
# 5. Verify DNS inside the namespace BEFORE starting Caddy
podman exec nextcloud getent hosts postgres
podman exec nextcloud getent hosts redis
# 6. App tier, then Caddy last
systemctl --user start <each app>
systemctl --user start caddy.service
# 7. Reload + smoke test
podman exec caddy caddy reload
```
**Do not `docker rm` anything.** Stopped containers are the fastest rollback and cost
only disk. Keep them a week.
---
## 8. What must change in this repo and on the host
**(a) Legacy `--link` in the backup scripts — a guaranteed break.**
`db-backup.sh`, `nextcloud-backup.sh`, `hiwikijs-`, `tandoor-`, `kakawa-`,
`bitwarden-backup.sh` and their `-restore.sh` counterparts all do
`docker run --rm --link postgres:db … pg_dump -h db`. `--link` is Docker-only and is
absent from `podman run`. Under rootless there is no default `bridge` network to fall
back on either.
Best fix — drop the helper container and dump in place (no networking, no `--user`
juggling, no shared network):
```bash
podman exec -e PGPASSWORD="..." postgres \
pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"
```
Also pin the base image: the scripts mix `postgres:alpine` and `postgres:17-alpine`.
Bonus: once data is owned by `tanshu` and reachable without `sudo`, the `sudo tar` and
`chown $(logname)` gymnastics in `nextcloud-backup.sh` can go away.
**(b) Root's crontab cannot reach rootless containers.**
```
*/5 * * * * docker exec -u www-data nextcloud php cron.php
```
Root cannot `podman exec` into another user's rootless containers. Move this to a
**systemd user timer** alongside the Quadlet units — which is the idiomatic approach
anyway. Same for `~/bin/empty-trash.sh`, which also targets a container named `cloud`
that no longer exists (it is `nextcloud`).
**(c) Ansible modules.** `docker_container` / `docker_image` / `docker_network` become
`containers.podman`'s `podman_container` / `podman_image` / `podman_network`, which
shell out to the `podman` CLI and need **no Python SDK** — the `pip: name=docker` task
disappears. Note `podman_network` lacks `docker_network`'s `connected:`/`appends:`
parameters, which the `*/roles/network/tasks/main.yaml` files rely on. Prefer declaring
the full `network:` list on each `podman_container` and deleting those network roles —
that is precisely what Quadlet does, so the two stay in sync.
Also update `caddyfile_path` everywhere (now `/home/tanshu/podman-data/caddy/conf/Caddyfile`).
**(d) `~/.zshrc:128`** and any other `docker ` invocations in `~/bin`, `/etc/cron*`,
and the **external deploy pipelines** for `mozimo`, `brewman`, `menu-admin`, `hops`,
`greatbear`, `sms-wa-bridge`. Those live outside this repo. If they are not updated,
the next deploy will recreate Docker containers next to your Podman ones.
**(e) `postgres` currently publishes `5432` on `0.0.0.0`** and UFW is inactive. Fix
while you are in here — it needs no published port at all, only `expose`/network access.
---
## 9. Rollback
1. **`docker start`** — old containers exist, stopped but not removed; bind mounts
untouched. Reverting is `systemctl start docker` + `docker start $(docker ps -aq)`.
Valid as long as you do not `docker rm`. ⚠︎ This only works if no app performed an
irreversible migration on first start under Podman (Postgres major upgrade,
Nextcloud schema migration, Trillium migration). Check that risk per service, and
prefer the snapshot over trusting this.
2. **Provider snapshot** — instant whole-host revert. Covers `/var/lib` damage and
certificate loss.
3. **Backups** — once §8(a) is fixed and restores are verified.
Note that the `chown` pass is itself hard to unwind precisely: reverting to Docker
requires the original ownership (0:0, 33:33, 70:0, 1000:1000). Record the table in
§4.1 before you start, and re-`chown` on rollback.
---
## 10. Verification checklist
**Linkage**
- [ ] `podman exec caddy getent hosts <name>` resolves for all 20 upstreams
- [ ] `getent hosts db` and `getent hosts redis` resolve from the apps that use them
- [ ] Every container is on the same set of networks it had under Docker
**Rootless plumbing**
- [ ] `loginctl show-user tanshu | grep Linger` → `Linger=yes`
- [ ] `which pesto` returns a path; `rootless_port_forwarder` is set **before** first start
- [ ] Caddy sees **real client IPs** — check `podman logs caddy` for a genuine remote IP
- [ ] `X-Forwarded-For` reaches Nextcloud/Gitea correctly
**HTTP / TLS**
- [ ] `curl -sSI https://<vhost>` 200/redirect for all 20 site blocks
- [ ] Certificates are the **existing** ones (compare serial against the pre-migration
snapshot) — nothing re-issued
- [ ] WebSocket / streaming paths (Gitea, Nextcloud, RustDesk)
**Applications**
- [ ] Each app can read **and write** — catches the uid mapping (§4.2)
- [ ] Postgres accepts connections from every client
- [ ] `postgres` is no longer published on `0.0.0.0:5432`
**Operations**
- [ ] Nextcloud cron fires via the new systemd user timer
- [ ] A full backup cycle of every `~/bin/*-backup.sh` completes and restores
- [ ] **Reboot the host**; all containers return with correct policies
- [ ] `systemctl --user status caddy` is clean; `podman exec caddy caddy reload` works
- [ ] Disk is acceptable after Docker images are pruned (32 GB reclaimable)
---
## 11. Risk register
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| `--link` backup scripts break silently | **Certain** | Backups stop unnoticed | Fix §8(a) before the window; verify a full cycle |
| Data invisible due to uid mismatch | **High** | Apps fail to start or write | `keep-id` per §4.2 + `chown -R tanshu`; test each app's write path |
| Compiling on monoco OOMs production | **High if done on-box** | Outage during build | Build off-box (§3.2) |
| Containers do not return after reboot | High if `linger` skipped | Long unnoticed outage | `enable-linger` + explicit reboot test |
| pesto is experimental / breaks | Medium | Wrong client IPs, or Caddy unreachable | Snapshot first; know how to fall back to `rootlessport` |
| Switching forwarder with containers running | Medium | Corrupt port-forward state | Enable pesto **before** first start (§6) |
| Low-port sysctl not set | Medium | Caddy cannot bind 80/443 | §6; verify after reboot |
| Let's Encrypt rate limit | Low | Certs unavailable for a week | Preserve `/var/lib/caddy/data`; verify certs load, don't re-issue |
| External deploy pipelines recreate Docker containers | **High** | Duplicate apps, port conflicts | Audit and update CI (§8(d)) |
| Self-built stack drifts from security fixes | **Certain over time** | Unpatched engine on 20 sites | Rebuild pipeline + calendar reminder; or reconsider §2 |
| Two runtimes writing the same data | Medium | Database corruption | Hard-stop Docker before starting Podman |
---
## 12. Open questions
1. **Are we upgrading to Ubuntu 26.04 as part of this?** §3.4 argues yes, and that it
should happen **before** the Podman migration. This decision changes the rest of the
plan: on 26.04 there are **no source builds**, and §3 in its entirety becomes
optional/deferred.
2. **Where do `mozimo`, `brewman`, `menu-admin`, `hops`, `greatbear`, `sms-wa-bridge`
deploy from?** If those pipelines run `docker` against monoco they must change in
lockstep. Should they move into this repo?
3. **Is a temporary client-IP regression acceptable?** Recommended: yes — migrate on
stock Podman, accept gateway IPs at Caddy, and treat Podman 6 + pesto as a separate
later project. The Quadlet units do not change, so it is a drop-in upgrade.
4. **`postgres`'s `0.0.0.0:5432` exposure** — fold into this work or handle separately?
Resolved during the audit:
- ~~**`gitea`'s dual uid**~~ — the container sets `USER_UID=1003`/`USER_GID=1003`, and
`git` is uid 1003 on the host (`/home/git/.ssh` is `1003:1003`). The real data lives
in `/var/lib/gitea/gitea/` owned `git:git`. Use **`UserNS=keep-id:uid=1003,gid=1003`**.
- ~~**Build host / packaging**~~ — moot if §3.4 is taken; revisit only if the pesto
project proceeds.