diff --git a/docs/backup-plan.md b/docs/backup-plan.md new file mode 100644 index 0000000..39a9ced --- /dev/null +++ b/docs/backup-plan.md @@ -0,0 +1,469 @@ +# Full backup plan for `monoco` + +Status: **plan, not yet executed** +Audited: live host, read-only. No changes made. +Host: EC2 `i-08edfd71ce301b8ba`, `ap-south-1c`, Ubuntu 24.04.5 aarch64 + +--- + +## 1. Objective + +A complete, **verified** backup of monoco that: + +1. Can restore the entire host (disaster recovery), and +2. Can restore any single service in isolation (the important case day-to-day), and +3. Is a hard prerequisite for the Docker → Podman migration, because that migration + changes file ownership on every data directory. + +"Verified" is doing real work in that sentence. §10 defines what verification means, +and it is not optional. + +--- + +## 2. What is actually on the box + +| | | +|---|---| +| Provider | **EC2 confirmed** (IMDSv2), `i-08edfd71ce301b8ba`, `ap-south-1c` | +| Storage | **Single 80 GB EBS root volume** (`nvme0n1p1`), ext4, **no LVM**, no separate data volume | +| Free space | **30 GB** of 77 GB | +| Containers | 26, all plain `docker run`, all state on host bind mounts | +| Total app data | **~19 GB** | +| **Irreplaceable data** | **~2.4 GB** | + +That last line is the most important number in this document. The gap between 19 GB +and 2.4 GB is almost entirely one thing: + +``` +/var/lib/audiobookshelf/audiobooks 15.0 GB 696 files. Re-downloadable media. +/var/lib/audiobookshelf/config 2.6 MB ← absdatabase.sqlite, irreplaceable +/var/lib/audiobookshelf/metadata 19 MB ← irreplaceable +``` + +So the real state of Audiobookshelf is **22 MB, not 15 GB**. Treating it as 15 GB is +what makes a "full backup" feel impossible; it isn't. Decide explicitly whether the +media is in scope (§8) — but do not let it drive the design. + +### 2.1 What is irreplaceable, by service + +| Service | Total | Back up | Skip (regenerable / from image) | +|---|---|---|---| +| `postgres` | 995 MB | 13 databases, ~890 MB raw | — | +| `gitea` | 747 MB | `git/` 302 MB + `gitea/` 445 MB **incl. gitea.db** | — | +| `nextcloud` | 1.2 GB | `data/` 324 MB, `config/` (config.php!), `custom_apps/` 5.8 MB | `apps/` 522 MB, `core/` 159 MB, `dist/` 83 MB, `3rdparty/` 55 MB, `lib/` 19 MB | +| `mozimo` | 147 MB | `pb_data/` (incl. `auxiliary.db` 18.6 MB + WAL) | `types.d.ts` | +| `mozimo-staging` | 137 MB | `pb_data/` | | +| `greatbear` | 101 MB | `pb_data/` | | +| `tandoor` | 98 MB | `mediafiles/` | `staticfiles/`, `externalfiles/` | +| `menu-app.*` ×3 | 210 MB | `/data` each | | +| `trillium` | 106 MB | `document.db` 3.8 MB + `backup/` 12 MB | `log/` 91 MB | +| `sms-wa-bridge` | 397 MB | `rules.json` (tiny) | `wwebjs_auth/` 307 MB — re-linkable via QR | +| `hops` | 19 MB | `pb_data/` | | +| `audiobookshelf` | 15 GB | `config/` + `metadata/` = **22 MB** | `audiobooks/` 15 GB | +| `bitwarden` | 4.2 MB | **Postgres `bitwarden` DB** | `data/db.sqlite3` — **stale**, see §3.3 | +| `yamtrack` | 2.3 MB | `data/db.sqlite3` | | +| `affine` | 916 KB | `config/` + `storage/` | | +| `caddy` | 736 KB | **all of it** — ACME account + certs | | +| `hiwikijs` | ? | **anonymous volume** — see §13.1 | `/var/lib/hiwikijs` is an empty decoy | +| `valkey` | ? | — | cache, rebuildable | + +**~2.4 GB** once media and the WhatsApp session are excluded. + +--- + +## 3. The consistency problem — read this before writing any backup + +This is the part that silently produces useless backups. Every one of these is being +written to *right now* while the container runs. + +### 3.1 SQLite in WAL mode + +Live WAL sidecars are present on disk, which proves the databases are hot: + +``` +/var/lib/mozimo/pb_data/auxiliary.db 18,632,704 + -wal 1,116,552 + -shm +/var/lib/mozimo/pb_data/data.db 204,800 + -wal 78,312 + -shm +/var/lib/gitea/gitea/gitea.db 8,147,968 (modified 06:07 today) +/var/lib/audiobookshelf/config/absdatabase.sqlite 2,576,384 +/var/lib/trillium/document.db 3,800,000 + -shm +/var/lib/yamtrack/data/db.sqlite3 + -wal + -shm +``` + +**A plain `tar` of a live SQLite database can produce a backup that will not open.** +The main file and the WAL are copied at different instants, so the result can be torn. +This affects `gitea-backup.sh` today. + +Three acceptable methods, in order of preference: + +1. **Use the application's own dump command** — always best, because the app + understands its own invariants: + - Gitea: `docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini` + → one zip containing the DB, repos, and config. + - PocketBase (`mozimo`, `mozimo-staging`, `greatbear`, `hops`): use the built-in + backup, or stop the container for the copy. +2. **`sqlite3 ".backup "`** — safe on a live database, needs `sqlite3` + (present on the host: `~/.sqlite_history` exists). +3. **Stop the container, copy the directory, start it.** Simple, provable, costs + seconds of downtime per service. For a pre-migration backup this is the right + default, because the containers are being stopped anyway. + +### 3.2 Postgres + +13 databases, ~890 MB raw. A hot `tar` of `/var/lib/postgresql/data` is **not** a +valid backup — you need a logical dump or a proper physical base backup. + +``` +affine 11 MB | bitwarden 10 MB | brewman_acc 174 MB | brewman_exp 16 MB +brewman_hinchco 41 MB | brewman_hops 122 MB | brewman_mhl 132 MB | hiwikijs 16 MB +medusa-criollo 21 MB | nextcloud 319 MB | postgres 7.5 MB | tandoor 16 MB +zitadel 9.3 MB +``` + +Note two orphans with no running container — decide whether they are in scope: +**`medusa-criollo`** (21 MB, no container references it) and **`zitadel`** (9.3 MB, +container deleted). Everything else maps to a live service. + +Method: `pg_dump -Fc` per database (parallel-safe, individually restorable) **plus** +`pg_dumpall --roles-only` for roles/globals, which per-database dumps do not capture. + +### 3.3 The bitwarden trap + +`bitwarden` has `DATABASE_URL=postgresql://…@postgres:5432/bitwarden`, and a 10 MB +`bitwarden` database exists in Postgres. **Postgres is authoritative.** + +But `/var/lib/bitwarden/data/db.sqlite3` still exists with `-wal`/`-shm` sidecars, and +`~/bin/bitwarden-backup.sh` dumps the Postgres DB while `bitwarden-upload.sh` uploads a +`~/Data/warden/production.sqlite3` that **does not exist** (and `~/bin/gdrive` is not +installed). + +So the current bitwarden backup story is three-quarters broken, and there is a stale +SQLite file that would look like a plausible restore source and silently lose months of +data. **Back up the Postgres database. Explicitly mark or delete the stale SQLite.** + +### 3.4 Nextcloud + +Files can be copied live; the **database cannot**. Dump the DB and the files together +so they correspond. Nextcloud's own guidance is maintenance mode during the backup: +`occ maintenance:mode --on`, dump, copy, `--off`. `config/config.php` must be included — +without it the instance is not restorable. + +--- + +## 4. Backup architecture + +Standard 3-2-1, using what this host actually has: + +| Layer | What | Why | Where | +|---|---|---|---| +| **1. EBS snapshot** | Whole 80 GB volume | Instant whole-system restore; the only thing that captures the OS, Docker images, and 15 GB of media in one shot | AWS, S3-backed, inherently offsite | +| **2. Logical backup (restic)** | The ~2.4 GB of real state | Provably restorable, per-service, deduplicated, **encrypted** (matters — see §5) | S3 bucket in `ap-south-1` | +| **3. Local staging** | Transient only | Working space during the run | `/var/backups/` — **not** a backup | + +Why both 1 and 2: an EBS snapshot is *crash-consistent*, not application-consistent — +it is equivalent to pulling the power cord. Restoring it means Postgres and every +SQLite database go through crash recovery. It is excellent for "the instance died" and +poor for "I need last night's database cleanly". The logical layer is the one that +gives you a *verified* restore. + +**Region note:** an EBS snapshot in `ap-south-1` does not survive the loss of that +region or that AWS account. Enable **cross-region snapshot copy** (or at minimum +replicate the restic repo to a second bucket) before calling this disaster recovery. + +--- + +## 5. Phase 0 — freeze the definitions (no downtime, minutes) + +Do this first: it is tiny, has zero impact, and it is the part that is easiest to +forget and most annoying to reconstruct. + +- **Every container's full config** — `docker inspect` for all containers, networks, + volumes → JSON. This is the source of truth for the migration. +- **Secrets and configuration.** The env vars hold live credentials: Shopify client + secret, Sentry auth token, PocketBase superuser passwords, JWT private keys, + `MIDDLEWARE_SECRET_KEY`, DB passwords, `AUTH_SECRET`, Vince admin password. Capture + them all — and note they are why the backup repository **must be encrypted at rest**. +- **Caddy** — the Caddyfile *and* `/var/lib/caddy/data` (ACME account key + ~20 issued + certificates). Losing the account key means re-issuing everything against Let's + Encrypt rate limits. +- **Deployment scripts** — everything in `~/bin/`, both crontabs, `~/.zshrc`. +- **Package and image inventory** — `dpkg --get-selections`, `docker images --digests`, + so the host can be rebuilt from scratch if it must be. +- **DNS records** for the ~20 hostnames pointing at `13.202.6.2`. + +None of this needs the services stopped. + +--- + +## 6. Phase 1 — EBS snapshot (no downtime) + +``` +aws ec2 create-snapshot \ + --volume-id \ + --description "monoco pre-podman-migration $(date -Iseconds)" +``` + +Then: +- Tag it with the date, purpose, and the fact that it **predates the Podman migration**. +- Wait for `State: completed` (not just `pending`). A pending snapshot is not a backup. +- **Copy it to a second region** if it is meant to survive regional loss. + +This is your fastest rollback if the migration goes badly wrong. `awscli` is not +installed — install it or take the snapshot from the console. + +--- + +## 7. Phase 2 — the logical backup (the one that matters) + +Container-native, so it works identically before and after the Podman migration — +write it this way and you do not have to rewrite it during the cutover. + +### 7.1 Postgres (~890 MB → est. 200–300 MB compressed) + +``` +# roles and globals first — per-DB dumps do NOT include these +docker exec postgres pg_dumpall -U postgres --roles-only | gzip > roles.sql.gz + +# one restorable file per database +for db in affine bitwarden brewman_acc brewman_exp brewman_hinchco \ + brewman_hops brewman_mhl hiwikijs nextcloud tandoor; do + docker exec postgres pg_dump -U postgres -Fc "$db" > "$db.dump" +done +``` + +Deliberately excluded: `postgres` (system), and the `medusa-criollo` / `zitadel` +orphans — unless you decide otherwise (§3.2). + +### 7.2 Gitea — use its own dump + +``` +docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini -f /data/dumps +``` + +One archive with `gitea.db`, the repositories, and config, produced consistently. +A `tar` of `/var/lib/gitea` will *probably* work and will occasionally produce a +corrupt `gitea.db` (§3.1). + +### 7.3 SQLite services — stop, copy, start + +Simplest provably-correct method, and these are fast to restart: + +``` +for c in mozimo mozimo-staging greatbear hops trillium yamtrack audiobookshelf; do + docker stop "$c" +done +# tar the relevant paths (§2.1 — skip audiobooks/, log/, staticfiles/) +for c in ...; do docker start "$c"; done +``` + +Alternatives if downtime is unacceptable: `sqlite3 "$db" ".backup"` per database, or +PocketBase's built-in backup for the four `pb_data` services. + +### 7.4 Nextcloud + +``` +docker exec -u www-data nextcloud php occ maintenance:mode --on +docker exec postgres pg_dump -U postgres -Fc nextcloud > nextcloud.dump +tar -czf nextcloud-files.tar.gz \ + -C /var/lib/nextcloud data config custom_apps +docker exec -u www-data nextcloud php occ maintenance:mode --off +``` + +### 7.5 Caddy, affine, sms-wa-bridge, menu apps, tandoor media + +Plain file copies — none of these need a quiesce: +`/var/lib/caddy` (all 736 KB), `/var/lib/affine/{config,storage}`, +`/var/lib/sms-wa-bridge/rules.json`, each `/var/lib/menu.*`, +`/var/lib/tandoor/mediafiles`. + +### 7.6 The anonymous volumes + +`hiwikijs` and `valkey` are **not** under `/var/lib` — they live in +`/var/lib/docker/volumes//_data` and are invisible to every existing script. +Export them to real directories before the migration (the migration plan requires it +anyway): + +``` +docker run --rm -v 9c1e3fde…:/from -v /var/backups/vol:/to alpine \ + sh -c 'cp -a /from/. /to/hiwikijs/' +``` + +`hiwikijs` matters; `valkey` is a rebuildable cache. + +### 7.7 Package it + +Install **restic** (static aarch64 binary — nothing is installed today), initialise an +encrypted repo on S3, and back up the staging directory: + +``` +restic -r s3:s3.ap-south-1.amazonaws.com/monoco-backup init +restic -r s3:… backup /var/backups/monoco-$(date +%F) --tag pre-podman-migration +restic -r s3:… check +``` + +restic gives you encryption, deduplication (so daily runs are cheap), and snapshots you +can mount and restore individual files from. No credentials are configured on this host +today, so this needs an IAM role on the instance or an access key placed deliberately. + +--- + +## 8. Phase 3 — bulk media (decision required) + +The 15 GB of audiobooks and the 307 MB WhatsApp session. + +- **If the audiobooks exist elsewhere** (purchased library, cloud, seedbox), exclude + them. They are already covered by the EBS snapshot for the "instance died" case, and + including them inflates every backup run for no recovery benefit. +- **If they exist only here**, they must be in scope — but do not put them in restic + daily. Handle them as a separate, infrequent, one-way copy. + +This decision changes the backup from 2.4 GB to 18 GB. Make it explicitly. + +--- + +## 9. Where the backup physically goes + +There is **30 GB free** on a single volume. A backup written there is not a backup: it +shares the failure domain with the thing it protects, and a 19 GB archive would fill +the disk and take production down. + +| Destination | Status | +|---|---| +| S3 `ap-south-1` (restic) | **Recommended.** Same region = fast and cheap; add cross-region for DR | +| EBS snapshot | Recommended, whole-system | +| Google Drive via `~/bin/gdrive` | **Broken** — binary not installed, `~/Data/warden` does not exist | +| Another server | No SSH hosts configured on the box; `registry.tanshu.com` (3.109.171.183) is a candidate if it has space | +| Local disk | Staging only | + +--- + +## 10. Verification — acceptance criteria + +A backup that has never been restored is a hypothesis. Before the migration proceeds: + +- [ ] `restic check --read-data-subset=5%` passes +- [ ] **Every Postgres dump restores** into a scratch database, and a row count on a + known table matches production +- [ ] `roles.sql.gz` restores (roles/globals are the classic omission) +- [ ] The Gitea dump extracts and `gitea.db` opens with `sqlite3 … "PRAGMA integrity_check"` +- [ ] One PocketBase `data.db` restores and opens +- [ ] `trillium/document.db` and `yamtrack/db.sqlite3` open +- [ ] Nextcloud restores **with** `config/config.php`, in a scratch instance +- [ ] `/var/lib/caddy/data` contains the ACME account key and the certificate set +- [ ] The EBS snapshot reaches `completed`, and is **booted as a test instance** with + services confirmed running +- [ ] A manifest of checksums exists for the archive set +- [ ] **Original uid:gid ownership of every `/var/lib` path is recorded** — required to + undo the migration's `chown` (§14) + +--- + +## 11. Ongoing backups — currently there are none + +This is the biggest single finding: **nothing is scheduled.** + +- The only cron entry on the host is `*/5 * * * * docker exec -u www-data nextcloud php cron.php`. +- `tanshu`, `ubuntu`, and `git` have no crontab. +- No systemd timer mentions backup (only `dpkg-db-backup.timer`). +- No restic, borg, rclone, duplicity, or aws CLI is installed. +- The five archives in `~/` are **manual one-offs** — Jun 2025, Feb 2026, Sep 2026, + Oct 2026 — for bitwarden, brewman_hops (SQL only), and nextcloud, plus obsolete + `mongodb.tar` and `overleaf.tar`. + +So backups depend on someone remembering. Recommended replacement — a `restic` run on a +systemd timer, with all the container-native dump steps from §7 baked into a script, +retention of **7 daily / 4 weekly / 6 monthly**, and **failure alerting**. A backup that +fails silently every night for three months is indistinguishable from no backup, which +is exactly the state today. + +--- + +## 12. Retention + +| Layer | Retention | +|---|---| +| Logical (restic) | 7 daily, 4 weekly, 6 monthly | +| Weekly full | 3 months | +| Monthly full | 12 months | +| EBS snapshots | Keep the pre-migration one indefinitely; daily snapshots 7–14 days | + +--- + +## 13. Bugs in the existing backup scripts + +Worth fixing regardless of the migration, because these are actively misleading: + +**13.1 `hiwikijs-backup.sh` backs up nothing.** It runs +`tar -C /var/lib hiwikijs`, but `/var/lib/hiwikijs` is an **empty directory** +(`drwxr-xr-x www-data`, dated Dec 2022). The real Wiki.js content is in the anonymous +volume `9c1e3fde…`. The script exits successfully and produces a valid archive of +nothing. Anyone trusting it has no Wiki.js backup at all. + +**13.2 `gitea-backup.sh` hot-copies a live SQLite database** (§3.1). It will usually +work, which is the worst property a backup can have. Replace with `gitea dump`. + +**13.3 `bitwarden-backup.sh` + `bitwarden-upload.sh` are inconsistent with each other +and with reality** (§3.3) — one dumps Postgres, the other uploads a SQLite file that +does not exist, and neither runs on a schedule. + +**13.4 `iot-backup.sh` and `kakawa-backup.sh` target services that no longer exist** +(the `iot_net`/`kakawa_net` Docker networks are still attached to Caddy, but the +containers are gone). + +**13.5 All DB scripts use `--link postgres:db`**, which does not exist in Podman and +will break at cutover. Under rootless Podman the `--user $(id -u)` and `sudo chown` +gymnastics are also unnecessary once the data is owned by you. + +**13.6 No coverage at all** for: `mozimo`, `mozimo-staging`, `greatbear`, `hops`, +the three `menu-app.*` containers, `affine`, `vince`, `audiobookshelf`, `valkey`, or +`/var/lib/caddy` (the certificates). + +--- + +## 14. Pre-migration specifics + +Beyond a normal full backup, the migration needs these captured, because it changes +ownership and layout: + +1. **Original `uid:gid` and mode of every bind-mount source** — the migration's + `chown -R tanshu:tanshu` is not trivially reversible without this table. +2. **The two anonymous volumes exported** (§7.6) — otherwise `hiwikijs` state is lost + permanently when Docker is removed. +3. **`docker inspect` JSON for all 26 containers** — the only complete record of the + runtime configuration. +4. **A restore rehearsal** onto a scratch host or the booted EBS snapshot, following the + migration plan's rollback path. + +--- + +## 15. Risks + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| Backups silently produce corrupt SQLite | **High** | Data loss discovered at restore time | §3.1 methods; integrity-check every DB in verification | +| `hiwikijs` backed up as an empty directory | **Certain today** | Permanent wiki loss | Export the anonymous volume (§7.6) | +| Postgres roles/globals omitted | Medium | Restore fails on permissions | `pg_dumpall --roles-only` (§7.1) | +| Stale bitwarden SQLite restored instead of Postgres | Medium | Months of vault data lost | §3.3 — mark or delete it | +| Let's Encrypt account key lost | Low | ~20 certs blocked by rate limits | Include all of `/var/lib/caddy` | +| Backup written to the same volume | Medium | Both lost together | S3/Snapshot only (§9) | +| 30 GB free filled by staging | Medium | Production outage during backup | Stage only the ~2.4 GB of real state; media excluded | +| Snapshot left in `pending` | Low | Restore from an incomplete snapshot | Wait for `completed`; boot-test it | +| Same-region snapshot as sole DR | Medium | Region/account loss unrecoverable | Cross-region copy | +| No alerting on backup failure | **High** | Months of silent failure | Monitoring on the timer (§11) | + +--- + +## 16. Open questions + +1. **Is the 15 GB of audiobook media in scope?** (§8) This decides 2.4 GB vs 18 GB. +2. **Where do backups go?** S3 in `ap-south-1` is the natural fit — is there an IAM role + on the instance, or should a dedicated bucket + credentials be set up? +3. **Are `medusa-criollo` and `zitadel` databases dead?** If so, exclude them and drop + them. +4. **The 1.3 MB `bitwarden-backup_202602261629.tar.gz`** — is the SQLite in it the + pre-Postgres state? If so it is a historical artifact worth keeping deliberately, + and the live `db.sqlite3` should be deleted so nobody restores the wrong thing. +5. **Is `registry.tanshu.com` a viable second backup target** (independent host, but + same region and presumably same provider account)? +6. **Should the ongoing backup be restic, or plain `pg_dump` + `tar` on a timer?** + restic is better but adds a component; the container-native dumps are needed either + way. diff --git a/docs/podman-compat-audit.md b/docs/podman-compat-audit.md new file mode 100644 index 0000000..61cf7ad --- /dev/null +++ b/docs/podman-compat-audit.md @@ -0,0 +1,318 @@ +# Podman compatibility audit — monoco container fleet + +Status: **planning only.** Read-only inspection; nothing on monoco was modified. +Scope: all 26 running containers, audited for rootless Podman + Quadlet. +Companion docs: [podman-migration.md](podman-migration.md), [backup-plan.md](backup-plan.md) + +--- + +## 1. Verdict + +**Nothing in this fleet blocks a rootless Podman migration.** There is no privileged +container, no capability add, no device, no host networking, no custom seccomp, no +Docker-socket mount, and no Compose file. Every image is **arm64**, so there is no +emulation. + +The breakages are all in the *surrounding tooling* (backup scripts, cron, Ansible +modules) and in *ownership/volume handling* — not in the images. Your instinct that +your own `registry.tanshu.com` images will be fine is correct; I checked them +specifically and they are among the cleanest in the fleet. + +Four things need real attention, and one of them is not obvious: + +| # | Issue | Severity | +|---|---|---| +| 1 | `hiwikijs` and `valkey` use **anonymous Docker volumes** — copying `/var/lib/*` misses them entirely | **data loss** | +| 2 | Every data tree needs re-owning, and **each app's uid differs** (0 / 33 / 70 / 999 / 1000 / 1003) | blocks startup | +| 3 | `--link` in backup scripts, and root's crontab doing `docker exec` | breaks backups + cron | +| 4 | Locally-built `sms-wa-bridge` has no registry — short-name resolution fails | won't start | + +--- + +## 2. What I verified as clean + +Worth stating explicitly, because it is why this migration is tractable: + +- **Architecture**: every image is `arm64` / `linux`. No binfmt/qemu needed. +- **Privileges**: `Privileged=false`, `CapAdd=[]`, `CapDrop=[]`, `Devices=[]` on all 26. +- **Namespaces**: no `--network host`, no `--pid host`, no `--ipc host`, no `--userns host`. +- **Security**: no custom `SecurityOpt`, no custom seccomp profile, no `--extra-hosts`. +- **Docker socket**: not mounted anywhere. No image talks to the Docker API. +- **Compose**: none. Every container is a plain `docker run` / Ansible `docker_container`. +- **shm**: all at the 64 MB default; nothing needs a larger `/dev/shm`. +- **Sysctls**: none set. +- **Resource limits**: `Memory=0`, `NanoCpus=0` on every container. (The Ansible role + for gitea *declares* `cpus: .10` / `memory: 512M`, but they are **not in effect** on + the running container — so no cgroup-delegation concerns for rootless.) +- **Custom DNS**: `Dns=[]` everywhere; aardvark-dns will replace Docker's `127.0.0.11`. +- **Hostnames**: all default (container ID). Nothing depends on a fixed hostname — + `sms-wa-bridge` explicitly cleans its own `Singleton*` locks on start because of this. + +--- + +## 3. The four real problems + +### 3.1 Anonymous volumes — the one that loses data + +Eight images declare `VOLUME`: + +| Image | VOLUME | Mounted? | +|---|---|---| +| `vaultwarden/server` | `/data` | ✅ bind-mounted | +| `gitea/gitea` | `/data` | ✅ bind-mounted | +| `registry.tanshu.com/{greatbear,hops}` | `/app/pb_data` | ✅ bind-mounted | +| `nextcloud` | `/var/www/html` | ✅ bind-mounted | +| `postgres:17-alpine` | `/var/lib/postgresql/data` | ✅ bind-mounted | +| **`ghcr.io/requarks/wiki`** | `/wiki/data/content` | ❌ **anonymous volume** | +| **`valkey/valkey:alpine`** | `/data` | ❌ **anonymous volume** | + +The two anonymous ones live in `/var/lib/docker/volumes//_data`, **not** under +`/var/lib/`. A copy of the data directories will silently omit them. + +`hiwikijs` is the serious one: `/var/lib/hiwikijs` exists but is an **empty directory** +from Dec 2022, and `~/bin/hiwikijs-backup.sh` archives exactly that empty directory and +exits successfully. So Wiki.js currently has **no backup at all** — this predates the +migration and should be fixed regardless of it. + +Fix: export both to real directories (e.g. `~/podman-data/hiwikijs/content`, +`~/podman-data/valkey`) and declare them as Quadlet `Volume=` bind mounts. `valkey` is a +rebuildable cache; `hiwikijs` is not. + +### 3.2 Ownership — one uid per container, and they all differ + +Under rootless Podman the user namespace maps only **one** uid:gid to your host user. +The rule is `UserNS=keep-id:uid=,gid=`, where `` is the uid +the application actually writes as — **not** the image's `USER` and **not** `docker exec id`. + +Measured (see §4 for the full matrix): + +| App uid | Containers | +|---|---| +| 0 | caddy, brewman ×5, greatbear, hops, affine, audiobookshelf, bitwarden, tandoor, sms-wa-bridge, vince | +| 33 | nextcloud | +| 70 | postgres | +| 999 | valkey | +| 1000 | mozimo, mozimo-staging, hiwikijs, menu-app ×3, yamtrack, trillium | +| 1003 | gitea | + +Trap: `docker exec id -u` reports the **image's** configured user, not the running +process. It says `0` for `postgres` (which actually writes as **70**), for `nextcloud` +(**33**), and for `gitea` (**1003**) — because those entrypoints drop privileges after +starting. Derive the uid from **the ownership of what the app writes**, which is what +the table above does. + +Also note `tanshu` is **uid 1001**, not 1000 — most of the uid-1000 data is currently +owned by the `ubuntu` user. "Chown it to me" is therefore not a no-op anywhere. + +The reassuring part: for the entrypoints that start as root and drop to an app user +(`postgres`, `nextcloud`, `gitea`, `trillium`, `yamtrack`), the root phase maps to a +subuid and its internal `chown` still lands on your host user. So a single +`keep-id:uid=` plus `chown -R tanshu:tanshu` should work — **but verify per +service**, because a file written during the root phase and never chowned will be +subuid-owned and unreadable by the dropped-privilege process. + +### 3.3 Tooling that assumes Docker + +- **`--link postgres:db`** in `db-backup.sh`, `bitwarden-`, `hiwikijs-`, `kakawa-`, + `nextcloud-`, `tandoor-backup.sh` and their restores. No Podman equivalent. +- **Root's crontab**: `*/5 * * * * docker exec -u www-data nextcloud php cron.php`. + Root **cannot** `podman exec` into another user's rootless containers — this must + become a systemd **user** timer. +- **`~/bin/empty-trash.sh`** execs a container named `cloud`, which no longer exists + (it is `nextcloud`). +- **Ansible**: `docker_container` / `docker_image` / `docker_network` → + `containers.podman`. Note `podman_network` has **no** `connected:`/`appends:` + parameters, which every `*/roles/network/tasks/main.yaml` in this repo relies on. + Declare the full `network:` list on each `podman_container` instead. +- **Everything outside this repo.** You said ~90% is Ansible and the rest is + `docker run`. The deploy pipelines for `mozimo`, `brewman`, `menu-admin`, `hops`, + `greatbear` and `sms-wa-bridge` live elsewhere — if they still call `docker` against + the host, they will recreate Docker containers alongside your Podman ones. + +### 3.4 Image naming + +Rootless Podman resolves short names against `unqualified-search-registries`. In a +**Quadlet/systemd context there is no TTY to confirm a short-name prompt**, so +ambiguous names fail the unit rather than prompting. + +- **`sms-wa-bridge`** was built locally with `docker build -t sms-wa-bridge` — no + registry. Podman will try `docker.io/library/sms-wa-bridge` and fail. Transfer it with + `docker save sms-wa-bridge | podman load` and reference **`localhost/sms-wa-bridge:latest`**. +- `nextcloud`, `caddy`, `postgres`, `valkey/valkey`, `gitea/gitea`, `vaultwarden/server`, + `vabene1111/recipes`, `triliumnext/notes` are all short or implicit-`docker.io` names. + **Fully qualify all of them** (`docker.io/library/nextcloud:latest` etc.) so resolution + never depends on host config. +- `registry.tanshu.com/*` and `ghcr.io/*` are already qualified. ✅ + +--- + +## 4. Per-container matrix + +`App uid` is what the application writes as. `Anon vol` = uses an anonymous Docker volume. + +| Container | Image | App uid | Anon vol | Verdict | Notes | +|---|---|---|---|---|---| +| `caddy` | `caddy:latest` | 0 | – | ⚠️ | Needs `ip_unprivileged_port_start≤80`; publishes 80, 443, **443/udp (HTTP/3)** — verify UDP forwarding | +| `postgres` | `postgres:17-alpine` | **70** | – | ⚠️ | PGDATA is `pgdata/` under the mount, owned uid 70. Drop the `0.0.0.0:5432` publish | +| `valkey` | `valkey/valkey:alpine` | **999** | **YES** | ⚠️ | Export the anonymous volume | +| `nextcloud` | `nextcloud:latest` | **33** | – | ✅ | `VOLUME /var/www/html` is bind-mounted; needs `POSTGRES_HOST=db` / `REDIS_HOST=redis` aliases | +| `gitea` | `gitea/gitea:latest` | **1003** | – | ⚠️ | `/var/lib/gitea/ssh` holds **SSH host keys** — copy them or clients see host-key warnings | +| `mozimo` | `registry.tanshu.com/mozimo:4a3863a` | 1000 | – | ✅ | Image `USER=node`; entrypoint starts PocketBase + SSR together | +| `mozimo-staging` | `registry.tanshu.com/mozimo:3929d3f` | 1000 | – | ✅ | | +| `greatbear` | `registry.tanshu.com/tanshu/greatbear` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted | +| `hops` | `registry.tanshu.com/hops:latest` | 0 | – | ✅ | `VOLUME /app/pb_data` bind-mounted | +| `menu-app.greatbear.in` | `registry.tanshu.com/menu-admin` | 1000 | – | ⚠️ | **Container names contain dots** — name the Quadlet units `menu-greatbear.container` etc. and set `ContainerName=` explicitly so the Caddyfile keeps resolving | +| `menu-app.hngmohali.com` | ″ | 1000 | – | ⚠️ | ″ | +| `menu-app.hopsngrains.com` | ″ | 1000 | – | ⚠️ | ″ | +| `hiwikijs` | `ghcr.io/requarks/wiki:latest` | 1000 | **YES** | ❌ | Content is in an anonymous volume; `/var/lib/hiwikijs` is an empty decoy | +| `affine` | `ghcr.io/toeverything/affine-graphql:stable` | 0 | – | ✅ | Needs postgres + valkey; indexer disabled (`AFFINE_INDEXER_ENABLED=false`), so no pgvector dependency | +| `audiobookshelf` | `ghcr.io/advplyr/audiobookshelf` | 0 | – | ✅ | `tini` entrypoint; 15 GB is media, state is 22 MB | +| `bitwarden` | `vaultwarden/server:latest` | 0 | – | ⚠️ | Authoritative store is **Postgres**; the `db.sqlite3` on disk is stale — do not restore it | +| `yamtrack` | `ghcr.io/fuzzygrim/yamtrack` | 1000 | – | ✅ | SQLite + WAL; needs a consistent copy | +| `trillium` | `triliumnext/notes:latest` | 1000 | – | ✅ | `document.db` + WAL; has its own `backup/` dir | +| `tandoor` | `vabene1111/recipes:latest` | 0 | – | ✅ | Needs `POSTGRES_HOST=db` alias | +| `sms-wa-bridge` | `sms-wa-bridge` (local) | 0 | – | ⚠️ | Chromium **is fine** (see below). Must be `localhost/sms-wa-bridge:latest` | +| `vince` | `ghcr.io/vinceanalytics/vince` | 0 | – | ⚠️ | **Distroless** — no shell, no `id`, so no exec-based checks. Container was run with `serve`, image `CMD` is null → set `Exec=serve` | +| `brewman-acc/exp/hinchco/hops/mhl` | `registry.tanshu.com/brewman:latest` | 0 | – | ✅ | 5 containers, identical config, one DB each | + +### The Chromium question, answered + +`sms-wa-bridge` is the classic rootless-Podman landmine: Chromium normally needs +`CLONE_NEWUSER` for its sandbox, which is blocked inside a container, and it can also +die on a 64 MB `/dev/shm`. I checked `server.js` and it already handles both: + +```js +puppeteer: { + headless: true, + executablePath: process.env.CHROMIUM_PATH || undefined, + args: ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"] +} +``` + +`--no-sandbox` removes the userns dependency and `--disable-dev-shm-usage` removes the +shm dependency. **No action needed** — this container will behave identically. + +--- + +## 5. Your migration thesis — validated, with three corrections + +> *"if we copy the data directories, change ownership and deploy on new server, we should be good to go"* + +**Broadly yes** — and with downtime free, this is a much safer plan than an in-place +cutover, because you can stop everything cleanly and copy quiesced data. Three +corrections: + +**Correction 1 — "the data directories" is not all the state.** Adding to `/var/lib/*`: +- the two **anonymous volumes** (§3.1) — `hiwikijs` is genuinely at risk; +- **`/var/lib/gitea/ssh`** (SSH host keys) — easy to miss, and clients will complain; +- **`/var/lib/caddy/data`** — the ACME account key and ~20 certificates. Lose it and you + re-issue everything against Let's Encrypt rate limits; +- **the environment variables** — that is where the credentials live (Shopify client + secret, Sentry token, JWT private keys, PocketBase passwords, DB passwords); +- **`/etc/crontab` entries and `~/bin/`** — the operational glue. + +**Correction 2 — "change ownership" is per-container, not global.** Chowning everything +to `tanshu` is right, but it only works if each container is also given the matching +`UserNS=keep-id:uid=`. Get the pairing wrong and the app starts and then fails +to write — the failure mode is a permissions error at runtime, not a clean startup +failure. The §4 table gives the pairing. + +**Correction 3 — Postgres wants a clean stop, and a dump is still worth it.** Copying +`PGDATA` after a clean shutdown is valid *at the same major version* (17.5 here). But a +logical dump is cheap and gives you a version-independent fallback. Do both. + +Since downtime is not a constraint, the copying step should be: + +``` +1. Stop all containers cleanly. +2. Copy: /var/lib//, /var/lib/gitea/ssh, the two anonymous volumes, + /var/lib/caddy/data, ~/bin/, env vars, docker inspect JSON. +3. Verify checksums on the copy. +4. chown -R tanshu:tanshu on the destination, per §3.2. +5. Load images (docker save | podman load, or podman pull + podman login). +6. Write Quadlet units, set UserNS / Network / Volume. +7. systemctl --user daemon-reload && start the data tier, then apps, then Caddy. +``` + +Because you stop cleanly, the SQLite WAL problem from the backup plan largely +disappears — no live WAL, no torn copies. That is the single biggest simplification +this approach buys you. + +--- + +## 6. Low ports and lingering + +You already know both, confirming for completeness: + +**Ports.** Rootless cannot bind < 1024. Caddy binds 80/443 **inside** its own network +namespace (where container-root has `CAP_NET_BIND_SERVICE`, which Podman grants by +default), but the **host-side publish** needs the sysctl: + +``` +sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80 +echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf +``` + +Verify after reboot — this is a favourite thing to forget on a fresh instance. + +**Lingering.** Without it, Quadlet **user** services stop when the login session ends, +which on a server is immediately, and they do not start at boot: + +``` +sudo loginctl enable-linger tanshu +``` + +Also confirm `/etc/subuid` and `/etc/subgid` have a range for the new user — monoco has +`tanshu:165536:65536`, and a fresh instance will assign a different base. Do not +hardcode `165536` anywhere. + +--- + +## 7. Cutover notes specific to a new instance + +**DNS and certificates are the thing to plan around.** All ~20 hostnames resolve to +`13.202.6.2`. On a new instance with a new IP: + +- Caddy cannot obtain certificates until DNS points at the new host (HTTP-01), and the + existing certs in `/var/lib/caddy/data` are only valid to *serve* for those names once + traffic arrives. +- **Check whether `13.202.6.2` is an Elastic IP.** If it is, reassign it to the new + instance and DNS, certificates, and clients are all untouched — by far the cleanest + cutover. Worth confirming before designing around a DNS change. + +**Logs do not migrate.** All 26 containers use the `json-file` driver; Podman uses +journald. If the existing logs matter, export them before decommissioning. Otherwise +`podman logs` / `journalctl --user -u ` replaces `docker logs`. + +**Image transfer.** `docker save | podman load` avoids re-downloading (and avoids +registry auth for the local image). For `registry.tanshu.com`, run `podman login` on the +new host so `podman auto-update`/pulls work. + +**The OS question is still open.** Per [podman-migration.md §3.4](podman-migration.md), +a **fresh 26.04 instance with stock Podman 5.7 needs no source builds at all**. Since you +are standing up a new instance anyway, this is now essentially free — and it makes the +"build Podman 6 for pesto" question entirely separable from the migration. + +--- + +## 8. Per-service verification after cutover + +For each container, in this order: + +1. **Starts and stays up** — `systemctl --user status ` +2. **Writes as the right uid** — create/update something through the app, then confirm + on the host that the new file is owned by `tanshu`. This is the check that catches a + wrong `keep-id`. +3. **Resolves its dependencies** — `podman exec getent hosts postgres` (and `db` / + `redis` aliases where used) +4. **Healthcheck reports healthy** — `podman inspect --format '{{.State.Health.Status}}'` +5. Special cases: + - `caddy`: real client IPs in the access log (this is where the rootless `rootlessport` + limitation shows up — see §3.4 of the migration plan); `https://` responds; + HTTP/3 (`curl --http3`) if you care + - `sms-wa-bridge`: WhatsApp re-links (or restores) and a test message sends + - `gitea`: `git clone` over **both** HTTPS and SSH (port 2222), no host-key warning + - `postgres`: every client app connects; nothing listening on host `5432` + - `nextcloud`: upload a file (exercises the uid-33 mapping and `REDIS_HOST`) + - `vince`: reachable on `stats.mozimo.in` (distroless, so no shell to diagnose with) +6. **Reboot the host** and confirm everything returns unattended diff --git a/docs/podman-migration.md b/docs/podman-migration.md new file mode 100644 index 0000000..e1a8f85 --- /dev/null +++ b/docs/podman-migration.md @@ -0,0 +1,679 @@ +# Migrating `monoco` from Docker to rootless Podman 6 + Quadlet + +Status: **plan, not yet executed** +Revised for: rootless Podman 6.x, data in `~/podman-data/`, per-app networks, systemd Quadlet units in `~/.config/containers/systemd/` +Audited: live `monoco` — Ubuntu 24.04.5 **aarch64**, 2 vCPU Neoverse-N1, 7.6 GB RAM, Docker 29.8.1, 26 containers + +> This replaces an earlier rootful draft. The rootful sections are gone; §2 explains +> what the rootless choice costs, so the tradeoff is on the record rather than a surprise later. + +--- + +## 1. Decisions locked in + +| Decision | Value | +|---|---| +| Runtime | Rootless Podman **6.x**, built from source | +| Units | Quadlet `.container` / `.network` files in `~/.config/containers/systemd/` | +| Data | `~/podman-data//` (was `/var/lib//`) | +| Networks | One per logical app, as today — **not** consolidated | +| Client IPs | Preserved via **pesto** (`rootless_port_forwarder = "pasta"`) | +| Caddy | Rootless, multi-homed onto every app network | + +--- + +## 2. What this costs — read once, then proceed + +You asked whether Podman 6 can be built from source. **Yes.** But three things should +be on the record before we start, because they are not obvious from the request: + +**1. Building 6.x solves one problem and creates a permanent maintenance obligation.** +The *only* thing the 6.x build buys over stock 4.9 is real client IPs at Caddy (via +pesto). Everything else in this plan works on stock Podman. In exchange you own +five source-built projects across three toolchains, with no distro security updates, +on a host serving ~20 sites. Pesto itself is upstream-described as *not default*, with +*no graceful upgrade path*. **§3.4 shows how upgrading to Ubuntu 26.04 removes the +source builds entirely — read that before committing to §3.** + +**2. Building 6.x does not reduce the rootless work at all.** The expensive part of +rootless is re-owning ~19 GB of data into a user namespace (§4). That cost is +identical on 4.9, 5.x, or 6.x. If the goal is "off Docker soon", the version is +orthogonal. + +**3. monoco is a 2 vCPU / 7.6 GB ARM64 box with ~2.5 GB RAM free.** Do not compile +on it (§3.2). Rust builds of netavark will thrash a machine that is already running +26 containers. + +None of this argues against doing it — rootless is a legitimate goal and pesto is a +real fix. It argues for building elsewhere and phasing carefully. + +--- + +## 3. Building Podman 6.x + Netavark 2 + pesto from source + +### 3.1 Component and version matrix + +Podman 6.0.0 (released 2026-06-24, latest patch 6.0.1) is not packaged for Ubuntu +24.04 by anyone. I checked `devel:kubic:libcontainers:unstable` (has netavark 1.12.2, +aardvark-dns 1.13.1, conmon, fuse-overlayfs — **no podman**), `isv:containers`, and +`devel:kubic:libcontainers:testing`. Debian unstable is still on podman 5.8.1 with a +bug open requesting 6.0.1. So source is genuinely the only path. + +| Component | Version needed | Ubuntu 24.04 has | Gap | +|---|---|---|---| +| **podman** | 6.0.x | 4.9.3 | **build** | +| — Go toolchain | **1.25.6** (`go.mod`) | go 1.22 | **bootstrap Go** | +| **netavark** | **2.x** (latest 2.1.0) — hard requirement | 1.4.0 | **build** | +| **aardvark-dns** | **2.x** | 1.4.0 | **build** | +| — Rust toolchain | **1.88** (`rust-version`) | rustc 1.75 | **bootstrap rustup** | +| **passt** (ships `pesto`) | **≥ 0^20260507.g1afd4ed** | 0^20240220 | **build** | +| **container-libs** common | v0.68.0 | — | **build** | +| buildah / skopeo | 1.44.0 / 1.23 | optional | only if you build/push images on monoco | +| libgpgme-dev, libbtrfs-dev, libseccomp-dev, libassuan-dev | any | ✅ present | none | +| build-essential, pkg-config | any | ❌ gcc not installed | `apt install` | + +Podman 6 breaking changes that matter here, all of which monoco satisfies: + +- **iptables support removed** → nftables required. Host has `nft` 1.0.9 ✅ +- **CNI removed** → netavark. Fine. +- **slirp4netns removed** → pasta required. +- **cgroups v1 removed** → `stat -fc %T /sys/fs/cgroup/` = `cgroup2fs` ✅ +- **BoltDB removed** → SQLite. +- Netavark and Aardvark **must be 2.0.0** — 1.x will not work. + +### 3.2 Build off-box + +monoco has 2 vCPU, 7.6 GB RAM, ~2.5 GB available, and 26 production containers. +Rust compilation of netavark + aardvark-dns is the worst case for that profile. + +**Build on a separate aarch64 machine or in CI, then ship artifacts.** All three +projects are portable: + +- **Podman / container-libs**: Go — `CGO_ENABLED=0` cross-compiles trivially; for + `GOARCH=arm64` from any host. +- **netavark / aardvark-dns**: Rust — `cargo build --release --target aarch64-unknown-linux-gnu`. +- **passt/pesto**: plain C `make`. + +Preferred output: four `.deb` packages (podman, netavark, aardvark-dns, passt) so +`apt` tracks them and you can rebuild on security updates. A tarball + install script +is acceptable but you lose upgrade tracking. + +Build reference: +- Podman: `go.podman.io/podman/v6` (the import path moved to a CNCF org in 6.0) +- netavark: `make` in `containers/netavark`, installs `netavark` + config +- aardvark-dns: `containers/aardvark-dns` +- passt: `passt.top`, installs `passt`, `pasta`, and **`pesto`** + +### 3.3 Install order on monoco + +``` +1. apt install build-essential pkg-config libgpgme-dev libbtrfs-dev libseccomp-dev libassuan-dev +2. apt install passt # baseline; then replace with your built passt+pesto +3. install netavark 2.x + aardvark-dns 2.x (to /usr/libexec/podman/) +4. install podman 6.x (to /usr/bin/podman) +5. verify: podman info --debug | grep -E 'networkBackend|version' +``` + +Keep Ubuntu's `podman` 4.9 package **uninstalled** — do not let apt install it as a +dependency and shadow your build. Pin with `apt-mark hold` if something pulls it in. + +### 3.4 What about upgrading to Ubuntu 26.04 LTS first? + +Short answer: **26.04 is a large improvement and worth doing, but it does not give you +pesto — so it does not remove the source builds if client IPs are the goal.** + +26.04 "Resolute" ships (vs 24.04 today): + +| Package | 24.04 (now) | 26.04 LTS | +|---|---|---| +| podman | 4.9.3 | **5.7.0** | +| passt | 0.0~git20240220 | **0.0~git20260120** | +| netavark | 1.4.0 | **1.16.1** | +| aardvark-dns | 1.4.0 | **1.16.0** | +| crun | — | 1.21 | +| buildah | — | 1.42.1 | + +Two independent blockers, either of which alone is fatal: + +1. **`rootless_port_forwarder` is Podman 6.0+.** The implementing PR + (`containers/podman#28478`) merged **2026-05-21** into `main` and shipped in + **6.0.0 on 2026-06-24**. 26.04 carries 5.7.0, which predates it. The Podman 6 + announcement lists pesto as a 6.0 feature, and the option is documented in + container-libs `main` as `rootlessport` (default) | `pasta`. +2. **`pesto` needs passt ≥ `0^20260507`.** 26.04 ships `0^20260120` — about three and + a half months too old. + +Also note Podman 6 **hard-requires netavark/aardvark-dns 2.0.0**, and 26.04 has 1.16 — +so moving to 6.x means building those too, on any base. + +**The useful reframe:** pesto is a *separable, optional* improvement, not a migration +prerequisite. Losing real client IPs degrades Caddy access logs, `X-Forwarded-For` +downstream, Nextcloud/Vaultwarden IP-based brute-force protection, and Vince's +server-side analytics — but it does not break the reverse proxy, and nothing goes down. +That means the migration should not wait on it. + +**Recommended sequencing** (revising §6 of the earlier draft): + +| # | Step | Notes | +|---|---|---| +| 1 | Full backup + **verified restores** | see `docs/backup-plan.md` | +| 2 | **OS upgrade 24.04 → 26.04**, Docker still running | supported LTS→LTS path; needs its own window | +| 3 | Migrate Docker → Podman on **stock 5.7**, `~/.config/containers/systemd/` | **no source builds at all** | +| 4 | *Later, separately:* decide whether client IPs justify Podman 6 + pesto | orthogonal; Quadlet units unchanged | + +Doing the OS upgrade **first** matters: migrating to a self-built Podman 6 on 24.04 and +*then* upgrading the OS leaves hand-built binaries fighting the distro packages at +exactly the moment you least want that. + +The tradeoff, stated plainly (see §2 for the rootless case): + +| | Source builds | Data re-owning | Real client IPs | +|---|---|---|---| +| Rootless, 26.04, stock 5.7 | none | required | ❌ | +| Rootless, 26.04, built 6.0 + pesto | 4 projects | required | ✅ (experimental) | +| Rootful, 26.04, stock 5.7 | none | **not required** | ✅ | + +--- + +## 4. The uid mapping problem — the biggest piece of actual work + +This is where rootless diverges hardest from Docker, and it is worth doing carefully. + +### 4.1 The trap: `tanshu` is not uid 1000 + +``` +id tanshu → uid=1001(tanshu) gid=1002(tanshu) +``` + +Most container data on this host is owned by **uid 1000**, which is the **`ubuntu`** +user, not you. So "the data is already owned by my user" is false everywhere. Anything +you don't explicitly re-own will be invisible inside a rootless container. + +Measured ownership of the actual bind-mount sources: + +| Container | Runs as (in-container) | Data source | Current host owner | +|---|---|---|---| +| `mozimo`, `mozimo-staging` | 1000 (`node`) | `/var/lib/mozimo/pb_data` | 1000:1000 | +| `hiwikijs` | 1000 (`node`) | *(anon volume)* | 1000:1000 | +| `menu-app.*` (×3) | 1000:1000 | `/var/lib/menu.*` | 1000:1000 | +| `vince` | 1000 | `/var/lib/mozimo/vince_data` | 1000:1000 | +| `yamtrack` | 1000 | `/var/lib/yamtrack/data` | 1000:1000 | +| `trillium` | 1000 (`node`) | `/var/lib/trillium` | 1000:1000 | +| `nextcloud` | 33 (`www-data`) | `/var/lib/nextcloud` | 33:33 | +| `postgres` | **70** | `/var/lib/postgresql/data/pgdata` | 70:0 | +| `valkey` | **999** | *(anon volume)* | 999:1000 | +| `gitea` | 1001 / 1003 (`git`) | `/var/lib/gitea`, `/home/git/.ssh` | 1001:1002, 1003:1003 | +| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | 0 | various | 0:0 | + +> The `postgres` entry is the important one to sanity-check: `docker exec` reports uid 0 +> (that's the image's configured user) but PGDATA is owned by **70**, which is the uid the +> server actually runs as. **Derive the app uid from what the app writes, not from +> `docker exec id`** — the entrypoint in `postgres`, `nextcloud`, `gitea`, `tandoor` +> and `trillium` drops privileges after starting. + +### 4.2 The rule + +Only **one** container uid:gid can be mapped to your host user. Use it for whichever +uid the app actually writes as: + +``` +UserNS=keep-id:uid=,gid= +``` + +| Service | Directive | +|---|---| +| `mozimo`, `mozimo-staging`, `hiwikijs`, `menu-app.*`, `vince`, `yamtrack`, `trillium` | `UserNS=keep-id:uid=1000,gid=1000` | +| `nextcloud` | `UserNS=keep-id:uid=33,gid=33` | +| `postgres` | `UserNS=keep-id:uid=70,gid=70` | +| `valkey` | `UserNS=keep-id:uid=999,gid=999` | +| `gitea` | `UserNS=keep-id:uid=1001,gid=1002` ⚠︎ verify | +| `caddy`, `audiobookshelf`, `affine`, `bitwarden`, `sms-wa-bridge`, `greatbear`, `hops`, `tandoor`, `brewman-*` | `UserNS=keep-id:uid=0,gid=0` | + +**Then chown every data tree to `tanshu:tanshu` (1001:1002).** That is the whole +trick: the app's uid inside the container *is* your uid on the host, so files stay +readable and writable by you, and `tar`/`pg_dump` backups keep working without +`podman unshare` or `sudo`. + +Do **not** use the default rootless mapping (container 0 → tanshu) for the non-root +apps — files would come out owned by subuid `165536+`, invisible to your backup +scripts. `keep-id` is what keeps the host side clean. + +⚠︎ `gitea` writes as two different uids (1001:1002 for `/data`, 1003:1003 for +`/data/git/.ssh`), and `keep-id` maps only one. Resolve this before cutover: either +relax the `.ssh` ownership, or mount `/home/git/.ssh` from a path owned by 1001. + +### 4.3 The chown pass + +Data moves from `/var/lib/X` to `~/podman-data/X`. `/home/tanshu` and `/var/lib` are +both on `/` (`/dev/root`), so **`mv` is an instant rename, not a copy** — verify with +`findmnt -T /var/lib/` that none of them is a separate mount first. + +``` +# after stopping Docker, per service: +mv /var/lib/ ~/podman-data/ +chown -R tanshu:tanshu ~/podman-data/ +``` + +`chown -R` is metadata-only, but audiobookshelf is 15 GB of many small files, so +budget a few minutes for that one. Total data is ~19 GB: + +``` +audiobookshelf 15G | nextcloud 1.2G | postgres 995M | gitea 747M +sms-wa-bridge 397M | mozimo 147M | mozimo-staging 137M | trillium 106M +greatbear 101M | tandoor 98M | menu.* 3×70M | hops 19M | caddy 736K | affine 916K +``` + +Plus two anonymous volumes that must be exported to real directories (they are +invisible to your backup scripts today): + +| Container | Volume | → | +|---|---|---| +| `hiwikijs` | `9c1e3fde…` (owned 1000:1000) | `~/podman-data/hiwikijs/content` | +| `valkey` | `6089e6fc…` (owned 999:1000) | `~/podman-data/valkey` | + +``` +docker run --rm -v 9c1e3fde…:/from -v ~/podman-data/hiwikijs:/to alpine \ + sh -c 'cd /from && cp -a . /to/content' +``` + +--- + +## 5. Target architecture + +### 5.1 Layout + +``` +~/.config/containers/systemd/ + caddy.container + postgres.container + nextcloud.container + nextcloud.network + gitea.network + menu-hopsngrains.network + … one .network per logical app + +~/podman-data/ + caddy/{conf,data,config,site} ← caddyfile_path becomes this + postgres/ nextcloud/ gitea/ audiobookshelf/ … + caddy/conf/Caddyfile ← regenerated by the Ansible caddy roles +``` + +### 5.2 Network naming — a Quadlet quirk to internalise + +A Quadlet file `$NAME.network` produces a Podman network actually called +**`systemd-$NAME`**. So `nextcloud.network` → network `systemd-nextcloud`. + +That is cosmetic for DNS (containers resolve each other by container name and alias), +but it matters for: +- `podman network ls` output — do not be confused by the prefix. +- Any `podman network connect` you run by hand. +- The `.network` file is **only read if some `.container` references it** — networks + are not created standalone. + +A minimal network file is enough; DNS is on by default: + +```ini +# ~/.config/containers/systemd/nextcloud.network +[Network] +``` + +### 5.3 A service unit + +```ini +# ~/.config/containers/systemd/nextcloud.container +[Unit] +Description=Nextcloud +After=network-online.target + +[Container] +ContainerName=nextcloud +Image=docker.io/library/nextcloud:latest +Network=nextcloud.network +UserNS=keep-id:uid=33,gid=33 +Volume=%h/podman-data/nextcloud:/var/www/html +Volume=/etc/localtime:/etc/localtime:ro + +[Service] +Restart=always +TimeoutStartSec=300 + +[Install] +WantedBy=default.target +``` + +`%h` is the systemd specifier for your home directory and works in Quadlet. + +### 5.4 Postgres and valkey: multi-homed with aliases + +`NetworkAlias=` exists in Quadlet 6.0, but it applies the alias to **all** networks +the container joins. Your aliases are per-network (`db` on only 3 of 13), so use the +`Network=` alias suffix instead — Quadlet passes it straight through to podman: + +```ini +# ~/.config/containers/systemd/postgres.container +[Container] +ContainerName=postgres +Image=docker.io/library/postgres:17-alpine +UserNS=keep-id:uid=70,gid=70 +Network=nextcloud.network:alias=db +Network=tandoor.network:alias=db +Network=hiwikijs.network:alias=db +Network=affine.network +Network=bitwarden.network +Network=brewman-acc.network +Network=brewman-exp.network +Network=brewman-hinchco.network +Network=brewman-hops.network +Network=brewman-mhl.network +Volume=%h/podman-data/postgres:/var/lib/postgresql/data +``` + +This is the direct analogue of today's 13-network Postgres hub. `valkey` is the same +pattern with `Network=nextcloud.network:alias=redis` plus `affine`, `yamtrack`. + +**Pin `postgres:17-alpine`.** The running container reports image `postgres:17-alpine` +but its image ID is the one now tagged `postgres:alpine`; the server is 17.5. + +### 5.5 Caddy: multi-homed, exactly as today + +```ini +# ~/.config/containers/systemd/caddy.container +[Unit] +After=network-online.target postgres.service + +[Container] +ContainerName=caddy +Image=docker.io/library/caddy:latest +UserNS=keep-id:uid=0,gid=0 +PublishPort=80:80 +PublishPort=443:443 +PublishPort=443:443/udp +Volume=%h/podman-data/caddy/conf:/etc/caddy +Volume=%h/podman-data/caddy/site:/srv +Volume=%h/podman-data/caddy/data:/data +Volume=%h/podman-data/caddy/config:/config +Network=nextcloud.network +Network=gitea.network +Network=mozimo.network +Network=trillium.network +Network=tandoor.network +Network=hiwikijs.network +Network=bitwarden.network +Network=yamtrack.network +Network=affine.network +Network=audiobookshelf.network +Network=brewman-acc.network +Network=brewman-exp.network +Network=brewman-hinchco.network +Network=brewman-hops.network +Network=brewman-mhl.network +Network=menu-hopsngrains.network +Network=menu-greatbear.network +Network=menu-hngmohali.network +Network=greatbear.network +Network=hops.network +Network=sms-wa-bridge.network + +[Service] +Restart=always + +[Install] +WantedBy=default.target +``` + +The Caddyfile itself needs **no changes** — every upstream is still a container name. +Only `caddyfile_path` in the Ansible vars moves to +`/home/tanshu/podman-data/caddy/conf/Caddyfile`. + +Note `Network=foo.network` in Caddy's unit makes Quadlet order the network units +first, so the "Caddy must be connected after the app exists" fragility of the current +Docker setup disappears entirely. + +Drop these three — they are stale (Caddy holds interfaces for networks with no +service behind them): `iot_net` (the Caddyfile's `iot.tanshu.com → grafana:3000` is +dead), `n8n_net`, `kakawa_net`. Remove that Caddyfile block too. + +--- + +## 6. Enabling pesto — do it before the first container starts + +> **Deferred by default.** See §3.4: this whole section only applies if you decide the +> client-IP regression justifies building Podman 6 + passt. On Ubuntu 26.04 with stock +> Podman 5.7, this option does not exist, and the migration proceeds without it. If you +> *do* enable pesto, do it before any container starts — the switch has no graceful +> upgrade path. + +Pesto is what preserves real client IPs. Per the Podman blog, switching the forwarder +on a running system has **no graceful upgrade path**, so enable it *before* bringing +anything up. + +```ini +# ~/.config/containers/containers.conf (user-level, rootless) +[network] +rootless_port_forwarder = "pasta" +``` + +Traffic path becomes: `client → host:port → pasta → rootless netns → netavark bridge → container`. +Pasta forwards without rewriting the source address, so Caddy sees the real client. + +Two prerequisites: + +1. **`passt` with `pesto`** — your source build from §3. Verify: `which pesto`. +2. **Low ports.** Rootless cannot bind < 1024. Caddy needs 80 and 443: + ``` + sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80 + # persist: + echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/99-podman.conf + ``` + This is a root-level change — unavoidable for rootless on privileged ports. + +**Good news on IPv6:** pasta's main behavioural difference from `rootlessport` is that +it does not transparently forward IPv6 host traffic to IPv4, so services need explicit +IPv6 binding. I checked: monoco has **no global IPv6 address and no default IPv6 +route**, and `mozimo.in`, `git.tanshu.com`, `cloud.hopsngrains.com`, `vault.tanshu.com` +resolve to IPv4 only (no AAAA). **So this limitation does not affect you.** + +**Enable lingering**, or nothing survives logout or reboot: + +``` +sudo loginctl enable-linger tanshu +``` +Currently `Linger=no`. Without this, Quadlet user services stop when your last session +ends — which on a server means immediately. + +--- + +## 7. Cutover sequence + +Pre-flight (no downtime): + +1. Capture ground truth: `docker inspect` for every container/network/volume → JSON. +2. Resolve the `gitea` dual-uid question (§4.2) and confirm each app's true uid. +3. Export the two anonymous volumes (§4.3). +4. **Test restores**, not just backups — at least the Postgres dumps. +5. Back up `/var/lib/caddy/data` explicitly. It holds the ACME account key and all + issued certificates. It is a bind mount so it carries over, but losing it means + re-issuing ~20 certs against Let's Encrypt rate limits. +6. Build and install the Podman 6 stack (§3). `podman login registry.tanshu.com`. +7. Pre-pull every image. For locally-built `sms-wa-bridge:latest` (no registry prefix) + avoid a re-download: `docker save sms-wa-bridge:latest | podman load`. +8. Write and review all Quadlet units; `systemctl --user daemon-reload` and check + `systemctl --user list-unit-files | grep -E 'caddy|nextcloud'`. +9. **Take a provider snapshot.** Cheapest, fastest rollback you will ever buy. +10. Rehearse on a clone of the host, including a reboot. + +Window (expect 10–20 minutes; 26 containers plus a 15 GB `chown`): + +``` +# 1. Stop the edge first so no request is half-served +docker stop caddy +docker stop $(docker ps -q) + +# 2. Freeze state; nothing may write to /var/lib during the move +sudo systemctl stop docker.socket docker.service containerd.service +sudo systemctl disable docker.service + +# 3. Move + re-own data (instant rename, then a metadata walk) +mv /var/lib/ ~/podman-data/ && chown -R tanshu:tanshu ~/podman-data/ + +# 4. Bring up the data tier first +systemctl --user start postgres.service valkey.service + +# 5. Verify DNS inside the namespace BEFORE starting Caddy +podman exec nextcloud getent hosts postgres +podman exec nextcloud getent hosts redis + +# 6. App tier, then Caddy last +systemctl --user start +systemctl --user start caddy.service + +# 7. Reload + smoke test +podman exec caddy caddy reload +``` + +**Do not `docker rm` anything.** Stopped containers are the fastest rollback and cost +only disk. Keep them a week. + +--- + +## 8. What must change in this repo and on the host + +**(a) Legacy `--link` in the backup scripts — a guaranteed break.** +`db-backup.sh`, `nextcloud-backup.sh`, `hiwikijs-`, `tandoor-`, `kakawa-`, +`bitwarden-backup.sh` and their `-restore.sh` counterparts all do +`docker run --rm --link postgres:db … pg_dump -h db`. `--link` is Docker-only and is +absent from `podman run`. Under rootless there is no default `bridge` network to fall +back on either. + +Best fix — drop the helper container and dump in place (no networking, no `--user` +juggling, no shared network): + +```bash +podman exec -e PGPASSWORD="..." postgres \ + pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME" +``` + +Also pin the base image: the scripts mix `postgres:alpine` and `postgres:17-alpine`. + +Bonus: once data is owned by `tanshu` and reachable without `sudo`, the `sudo tar` and +`chown $(logname)` gymnastics in `nextcloud-backup.sh` can go away. + +**(b) Root's crontab cannot reach rootless containers.** +``` +*/5 * * * * docker exec -u www-data nextcloud php cron.php +``` +Root cannot `podman exec` into another user's rootless containers. Move this to a +**systemd user timer** alongside the Quadlet units — which is the idiomatic approach +anyway. Same for `~/bin/empty-trash.sh`, which also targets a container named `cloud` +that no longer exists (it is `nextcloud`). + +**(c) Ansible modules.** `docker_container` / `docker_image` / `docker_network` become +`containers.podman`'s `podman_container` / `podman_image` / `podman_network`, which +shell out to the `podman` CLI and need **no Python SDK** — the `pip: name=docker` task +disappears. Note `podman_network` lacks `docker_network`'s `connected:`/`appends:` +parameters, which the `*/roles/network/tasks/main.yaml` files rely on. Prefer declaring +the full `network:` list on each `podman_container` and deleting those network roles — +that is precisely what Quadlet does, so the two stay in sync. + +Also update `caddyfile_path` everywhere (now `/home/tanshu/podman-data/caddy/conf/Caddyfile`). + +**(d) `~/.zshrc:128`** and any other `docker ` invocations in `~/bin`, `/etc/cron*`, +and the **external deploy pipelines** for `mozimo`, `brewman`, `menu-admin`, `hops`, +`greatbear`, `sms-wa-bridge`. Those live outside this repo. If they are not updated, +the next deploy will recreate Docker containers next to your Podman ones. + +**(e) `postgres` currently publishes `5432` on `0.0.0.0`** and UFW is inactive. Fix +while you are in here — it needs no published port at all, only `expose`/network access. + +--- + +## 9. Rollback + +1. **`docker start`** — old containers exist, stopped but not removed; bind mounts + untouched. Reverting is `systemctl start docker` + `docker start $(docker ps -aq)`. + Valid as long as you do not `docker rm`. ⚠︎ This only works if no app performed an + irreversible migration on first start under Podman (Postgres major upgrade, + Nextcloud schema migration, Trillium migration). Check that risk per service, and + prefer the snapshot over trusting this. +2. **Provider snapshot** — instant whole-host revert. Covers `/var/lib` damage and + certificate loss. +3. **Backups** — once §8(a) is fixed and restores are verified. + +Note that the `chown` pass is itself hard to unwind precisely: reverting to Docker +requires the original ownership (0:0, 33:33, 70:0, 1000:1000). Record the table in +§4.1 before you start, and re-`chown` on rollback. + +--- + +## 10. Verification checklist + +**Linkage** +- [ ] `podman exec caddy getent hosts ` resolves for all 20 upstreams +- [ ] `getent hosts db` and `getent hosts redis` resolve from the apps that use them +- [ ] Every container is on the same set of networks it had under Docker + +**Rootless plumbing** +- [ ] `loginctl show-user tanshu | grep Linger` → `Linger=yes` +- [ ] `which pesto` returns a path; `rootless_port_forwarder` is set **before** first start +- [ ] Caddy sees **real client IPs** — check `podman logs caddy` for a genuine remote IP +- [ ] `X-Forwarded-For` reaches Nextcloud/Gitea correctly + +**HTTP / TLS** +- [ ] `curl -sSI https://` 200/redirect for all 20 site blocks +- [ ] Certificates are the **existing** ones (compare serial against the pre-migration + snapshot) — nothing re-issued +- [ ] WebSocket / streaming paths (Gitea, Nextcloud, RustDesk) + +**Applications** +- [ ] Each app can read **and write** — catches the uid mapping (§4.2) +- [ ] Postgres accepts connections from every client +- [ ] `postgres` is no longer published on `0.0.0.0:5432` + +**Operations** +- [ ] Nextcloud cron fires via the new systemd user timer +- [ ] A full backup cycle of every `~/bin/*-backup.sh` completes and restores +- [ ] **Reboot the host**; all containers return with correct policies +- [ ] `systemctl --user status caddy` is clean; `podman exec caddy caddy reload` works +- [ ] Disk is acceptable after Docker images are pruned (32 GB reclaimable) + +--- + +## 11. Risk register + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| `--link` backup scripts break silently | **Certain** | Backups stop unnoticed | Fix §8(a) before the window; verify a full cycle | +| Data invisible due to uid mismatch | **High** | Apps fail to start or write | `keep-id` per §4.2 + `chown -R tanshu`; test each app's write path | +| Compiling on monoco OOMs production | **High if done on-box** | Outage during build | Build off-box (§3.2) | +| Containers do not return after reboot | High if `linger` skipped | Long unnoticed outage | `enable-linger` + explicit reboot test | +| pesto is experimental / breaks | Medium | Wrong client IPs, or Caddy unreachable | Snapshot first; know how to fall back to `rootlessport` | +| Switching forwarder with containers running | Medium | Corrupt port-forward state | Enable pesto **before** first start (§6) | +| Low-port sysctl not set | Medium | Caddy cannot bind 80/443 | §6; verify after reboot | +| Let's Encrypt rate limit | Low | Certs unavailable for a week | Preserve `/var/lib/caddy/data`; verify certs load, don't re-issue | +| External deploy pipelines recreate Docker containers | **High** | Duplicate apps, port conflicts | Audit and update CI (§8(d)) | +| Self-built stack drifts from security fixes | **Certain over time** | Unpatched engine on 20 sites | Rebuild pipeline + calendar reminder; or reconsider §2 | +| Two runtimes writing the same data | Medium | Database corruption | Hard-stop Docker before starting Podman | + +--- + +## 12. Open questions + +1. **Are we upgrading to Ubuntu 26.04 as part of this?** §3.4 argues yes, and that it + should happen **before** the Podman migration. This decision changes the rest of the + plan: on 26.04 there are **no source builds**, and §3 in its entirety becomes + optional/deferred. +2. **Where do `mozimo`, `brewman`, `menu-admin`, `hops`, `greatbear`, `sms-wa-bridge` + deploy from?** If those pipelines run `docker` against monoco they must change in + lockstep. Should they move into this repo? +3. **Is a temporary client-IP regression acceptable?** Recommended: yes — migrate on + stock Podman, accept gateway IPs at Caddy, and treat Podman 6 + pesto as a separate + later project. The Quadlet units do not change, so it is a drop-in upgrade. +4. **`postgres`'s `0.0.0.0:5432` exposure** — fold into this work or handle separately? + +Resolved during the audit: + +- ~~**`gitea`'s dual uid**~~ — the container sets `USER_UID=1003`/`USER_GID=1003`, and + `git` is uid 1003 on the host (`/home/git/.ssh` is `1003:1003`). The real data lives + in `/var/lib/gitea/gitea/` owned `git:git`. Use **`UserNS=keep-id:uid=1003,gid=1003`**. +- ~~**Build host / packaging**~~ — moot if §3.4 is taken; revisit only if the pesto + project proceeds.