tanshu/playbooks
1
0 Fork 0
Code Issues Pull Requests Releases Wiki Activity

Run Postgres as a rootless Podman Quadlet, and move the 17 cluster across

Browse Source 
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept
its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same
shape as Caddy: it knows nothing about its clients, and a service attaches itself
by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing
here changes when a service is added or removed.

Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both
the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its
default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker),
and the 18 entrypoint refuses to start when it finds a cluster in the old layout.
So postgres_major is the single knob: it picks the image tag and the PGDATA path
together, and the mount is now the parent directory. That also settles the
migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster
comes across as a pg_dumpall.

Three things that bit us and are worth keeping:

- PGDATA and its version directory are created by the role, not left to the
  container. The entrypoint runs as container-root, which under keep-id is a
  subuid on the host, and it only chowns PGDATA and its contents -- so
  ~/podman-data/postgres/18 would have been stranded subuid-owned, and the host
  user could not even rm -rf it without podman unshare.

- NetworkAlias= applies to every network the container joins, which is what a
  database hub wants, but it is one fixed value in a shared drop-in rather than
  per-service: two services writing their own would make Quadlet pass
  --network-alias twice. It is not in postgres.container itself because on a
  fresh host that unit has no Network= at all.

- The credentials live in a 0600 file read with EnvironmentFile=, not in
  Environment= lines: the unit is world-readable in ~/.config.

The play no longer uses become, and does not apply the rootless-Podman
prerequisites -- those are the shared task file at caddy/roles/podman, and the
three variables they need are duplicated at the top of vars/default.yml.

migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer
and restore commands, the checks that prove the restore is complete (per-database
table counts against the source), and the verified result.

The dead nginx/php-fpm templates in postgres/files/ are removed.
This commit is contained in:
tanshu committed 2026-10-16 06:52:27 +05:30
1 parent 060173db5e
commit 4297e5a71e
11 files changed
+1079 -88
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

postgres/migrate-from-monoco.md
+315
View File
@@ -0,0 +1,315 @@
# Moving the cluster from monoco (Postgres 17 → 18)
> **Status: done — do not re-run against the live cluster.** Executed
> 2026-10-15, Postgres 17.5 → 18.6. Verified afterwards: 12 databases restored
> with per-database table counts *identical* to monoco, 13 databases and 3 roles
> (`postgres`, `oc_tanshu`, `zitadel`), `btree_gist`/`pg_trgm`/`unaccent` present,
> no listener on host `5432`, and every directory in the data root owned by
> `tanshu` rather than a subuid.
>
> The steps below are kept as the record of how it was done, and for the next
> major-version move. Running them again would now fail on the first
> `CREATE DATABASE` and leave a half-loaded cluster — and the "start over" block
> in step 5 *destroys* `~/podman-data/postgres/18`.
One-time move of every database from `monoco`'s Docker Postgres to `emily`'s
rootless Quadlet Postgres, across a major-version upgrade. Run the commands from
the machine that holds the `monoco` and `emily` SSH aliases.
## Why this is a dump and not a copy
Two independent reasons, either of which is decisive:
- **17 data files cannot be read by an 18 server.** A copy of `PGDATA` produces a
cluster that 18 refuses to open — it would have to go through `pg_upgrade` with
both versions present.
- **The 18 image moved PGDATA.** In 17 it was `/var/lib/postgresql/data/pgdata`; in
18 it is `/var/lib/postgresql/18/docker`, and the entrypoint deliberately
refuses to start when it finds a cluster in the old location, to stop an image
bump from silently looking like data loss.
A logical dump sidesteps both, and at this size it is also the fastest thing to
get right. `pg_upgrade` would avoid re-importing, but it needs the 17 data
directory copied to `emily` first *and* a purpose-built image with both versions
(`tianon/postgres-upgrade:17-to-18`), for a cluster of well under a gigabyte. Not
worth the extra failure modes.
## What is being moved (measured on monoco)
Source: Docker container `postgres`, `postgres:17-alpine`, server **17.5**,
`PGDATA=/var/lib/postgresql/data/pgdata`, ~995 MB on disk.
| Database | Size | Database | Size |
|---|---|---|---|
| `nextcloud` | 320 MB | `hiwikijs` | 16 MB |
| `brewman_acc` | 174 MB | `tandoor` | 16 MB |
| `brewman_mhl` | 132 MB | `brewman_exp` | 16 MB |
| `brewman_hops` | 122 MB | `affine` | 11 MB |
| `brewman_hinchco` | 41 MB | `bitwarden` | 10 MB |
| `medusa-criollo` | 21 MB | `zitadel` | 9 MB |
Roles: `postgres` (superuser), `oc_tanshu` (creates databases), `zitadel`.
No custom tablespaces. `template1` is stock; the `postgres` database has no user
tables. `medusa-criollo` and `zitadel` have no running container — nothing writes
them — so they come along for completeness.
## Order of operations
1. Deploy the 18 cluster on `emily` (`postgres/playbook.yml`) and leave it **empty**.
2. Dump on `monoco`.
3. Transfer.
4. Restore into `emily`.
5. Verify, then point the app playbooks at it.
The target must be empty before the restore. `psql` will happily skip a
`CREATE DATABASE` for a database that already exists, so a second restore into a
populated cluster is not an error but is also not a clean overwrite.
## 1. Pre-flight
**The 18 cluster has to exist before any of this.** If it does not, deploy it:
```bash
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
```
Then check it is up and empty:
```bash
# The 18 cluster exists, is running, and holds nothing but the defaults
ssh emily 'systemctl --user is-active postgres.service
podman exec postgres psql -U postgres -tAc "select datname from pg_database order by 1"'
# expect: active, then just: postgres, template0, template1
# Room for the dump plus the restored data (~2 GB)
ssh emily 'df -h / | tail -1'
ssh monoco 'df -h / | tail -1'
```
If that says `inactive`, or `podman exec` answers `no container with name or ID
"postgres" found`, the unit has not been deployed on `emily` — stop and do that
first. The restore in step 5 would otherwise pipe the whole dump into a
non-existent container and leave almost no trace.
### Optional: quiesce the writers
`pg_dumpall` takes one snapshot per database, so each database is internally
consistent but the cluster is not a single point in time. Every app here owns its
own database and there are no cross-database references, so a live dump is fine
in practice. If you want zero doubt, stop the writers for the duration:
```bash
WRITERS='nextcloud gitea affine bitwarden hiwikijs tandoor
brewman-acc brewman-hops brewman-mhl brewman-exp brewman-hinchco'
ssh monoco "docker stop $WRITERS" # before the dump
ssh monoco "docker start $WRITERS" # after step 3
```
That is real downtime for those sites, so take it or leave it deliberately.
## 2. Dump on monoco
```bash
ssh monoco 'set -euo pipefail
mkdir -p ~/pg-migrate && cd ~/pg-migrate
TS=$(date +%Y%m%d-%H%M)
docker exec -e PGPASSWORD=123456 postgres pg_dumpall -U postgres \
| zstd -T0 -6 -o "pg17-cluster-$TS.sql.zst"
zstd -t "pg17-cluster-$TS.sql.zst"
sha256sum "pg17-cluster-$TS.sql.zst" | tee "pg17-cluster-$TS.sql.zst.sha256"
ls -lh'
```
It prints nothing while it runs — `pg_dump` buffers, so there is no progress bar
— and takes a couple of minutes for ~900 MB. `zstd -t` is an integrity check;
there is no point transferring a truncated dump.
`pg_dumpall` is used rather than per-database `pg_dump` because it carries the
roles and `template1` in the same stream, and this cluster has three roles that
would otherwise have to be recreated by hand.
## 3. Verify the dump before trusting it
```bash
ssh monoco 'cd ~/pg-migrate && F=$(ls -t pg17-cluster-*.sql.zst | head -1)
echo "file: $F"
echo "databases connected : $(zstd -dc "$F" | grep -cE "^\\\\connect")" # expect 14
echo "databases created : $(zstd -dc "$F" | grep -cE "^CREATE DATABASE")" # expect 12
echo "role statements : $(zstd -dc "$F" | grep -cE "^(CREATE|ALTER) ROLE")" # expect 6
zstd -dc "$F" | head -5'
```
The counts are the shape of this particular cluster: 12 databases get a
`CREATE DATABASE`, and `postgres` + `template1` are dumped in place (14
`\connect` lines). The 6 role statements are 3 roles × (`CREATE` + `ALTER`).
Also check which extensions the dump creates, because step 5 runs with
`ON_ERROR_STOP=1` and a missing extension aborts the whole restore:
```bash
ssh emily 'cd ~/pg-migrate && zstd -dc pg17-cluster.sql.zst | grep -E "^CREATE EXTENSION" | sort | uniq -c'
```
This cluster needs `btree_gist` (×5), `pg_trgm` and `unaccent` — all contrib
modules, and the official image builds contrib in
(`make -C contrib install`), so they are present. An extension that is *not*
contrib — `pgvector`, `postgis`, `timescaledb` — would not be, and would have to
be installed into the image before the restore can succeed. (`docs/podman-compat-audit.md`
notes affine's indexer is disabled, which is why there is no `pgvector` here.)
## 4. Transfer to emily
```bash
# stream it straight across; no copy on the workstation and no second hop
ssh monoco 'cat ~/pg-migrate/pg17-cluster-*.sql.zst' \
| ssh emily 'mkdir -p ~/pg-migrate && cat > ~/pg-migrate/pg17-cluster.sql.zst'
```
Then gate on the checksum — a truncated dump restores into a partial cluster that
looks fine until an app reads a missing table:
```bash
a=$(ssh monoco 'sha256sum ~/pg-migrate/pg17-cluster-*.sql.zst | cut -d" " -f1')
b=$(ssh emily 'sha256sum ~/pg-migrate/pg17-cluster.sql.zst | cut -d" " -f1')
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
|| { echo "MISMATCH — do not restore: $a vs $b"; false; }
```
`rsync` cannot copy remote-to-remote directly, so the pipe above is the shortest
path. If the link is slow or drops, do it in two hops instead and check the
checksum on both sides:
```bash
rsync -avP monoco:~/pg-migrate/ ~/pg-migrate/
rsync -avP ~/pg-migrate/ emily:~/pg-migrate/
```
## 5. Restore into the 18 cluster on emily
```bash
ssh emily 'cd ~/pg-migrate
# Fail here, loudly, if the unit was never deployed. Without this, the
# pipeline below feeds the dump to a podman exec that cannot find a container
# and the only trace is a line in restore.err that a grep for ERROR misses.
podman exec postgres pg_isready -U postgres -d postgres \
|| { echo "FAILED: postgres is not running on emily — deploy it first (step 1)."; exit 1; }
set -o pipefail
zstd -dc pg17-cluster.sql.zst \
| grep -v "^CREATE ROLE postgres;$" \
| podman exec -i -e PGPASSWORD=123456 postgres \
psql -U postgres -d postgres -v ON_ERROR_STOP=1 -q 2> restore.err \
|| { echo "FAILED: restore stopped on an error:"; tail -20 restore.err; exit 1; }
echo "OK — restore clean. Databases now:"
podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
"select count(*) from pg_database where datallowconn and not datistemplate"'
```
**Expect `13`** — the `postgres` database plus the 12 restored ones. That count is
the positive proof the restore happened; an empty error log on its own proves
nothing, which is exactly how a missing container slipped through the first time.
Three details are doing real work here:
- **`pg_isready` first.** `podman exec` on a missing container writes
`Error: no container with name or ID "postgres" found` and exits 1 — but the
pipeline still "completes", and that message is invisible to a case-sensitive
`grep ERROR`. This is the check that turns that into an immediate, obvious stop.
- **`grep -v '^CREATE ROLE postgres;$'`.** That is the one statement expected to
fail: `pg_dumpall` emits it and initdb has already created the role. The
`ALTER ROLE postgres ... PASSWORD` that follows is kept and succeeds, so the
superuser ends up with monoco's password hash — the same `123456` anyway.
Removing that exact line, rather than tolerating the error, is what allows
`ON_ERROR_STOP=1`: any *unexpected* error now aborts the restore and is printed
instead of being skipped.
- **The database count at the end**, as above.
The dump was made by `pg_dump` 17.5 and is replayed by `psql` 18: forward
restores are the supported direction, and source and target are both the
Alpine/musl image, so the `en_US.utf8` locale the databases were created with
resolves on both sides.
If the restore does stop, fix the cause and start over — `ON_ERROR_STOP=1` can
leave the cluster half-loaded, and re-running on top of that adds "already
exists" noise rather than converging.
```bash
# destroys everything in the emily cluster
ssh emily 'systemctl --user stop postgres.service
rm -rf ~/podman-data/postgres/18'
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
```
That `rm -rf` needs no `podman unshare` precisely because the role creates the
version directory as `tanshu` rather than letting the container strand it as a
subuid-owned directory. If it ever fails with permission denied, that is the
assumption breaking — check `ls -ln ~/podman-data/postgres/18`.
## 6. Verify
```bash
# server version, and the databases with their sizes — compare against the table above
ssh emily 'podman exec postgres postgres --version'
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
"select datname, pg_size_pretty(pg_database_size(datname)) from pg_database order by pg_database_size(datname) desc"'
# all three roles made it
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -c "\du"'
```
Then the check that actually matters — the same per-database table count on both
sides. Feed the loop to the remote shell over stdin so the quoting stays sane:
```bash
ssh monoco 'bash -s' <<'EOF'
EXEC="docker exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF
ssh emily 'bash -s' <<'EOF'
EXEC="podman exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF
```
Save both outputs and diff them — identical table counts per database is the
signal that the restore was complete. Sizes will differ slightly and that is
normal (toast/fillfactor are not byte-reproducible).
Also confirm ownership is what the playbook promises, because a wrong `keep-id`
shows up here and nowhere else:
```bash
ssh emily 'ls -ln ~/podman-data/postgres/18/docker | head'
# PGDATA and its contents must be owned by tanshu:tanshu, not a 165536+ subuid
```
## 7. Afterwards
- **Keep monoco's cluster.** Do not `docker rm postgres` and do not drop
databases: a stopped, intact 17 cluster is the fastest rollback there is.
Leave the dump in `~/pg-migrate` on both hosts for the same reason.
- **Stop the old writers before the new ones start** if both hosts could serve
the same app, or you will have two databases diverging behind the same site.
- **Apps connect unchanged.** `pg_dumpall` restores each database under its
original name with its original role grants, and the app playbooks'
`postgres_container: postgres` resolves over the attached app networks — same
container name, same `db` alias, same password.
- **The dump is not a backup regime.** It is a migration artifact. Per-database
`pg_dump` on a schedule is still owed (see the Backups section of
[readme.md](readme.md)).
## Rollback
There is nothing to reverse on `emily` beyond stopping the new container:
`systemctl --user stop postgres.service`. `monoco` was never modified — its
container was only read — so rolling back is starting the old stack again.
The one exception is the quiesce step, if you took it: `docker start` the
`WRITERS` list above.