playbooks/postgres/migrate-from-monoco.md

Moving the cluster from monoco (Postgres 17 → 18)

Status: done — do not re-run against the live cluster. Executed 2026-10-15, Postgres 17.5 → 18.6. Verified afterwards: 12 databases restored with per-database table counts identical to monoco, 13 databases and 3 roles (postgres, oc_tanshu, zitadel), btree_gist/pg_trgm/unaccent present, no listener on host 5432, and every directory in the data root owned by tanshu rather than a subuid.

The steps below are kept as the record of how it was done, and for the next major-version move. Running them again would now fail on the first CREATE DATABASE and leave a half-loaded cluster — and the "start over" block in step 5 destroys ~/podman-data/postgres/18.

One-time move of every database from monoco's Docker Postgres to emily's rootless Quadlet Postgres, across a major-version upgrade. Run the commands from the machine that holds the monoco and emily SSH aliases.

Why this is a dump and not a copy

Two independent reasons, either of which is decisive:

• 17 data files cannot be read by an 18 server. A copy of PGDATA produces a cluster that 18 refuses to open — it would have to go through pg_upgrade with both versions present.
• The 18 image moved PGDATA. In 17 it was /var/lib/postgresql/data/pgdata; in 18 it is /var/lib/postgresql/18/docker, and the entrypoint deliberately refuses to start when it finds a cluster in the old location, to stop an image bump from silently looking like data loss.

A logical dump sidesteps both, and at this size it is also the fastest thing to get right. pg_upgrade would avoid re-importing, but it needs the 17 data directory copied to emily first and a purpose-built image with both versions (tianon/postgres-upgrade:17-to-18), for a cluster of well under a gigabyte. Not worth the extra failure modes.

What is being moved (measured on monoco)

Source: Docker container postgres, postgres:17-alpine, server 17.5, PGDATA=/var/lib/postgresql/data/pgdata, ~995 MB on disk.

Database	Size	Database	Size
nextcloud	320 MB	hiwikijs	16 MB
brewman_acc	174 MB	tandoor	16 MB
brewman_mhl	132 MB	brewman_exp	16 MB
brewman_hops	122 MB	affine	11 MB
brewman_hinchco	41 MB	bitwarden	10 MB
medusa-criollo	21 MB	zitadel	9 MB

Roles: postgres (superuser), oc_tanshu (creates databases), zitadel. No custom tablespaces. template1 is stock; the postgres database has no user tables. medusa-criollo and zitadel have no running container — nothing writes them — so they come along for completeness.

Order of operations

1. Deploy the 18 cluster on emily (postgres/playbook.yml) and leave it empty.
2. Dump on monoco.
3. Transfer.
4. Restore into emily.
5. Verify, then point the app playbooks at it.

The target must be empty before the restore. psql will happily skip a CREATE DATABASE for a database that already exists, so a second restore into a populated cluster is not an error but is also not a clean overwrite.

1. Pre-flight

The 18 cluster has to exist before any of this. If it does not, deploy it:

cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml

Then check it is up and empty:

# The 18 cluster exists, is running, and holds nothing but the defaults
ssh emily 'systemctl --user is-active postgres.service
           podman exec postgres psql -U postgres -tAc "select datname from pg_database order by 1"'
# expect: active, then just: postgres, template0, template1

# Room for the dump plus the restored data (~2 GB)
ssh emily 'df -h / | tail -1'
ssh monoco 'df -h / | tail -1'

If that says inactive, or podman exec answers no container with name or ID "postgres" found, the unit has not been deployed on emily — stop and do that first. The restore in step 5 would otherwise pipe the whole dump into a non-existent container and leave almost no trace.

Optional: quiesce the writers

pg_dumpall takes one snapshot per database, so each database is internally consistent but the cluster is not a single point in time. Every app here owns its own database and there are no cross-database references, so a live dump is fine in practice. If you want zero doubt, stop the writers for the duration:

WRITERS='nextcloud gitea affine bitwarden hiwikijs tandoor
         brewman-acc brewman-hops brewman-mhl brewman-exp brewman-hinchco'

ssh monoco "docker stop $WRITERS"     # before the dump
ssh monoco "docker start $WRITERS"    # after step 3

That is real downtime for those sites, so take it or leave it deliberately.

2. Dump on monoco

ssh monoco 'set -euo pipefail
  mkdir -p ~/pg-migrate && cd ~/pg-migrate
  TS=$(date +%Y%m%d-%H%M)
  docker exec -e PGPASSWORD=123456 postgres pg_dumpall -U postgres \
    | zstd -T0 -6 -o "pg17-cluster-$TS.sql.zst"
  zstd -t "pg17-cluster-$TS.sql.zst"
  sha256sum "pg17-cluster-$TS.sql.zst" | tee "pg17-cluster-$TS.sql.zst.sha256"
  ls -lh'

It prints nothing while it runs — pg_dump buffers, so there is no progress bar — and takes a couple of minutes for ~900 MB. zstd -t is an integrity check; there is no point transferring a truncated dump.

pg_dumpall is used rather than per-database pg_dump because it carries the roles and template1 in the same stream, and this cluster has three roles that would otherwise have to be recreated by hand.

3. Verify the dump before trusting it

ssh monoco 'cd ~/pg-migrate && F=$(ls -t pg17-cluster-*.sql.zst | head -1)
  echo "file: $F"
  echo "databases connected : $(zstd -dc "$F" | grep -cE "^\\\\connect")"        # expect 14
  echo "databases created   : $(zstd -dc "$F" | grep -cE "^CREATE DATABASE")"    # expect 12
  echo "role statements     : $(zstd -dc "$F" | grep -cE "^(CREATE|ALTER) ROLE")" # expect 6
  zstd -dc "$F" | head -5'

The counts are the shape of this particular cluster: 12 databases get a CREATE DATABASE, and postgres + template1 are dumped in place (14 \connect lines). The 6 role statements are 3 roles × (CREATE + ALTER).

Also check which extensions the dump creates, because step 5 runs with ON_ERROR_STOP=1 and a missing extension aborts the whole restore:

ssh emily 'cd ~/pg-migrate && zstd -dc pg17-cluster.sql.zst | grep -E "^CREATE EXTENSION" | sort | uniq -c'

This cluster needs btree_gist (×5), pg_trgm and unaccent — all contrib modules, and the official image builds contrib in (make -C contrib install), so they are present. An extension that is not contrib — pgvector, postgis, timescaledb — would not be, and would have to be installed into the image before the restore can succeed. (docs/podman-compat-audit.md notes affine's indexer is disabled, which is why there is no pgvector here.)

4. Transfer to emily

# stream it straight across; no copy on the workstation and no second hop
ssh monoco 'cat ~/pg-migrate/pg17-cluster-*.sql.zst' \
  | ssh emily 'mkdir -p ~/pg-migrate && cat > ~/pg-migrate/pg17-cluster.sql.zst'

Then gate on the checksum — a truncated dump restores into a partial cluster that looks fine until an app reads a missing table:

a=$(ssh monoco 'sha256sum ~/pg-migrate/pg17-cluster-*.sql.zst | cut -d" " -f1')
b=$(ssh emily  'sha256sum ~/pg-migrate/pg17-cluster.sql.zst | cut -d" " -f1')
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
  || { echo "MISMATCH — do not restore: $a vs $b"; false; }

rsync cannot copy remote-to-remote directly, so the pipe above is the shortest path. If the link is slow or drops, do it in two hops instead and check the checksum on both sides:

rsync -avP monoco:~/pg-migrate/ ~/pg-migrate/
rsync -avP ~/pg-migrate/ emily:~/pg-migrate/

5. Restore into the 18 cluster on emily

ssh emily 'cd ~/pg-migrate
  # Fail here, loudly, if the unit was never deployed. Without this, the
  # pipeline below feeds the dump to a podman exec that cannot find a container
  # and the only trace is a line in restore.err that a grep for ERROR misses.
  podman exec postgres pg_isready -U postgres -d postgres \
    || { echo "FAILED: postgres is not running on emily — deploy it first (step 1)."; exit 1; }

  set -o pipefail
  zstd -dc pg17-cluster.sql.zst \
    | grep -v "^CREATE ROLE postgres;$" \
    | podman exec -i -e PGPASSWORD=123456 postgres \
        psql -U postgres -d postgres -v ON_ERROR_STOP=1 -q 2> restore.err \
    || { echo "FAILED: restore stopped on an error:"; tail -20 restore.err; exit 1; }

  echo "OK — restore clean. Databases now:"
  podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
    "select count(*) from pg_database where datallowconn and not datistemplate"'

Expect 13 — the postgres database plus the 12 restored ones. That count is the positive proof the restore happened; an empty error log on its own proves nothing, which is exactly how a missing container slipped through the first time.

Three details are doing real work here:

• pg_isready first. podman exec on a missing container writes Error: no container with name or ID "postgres" found and exits 1 — but the pipeline still "completes", and that message is invisible to a case-sensitive grep ERROR. This is the check that turns that into an immediate, obvious stop.
• grep -v '^CREATE ROLE postgres;$'. That is the one statement expected to fail: pg_dumpall emits it and initdb has already created the role. The ALTER ROLE postgres ... PASSWORD that follows is kept and succeeds, so the superuser ends up with monoco's password hash — the same 123456 anyway. Removing that exact line, rather than tolerating the error, is what allows ON_ERROR_STOP=1: any unexpected error now aborts the restore and is printed instead of being skipped.
• The database count at the end, as above.

The dump was made by pg_dump 17.5 and is replayed by psql 18: forward restores are the supported direction, and source and target are both the Alpine/musl image, so the en_US.utf8 locale the databases were created with resolves on both sides.

If the restore does stop, fix the cause and start over — ON_ERROR_STOP=1 can leave the cluster half-loaded, and re-running on top of that adds "already exists" noise rather than converging.

# destroys everything in the emily cluster
ssh emily 'systemctl --user stop postgres.service
           rm -rf ~/podman-data/postgres/18'
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml

That rm -rf needs no podman unshare precisely because the role creates the version directory as tanshu rather than letting the container strand it as a subuid-owned directory. If it ever fails with permission denied, that is the assumption breaking — check ls -ln ~/podman-data/postgres/18.

6. Verify

# server version, and the databases with their sizes — compare against the table above
ssh emily 'podman exec postgres postgres --version'
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
  "select datname, pg_size_pretty(pg_database_size(datname)) from pg_database order by pg_database_size(datname) desc"'

# all three roles made it
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -c "\du"'

Then the check that actually matters — the same per-database table count on both sides. Feed the loop to the remote shell over stdin so the quoting stays sane:

ssh monoco 'bash -s' <<'EOF'
EXEC="docker exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
  printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF

ssh emily 'bash -s' <<'EOF'
EXEC="podman exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
  printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF

Save both outputs and diff them — identical table counts per database is the signal that the restore was complete. Sizes will differ slightly and that is normal (toast/fillfactor are not byte-reproducible).

Also confirm ownership is what the playbook promises, because a wrong keep-id shows up here and nowhere else:

ssh emily 'ls -ln ~/podman-data/postgres/18/docker | head'
# PGDATA and its contents must be owned by tanshu:tanshu, not a 165536+ subuid

7. Afterwards

• Keep monoco's cluster. Do not docker rm postgres and do not drop databases: a stopped, intact 17 cluster is the fastest rollback there is. Leave the dump in ~/pg-migrate on both hosts for the same reason.
• Stop the old writers before the new ones start if both hosts could serve the same app, or you will have two databases diverging behind the same site.
• Apps connect unchanged. pg_dumpall restores each database under its original name with its original role grants, and the app playbooks' postgres_container: postgres resolves over the attached app networks — same container name, same db alias, same password.
• The dump is not a backup regime. It is a migration artifact. Per-database pg_dump on a schedule is still owed (see the Backups section of readme.md).

Rollback

There is nothing to reverse on emily beyond stopping the new container: systemctl --user stop postgres.service. monoco was never modified — its container was only read — so rolling back is starting the old stack again.

The one exception is the quiesce step, if you took it: docker start the WRITERS list above.