playbooks/postgres/migrate-from-monoco.md

# Moving the cluster from monoco (Postgres 17 → 18)

> **Status: done — do not re-run against the live cluster.** Executed
> 2026-10-15, Postgres 17.5 → 18.6. Verified afterwards: 12 databases restored
> with per-database table counts *identical* to monoco, 13 databases and 3 roles
> (`postgres`, `oc_tanshu`, `zitadel`), `btree_gist`/`pg_trgm`/`unaccent` present,
> no listener on host `5432`, and every directory in the data root owned by
> `tanshu` rather than a subuid.
>
> The steps below are kept as the record of how it was done, and for the next
> major-version move. Running them again would now fail on the first
> `CREATE DATABASE` and leave a half-loaded cluster — and the "start over" block
> in step 5 *destroys* `~/podman-data/postgres/18`.

One-time move of every database from `monoco`'s Docker Postgres to `emily`'s
rootless Quadlet Postgres, across a major-version upgrade. Run the commands from
the machine that holds the `monoco` and `emily` SSH aliases.

## Why this is a dump and not a copy

Two independent reasons, either of which is decisive:

- **17 data files cannot be read by an 18 server.** A copy of `PGDATA` produces a
  cluster that 18 refuses to open — it would have to go through `pg_upgrade` with
  both versions present.
- **The 18 image moved PGDATA.** In 17 it was `/var/lib/postgresql/data/pgdata`; in
  18 it is `/var/lib/postgresql/18/docker`, and the entrypoint deliberately
  refuses to start when it finds a cluster in the old location, to stop an image
  bump from silently looking like data loss.

A logical dump sidesteps both, and at this size it is also the fastest thing to
get right. `pg_upgrade` would avoid re-importing, but it needs the 17 data
directory copied to `emily` first *and* a purpose-built image with both versions
(`tianon/postgres-upgrade:17-to-18`), for a cluster of well under a gigabyte. Not
worth the extra failure modes.

## What is being moved (measured on monoco)

Source: Docker container `postgres`, `postgres:17-alpine`, server **17.5**,
`PGDATA=/var/lib/postgresql/data/pgdata`, ~995 MB on disk.

| Database | Size | Database | Size |
|---|---|---|---|
| `nextcloud` | 320 MB | `hiwikijs` | 16 MB |
| `brewman_acc` | 174 MB | `tandoor` | 16 MB |
| `brewman_mhl` | 132 MB | `brewman_exp` | 16 MB |
| `brewman_hops` | 122 MB | `affine` | 11 MB |
| `brewman_hinchco` | 41 MB | `bitwarden` | 10 MB |
| `medusa-criollo` | 21 MB | `zitadel` | 9 MB |

Roles: `postgres` (superuser), `oc_tanshu` (creates databases), `zitadel`.
No custom tablespaces. `template1` is stock; the `postgres` database has no user
tables. `medusa-criollo` and `zitadel` have no running container — nothing writes
them — so they come along for completeness.

## Order of operations

1. Deploy the 18 cluster on `emily` (`postgres/playbook.yml`) and leave it **empty**.
2. Dump on `monoco`.
3. Transfer.
4. Restore into `emily`.
5. Verify, then point the app playbooks at it.

The target must be empty before the restore. `psql` will happily skip a
`CREATE DATABASE` for a database that already exists, so a second restore into a
populated cluster is not an error but is also not a clean overwrite.

## 1. Pre-flight

**The 18 cluster has to exist before any of this.** If it does not, deploy it:

```bash
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
```

Then check it is up and empty:

```bash
# The 18 cluster exists, is running, and holds nothing but the defaults
ssh emily 'systemctl --user is-active postgres.service
           podman exec postgres psql -U postgres -tAc "select datname from pg_database order by 1"'
# expect: active, then just: postgres, template0, template1

# Room for the dump plus the restored data (~2 GB)
ssh emily 'df -h / | tail -1'
ssh monoco 'df -h / | tail -1'
```

If that says `inactive`, or `podman exec` answers `no container with name or ID
"postgres" found`, the unit has not been deployed on `emily` — stop and do that
first. The restore in step 5 would otherwise pipe the whole dump into a
non-existent container and leave almost no trace.

### Optional: quiesce the writers

`pg_dumpall` takes one snapshot per database, so each database is internally
consistent but the cluster is not a single point in time. Every app here owns its
own database and there are no cross-database references, so a live dump is fine
in practice. If you want zero doubt, stop the writers for the duration:

```bash
WRITERS='nextcloud gitea affine bitwarden hiwikijs tandoor
         brewman-acc brewman-hops brewman-mhl brewman-exp brewman-hinchco'

ssh monoco "docker stop $WRITERS"     # before the dump
ssh monoco "docker start $WRITERS"    # after step 3
```

That is real downtime for those sites, so take it or leave it deliberately.

## 2. Dump on monoco

```bash
ssh monoco 'set -euo pipefail
  mkdir -p ~/pg-migrate && cd ~/pg-migrate
  TS=$(date +%Y%m%d-%H%M)
  docker exec -e PGPASSWORD=123456 postgres pg_dumpall -U postgres \
    | zstd -T0 -6 -o "pg17-cluster-$TS.sql.zst"
  zstd -t "pg17-cluster-$TS.sql.zst"
  sha256sum "pg17-cluster-$TS.sql.zst" | tee "pg17-cluster-$TS.sql.zst.sha256"
  ls -lh'
```

It prints nothing while it runs — `pg_dump` buffers, so there is no progress bar
— and takes a couple of minutes for ~900 MB. `zstd -t` is an integrity check;
there is no point transferring a truncated dump.

`pg_dumpall` is used rather than per-database `pg_dump` because it carries the
roles and `template1` in the same stream, and this cluster has three roles that
would otherwise have to be recreated by hand.

## 3. Verify the dump before trusting it

```bash
ssh monoco 'cd ~/pg-migrate && F=$(ls -t pg17-cluster-*.sql.zst | head -1)
  echo "file: $F"
  echo "databases connected : $(zstd -dc "$F" | grep -cE "^\\\\connect")"        # expect 14
  echo "databases created   : $(zstd -dc "$F" | grep -cE "^CREATE DATABASE")"    # expect 12
  echo "role statements     : $(zstd -dc "$F" | grep -cE "^(CREATE|ALTER) ROLE")" # expect 6
  zstd -dc "$F" | head -5'
```

The counts are the shape of this particular cluster: 12 databases get a
`CREATE DATABASE`, and `postgres` + `template1` are dumped in place (14
`\connect` lines). The 6 role statements are 3 roles × (`CREATE` + `ALTER`).

Also check which extensions the dump creates, because step 5 runs with
`ON_ERROR_STOP=1` and a missing extension aborts the whole restore:

```bash
ssh emily 'cd ~/pg-migrate && zstd -dc pg17-cluster.sql.zst | grep -E "^CREATE EXTENSION" | sort | uniq -c'
```

This cluster needs `btree_gist` (×5), `pg_trgm` and `unaccent` — all contrib
modules, and the official image builds contrib in
(`make -C contrib install`), so they are present. An extension that is *not*
contrib — `pgvector`, `postgis`, `timescaledb` — would not be, and would have to
be installed into the image before the restore can succeed. (`docs/podman-compat-audit.md`
notes affine's indexer is disabled, which is why there is no `pgvector` here.)

## 4. Transfer to emily

```bash
# stream it straight across; no copy on the workstation and no second hop
ssh monoco 'cat ~/pg-migrate/pg17-cluster-*.sql.zst' \
  | ssh emily 'mkdir -p ~/pg-migrate && cat > ~/pg-migrate/pg17-cluster.sql.zst'
```

Then gate on the checksum — a truncated dump restores into a partial cluster that
looks fine until an app reads a missing table:

```bash
a=$(ssh monoco 'sha256sum ~/pg-migrate/pg17-cluster-*.sql.zst | cut -d" " -f1')
b=$(ssh emily  'sha256sum ~/pg-migrate/pg17-cluster.sql.zst | cut -d" " -f1')
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
  || { echo "MISMATCH — do not restore: $a vs $b"; false; }
```

`rsync` cannot copy remote-to-remote directly, so the pipe above is the shortest
path. If the link is slow or drops, do it in two hops instead and check the
checksum on both sides:

```bash
rsync -avP monoco:~/pg-migrate/ ~/pg-migrate/
rsync -avP ~/pg-migrate/ emily:~/pg-migrate/
```

## 5. Restore into the 18 cluster on emily

```bash
ssh emily 'cd ~/pg-migrate
  # Fail here, loudly, if the unit was never deployed. Without this, the
  # pipeline below feeds the dump to a podman exec that cannot find a container
  # and the only trace is a line in restore.err that a grep for ERROR misses.
  podman exec postgres pg_isready -U postgres -d postgres \
    || { echo "FAILED: postgres is not running on emily — deploy it first (step 1)."; exit 1; }

  set -o pipefail
  zstd -dc pg17-cluster.sql.zst \
    | grep -v "^CREATE ROLE postgres;$" \
    | podman exec -i -e PGPASSWORD=123456 postgres \
        psql -U postgres -d postgres -v ON_ERROR_STOP=1 -q 2> restore.err \
    || { echo "FAILED: restore stopped on an error:"; tail -20 restore.err; exit 1; }

  echo "OK — restore clean. Databases now:"
  podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
    "select count(*) from pg_database where datallowconn and not datistemplate"'
```

**Expect `13`** — the `postgres` database plus the 12 restored ones. That count is
the positive proof the restore happened; an empty error log on its own proves
nothing, which is exactly how a missing container slipped through the first time.

Three details are doing real work here:

- **`pg_isready` first.** `podman exec` on a missing container writes
  `Error: no container with name or ID "postgres" found` and exits 1 — but the
  pipeline still "completes", and that message is invisible to a case-sensitive
  `grep ERROR`. This is the check that turns that into an immediate, obvious stop.
- **`grep -v '^CREATE ROLE postgres;$'`.** That is the one statement expected to
  fail: `pg_dumpall` emits it and initdb has already created the role. The
  `ALTER ROLE postgres ... PASSWORD` that follows is kept and succeeds, so the
  superuser ends up with monoco's password hash — the same `123456` anyway.
  Removing that exact line, rather than tolerating the error, is what allows
  `ON_ERROR_STOP=1`: any *unexpected* error now aborts the restore and is printed
  instead of being skipped.
- **The database count at the end**, as above.

The dump was made by `pg_dump` 17.5 and is replayed by `psql` 18: forward
restores are the supported direction, and source and target are both the
Alpine/musl image, so the `en_US.utf8` locale the databases were created with
resolves on both sides.

If the restore does stop, fix the cause and start over — `ON_ERROR_STOP=1` can
leave the cluster half-loaded, and re-running on top of that adds "already
exists" noise rather than converging.

```bash
# destroys everything in the emily cluster
ssh emily 'systemctl --user stop postgres.service
           rm -rf ~/podman-data/postgres/18'
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
```

That `rm -rf` needs no `podman unshare` precisely because the role creates the
version directory as `tanshu` rather than letting the container strand it as a
subuid-owned directory. If it ever fails with permission denied, that is the
assumption breaking — check `ls -ln ~/podman-data/postgres/18`.

## 6. Verify

```bash
# server version, and the databases with their sizes — compare against the table above
ssh emily 'podman exec postgres postgres --version'
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
  "select datname, pg_size_pretty(pg_database_size(datname)) from pg_database order by pg_database_size(datname) desc"'

# all three roles made it
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -c "\du"'
```

Then the check that actually matters — the same per-database table count on both
sides. Feed the loop to the remote shell over stdin so the quoting stays sane:

```bash
ssh monoco 'bash -s' <<'EOF'
EXEC="docker exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
  printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF

ssh emily 'bash -s' <<'EOF'
EXEC="podman exec -e PGPASSWORD=123456 postgres"
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
  printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
done
EOF
```

Save both outputs and diff them — identical table counts per database is the
signal that the restore was complete. Sizes will differ slightly and that is
normal (toast/fillfactor are not byte-reproducible).

Also confirm ownership is what the playbook promises, because a wrong `keep-id`
shows up here and nowhere else:

```bash
ssh emily 'ls -ln ~/podman-data/postgres/18/docker | head'
# PGDATA and its contents must be owned by tanshu:tanshu, not a 165536+ subuid
```

## 7. Afterwards

- **Keep monoco's cluster.** Do not `docker rm postgres` and do not drop
  databases: a stopped, intact 17 cluster is the fastest rollback there is.
  Leave the dump in `~/pg-migrate` on both hosts for the same reason.
- **Stop the old writers before the new ones start** if both hosts could serve
  the same app, or you will have two databases diverging behind the same site.
- **Apps connect unchanged.** `pg_dumpall` restores each database under its
  original name with its original role grants, and the app playbooks'
  `postgres_container: postgres` resolves over the attached app networks — same
  container name, same `db` alias, same password.
- **The dump is not a backup regime.** It is a migration artifact. Per-database
  `pg_dump` on a schedule is still owed (see the Backups section of
  [readme.md](readme.md)).

## Rollback

There is nothing to reverse on `emily` beyond stopping the new container:
`systemctl --user stop postgres.service`. `monoco` was never modified — its
container was only read — so rolling back is starting the old stack again.

The one exception is the quiesce step, if you took it: `docker start` the
`WRITERS` list above.