Run Postgres as a rootless Podman Quadlet, and move the 17 cluster across
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same shape as Caddy: it knows nothing about its clients, and a service attaches itself by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing here changes when a service is added or removed. Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker), and the 18 entrypoint refuses to start when it finds a cluster in the old layout. So postgres_major is the single knob: it picks the image tag and the PGDATA path together, and the mount is now the parent directory. That also settles the migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster comes across as a pg_dumpall. Three things that bit us and are worth keeping: - PGDATA and its version directory are created by the role, not left to the container. The entrypoint runs as container-root, which under keep-id is a subuid on the host, and it only chowns PGDATA and its contents -- so ~/podman-data/postgres/18 would have been stranded subuid-owned, and the host user could not even rm -rf it without podman unshare. - NetworkAlias= applies to every network the container joins, which is what a database hub wants, but it is one fixed value in a shared drop-in rather than per-service: two services writing their own would make Quadlet pass --network-alias twice. It is not in postgres.container itself because on a fresh host that unit has no Network= at all. - The credentials live in a 0600 file read with EnvironmentFile=, not in Environment= lines: the unit is world-readable in ~/.config. The play no longer uses become, and does not apply the rootless-Podman prerequisites -- those are the shared task file at caddy/roles/podman, and the three variables they need are duplicated at the top of vars/default.yml. migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer and restore commands, the checks that prove the restore is complete (per-database table counts against the source), and the verified result. The dead nginx/php-fpm templates in postgres/files/ are removed.
This commit is contained in:
tanshu committed
1 parent
060173db5e
commit
4297e5a71e
11 files changed
+1079
-88
No files matched your search
@@ -1,2 +0,0 @@
|
|||||||
<?php
|
|
||||||
phpinfo();
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
server {
|
|
||||||
listen {{ http_port }};
|
|
||||||
root /var/www/html;
|
|
||||||
index index.php index.html index.htm index.nginx-debian.html;
|
|
||||||
server_name {{ http_host }};
|
|
||||||
|
|
||||||
location / {
|
|
||||||
try_files $uri $uri/ =404;
|
|
||||||
}
|
|
||||||
|
|
||||||
location ~ \.php$ {
|
|
||||||
include snippets/fastcgi-php.conf;
|
|
||||||
fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
|
|
||||||
}
|
|
||||||
|
|
||||||
location ~ /\.ht {
|
|
||||||
deny all;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,315 @@
|
|||||||
|
# Moving the cluster from monoco (Postgres 17 → 18)
|
||||||
|
|
||||||
|
> **Status: done — do not re-run against the live cluster.** Executed
|
||||||
|
> 2026-10-15, Postgres 17.5 → 18.6. Verified afterwards: 12 databases restored
|
||||||
|
> with per-database table counts *identical* to monoco, 13 databases and 3 roles
|
||||||
|
> (`postgres`, `oc_tanshu`, `zitadel`), `btree_gist`/`pg_trgm`/`unaccent` present,
|
||||||
|
> no listener on host `5432`, and every directory in the data root owned by
|
||||||
|
> `tanshu` rather than a subuid.
|
||||||
|
>
|
||||||
|
> The steps below are kept as the record of how it was done, and for the next
|
||||||
|
> major-version move. Running them again would now fail on the first
|
||||||
|
> `CREATE DATABASE` and leave a half-loaded cluster — and the "start over" block
|
||||||
|
> in step 5 *destroys* `~/podman-data/postgres/18`.
|
||||||
|
|
||||||
|
One-time move of every database from `monoco`'s Docker Postgres to `emily`'s
|
||||||
|
rootless Quadlet Postgres, across a major-version upgrade. Run the commands from
|
||||||
|
the machine that holds the `monoco` and `emily` SSH aliases.
|
||||||
|
|
||||||
|
## Why this is a dump and not a copy
|
||||||
|
|
||||||
|
Two independent reasons, either of which is decisive:
|
||||||
|
|
||||||
|
- **17 data files cannot be read by an 18 server.** A copy of `PGDATA` produces a
|
||||||
|
cluster that 18 refuses to open — it would have to go through `pg_upgrade` with
|
||||||
|
both versions present.
|
||||||
|
- **The 18 image moved PGDATA.** In 17 it was `/var/lib/postgresql/data/pgdata`; in
|
||||||
|
18 it is `/var/lib/postgresql/18/docker`, and the entrypoint deliberately
|
||||||
|
refuses to start when it finds a cluster in the old location, to stop an image
|
||||||
|
bump from silently looking like data loss.
|
||||||
|
|
||||||
|
A logical dump sidesteps both, and at this size it is also the fastest thing to
|
||||||
|
get right. `pg_upgrade` would avoid re-importing, but it needs the 17 data
|
||||||
|
directory copied to `emily` first *and* a purpose-built image with both versions
|
||||||
|
(`tianon/postgres-upgrade:17-to-18`), for a cluster of well under a gigabyte. Not
|
||||||
|
worth the extra failure modes.
|
||||||
|
|
||||||
|
## What is being moved (measured on monoco)
|
||||||
|
|
||||||
|
Source: Docker container `postgres`, `postgres:17-alpine`, server **17.5**,
|
||||||
|
`PGDATA=/var/lib/postgresql/data/pgdata`, ~995 MB on disk.
|
||||||
|
|
||||||
|
| Database | Size | Database | Size |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `nextcloud` | 320 MB | `hiwikijs` | 16 MB |
|
||||||
|
| `brewman_acc` | 174 MB | `tandoor` | 16 MB |
|
||||||
|
| `brewman_mhl` | 132 MB | `brewman_exp` | 16 MB |
|
||||||
|
| `brewman_hops` | 122 MB | `affine` | 11 MB |
|
||||||
|
| `brewman_hinchco` | 41 MB | `bitwarden` | 10 MB |
|
||||||
|
| `medusa-criollo` | 21 MB | `zitadel` | 9 MB |
|
||||||
|
|
||||||
|
Roles: `postgres` (superuser), `oc_tanshu` (creates databases), `zitadel`.
|
||||||
|
No custom tablespaces. `template1` is stock; the `postgres` database has no user
|
||||||
|
tables. `medusa-criollo` and `zitadel` have no running container — nothing writes
|
||||||
|
them — so they come along for completeness.
|
||||||
|
|
||||||
|
## Order of operations
|
||||||
|
|
||||||
|
1. Deploy the 18 cluster on `emily` (`postgres/playbook.yml`) and leave it **empty**.
|
||||||
|
2. Dump on `monoco`.
|
||||||
|
3. Transfer.
|
||||||
|
4. Restore into `emily`.
|
||||||
|
5. Verify, then point the app playbooks at it.
|
||||||
|
|
||||||
|
The target must be empty before the restore. `psql` will happily skip a
|
||||||
|
`CREATE DATABASE` for a database that already exists, so a second restore into a
|
||||||
|
populated cluster is not an error but is also not a clean overwrite.
|
||||||
|
|
||||||
|
## 1. Pre-flight
|
||||||
|
|
||||||
|
**The 18 cluster has to exist before any of this.** If it does not, deploy it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Then check it is up and empty:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# The 18 cluster exists, is running, and holds nothing but the defaults
|
||||||
|
ssh emily 'systemctl --user is-active postgres.service
|
||||||
|
podman exec postgres psql -U postgres -tAc "select datname from pg_database order by 1"'
|
||||||
|
# expect: active, then just: postgres, template0, template1
|
||||||
|
|
||||||
|
# Room for the dump plus the restored data (~2 GB)
|
||||||
|
ssh emily 'df -h / | tail -1'
|
||||||
|
ssh monoco 'df -h / | tail -1'
|
||||||
|
```
|
||||||
|
|
||||||
|
If that says `inactive`, or `podman exec` answers `no container with name or ID
|
||||||
|
"postgres" found`, the unit has not been deployed on `emily` — stop and do that
|
||||||
|
first. The restore in step 5 would otherwise pipe the whole dump into a
|
||||||
|
non-existent container and leave almost no trace.
|
||||||
|
|
||||||
|
### Optional: quiesce the writers
|
||||||
|
|
||||||
|
`pg_dumpall` takes one snapshot per database, so each database is internally
|
||||||
|
consistent but the cluster is not a single point in time. Every app here owns its
|
||||||
|
own database and there are no cross-database references, so a live dump is fine
|
||||||
|
in practice. If you want zero doubt, stop the writers for the duration:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
WRITERS='nextcloud gitea affine bitwarden hiwikijs tandoor
|
||||||
|
brewman-acc brewman-hops brewman-mhl brewman-exp brewman-hinchco'
|
||||||
|
|
||||||
|
ssh monoco "docker stop $WRITERS" # before the dump
|
||||||
|
ssh monoco "docker start $WRITERS" # after step 3
|
||||||
|
```
|
||||||
|
|
||||||
|
That is real downtime for those sites, so take it or leave it deliberately.
|
||||||
|
|
||||||
|
## 2. Dump on monoco
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh monoco 'set -euo pipefail
|
||||||
|
mkdir -p ~/pg-migrate && cd ~/pg-migrate
|
||||||
|
TS=$(date +%Y%m%d-%H%M)
|
||||||
|
docker exec -e PGPASSWORD=123456 postgres pg_dumpall -U postgres \
|
||||||
|
| zstd -T0 -6 -o "pg17-cluster-$TS.sql.zst"
|
||||||
|
zstd -t "pg17-cluster-$TS.sql.zst"
|
||||||
|
sha256sum "pg17-cluster-$TS.sql.zst" | tee "pg17-cluster-$TS.sql.zst.sha256"
|
||||||
|
ls -lh'
|
||||||
|
```
|
||||||
|
|
||||||
|
It prints nothing while it runs — `pg_dump` buffers, so there is no progress bar
|
||||||
|
— and takes a couple of minutes for ~900 MB. `zstd -t` is an integrity check;
|
||||||
|
there is no point transferring a truncated dump.
|
||||||
|
|
||||||
|
`pg_dumpall` is used rather than per-database `pg_dump` because it carries the
|
||||||
|
roles and `template1` in the same stream, and this cluster has three roles that
|
||||||
|
would otherwise have to be recreated by hand.
|
||||||
|
|
||||||
|
## 3. Verify the dump before trusting it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh monoco 'cd ~/pg-migrate && F=$(ls -t pg17-cluster-*.sql.zst | head -1)
|
||||||
|
echo "file: $F"
|
||||||
|
echo "databases connected : $(zstd -dc "$F" | grep -cE "^\\\\connect")" # expect 14
|
||||||
|
echo "databases created : $(zstd -dc "$F" | grep -cE "^CREATE DATABASE")" # expect 12
|
||||||
|
echo "role statements : $(zstd -dc "$F" | grep -cE "^(CREATE|ALTER) ROLE")" # expect 6
|
||||||
|
zstd -dc "$F" | head -5'
|
||||||
|
```
|
||||||
|
|
||||||
|
The counts are the shape of this particular cluster: 12 databases get a
|
||||||
|
`CREATE DATABASE`, and `postgres` + `template1` are dumped in place (14
|
||||||
|
`\connect` lines). The 6 role statements are 3 roles × (`CREATE` + `ALTER`).
|
||||||
|
|
||||||
|
Also check which extensions the dump creates, because step 5 runs with
|
||||||
|
`ON_ERROR_STOP=1` and a missing extension aborts the whole restore:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh emily 'cd ~/pg-migrate && zstd -dc pg17-cluster.sql.zst | grep -E "^CREATE EXTENSION" | sort | uniq -c'
|
||||||
|
```
|
||||||
|
|
||||||
|
This cluster needs `btree_gist` (×5), `pg_trgm` and `unaccent` — all contrib
|
||||||
|
modules, and the official image builds contrib in
|
||||||
|
(`make -C contrib install`), so they are present. An extension that is *not*
|
||||||
|
contrib — `pgvector`, `postgis`, `timescaledb` — would not be, and would have to
|
||||||
|
be installed into the image before the restore can succeed. (`docs/podman-compat-audit.md`
|
||||||
|
notes affine's indexer is disabled, which is why there is no `pgvector` here.)
|
||||||
|
|
||||||
|
## 4. Transfer to emily
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# stream it straight across; no copy on the workstation and no second hop
|
||||||
|
ssh monoco 'cat ~/pg-migrate/pg17-cluster-*.sql.zst' \
|
||||||
|
| ssh emily 'mkdir -p ~/pg-migrate && cat > ~/pg-migrate/pg17-cluster.sql.zst'
|
||||||
|
```
|
||||||
|
|
||||||
|
Then gate on the checksum — a truncated dump restores into a partial cluster that
|
||||||
|
looks fine until an app reads a missing table:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
a=$(ssh monoco 'sha256sum ~/pg-migrate/pg17-cluster-*.sql.zst | cut -d" " -f1')
|
||||||
|
b=$(ssh emily 'sha256sum ~/pg-migrate/pg17-cluster.sql.zst | cut -d" " -f1')
|
||||||
|
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
|
||||||
|
|| { echo "MISMATCH — do not restore: $a vs $b"; false; }
|
||||||
|
```
|
||||||
|
|
||||||
|
`rsync` cannot copy remote-to-remote directly, so the pipe above is the shortest
|
||||||
|
path. If the link is slow or drops, do it in two hops instead and check the
|
||||||
|
checksum on both sides:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rsync -avP monoco:~/pg-migrate/ ~/pg-migrate/
|
||||||
|
rsync -avP ~/pg-migrate/ emily:~/pg-migrate/
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Restore into the 18 cluster on emily
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh emily 'cd ~/pg-migrate
|
||||||
|
# Fail here, loudly, if the unit was never deployed. Without this, the
|
||||||
|
# pipeline below feeds the dump to a podman exec that cannot find a container
|
||||||
|
# and the only trace is a line in restore.err that a grep for ERROR misses.
|
||||||
|
podman exec postgres pg_isready -U postgres -d postgres \
|
||||||
|
|| { echo "FAILED: postgres is not running on emily — deploy it first (step 1)."; exit 1; }
|
||||||
|
|
||||||
|
set -o pipefail
|
||||||
|
zstd -dc pg17-cluster.sql.zst \
|
||||||
|
| grep -v "^CREATE ROLE postgres;$" \
|
||||||
|
| podman exec -i -e PGPASSWORD=123456 postgres \
|
||||||
|
psql -U postgres -d postgres -v ON_ERROR_STOP=1 -q 2> restore.err \
|
||||||
|
|| { echo "FAILED: restore stopped on an error:"; tail -20 restore.err; exit 1; }
|
||||||
|
|
||||||
|
echo "OK — restore clean. Databases now:"
|
||||||
|
podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
|
||||||
|
"select count(*) from pg_database where datallowconn and not datistemplate"'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expect `13`** — the `postgres` database plus the 12 restored ones. That count is
|
||||||
|
the positive proof the restore happened; an empty error log on its own proves
|
||||||
|
nothing, which is exactly how a missing container slipped through the first time.
|
||||||
|
|
||||||
|
Three details are doing real work here:
|
||||||
|
|
||||||
|
- **`pg_isready` first.** `podman exec` on a missing container writes
|
||||||
|
`Error: no container with name or ID "postgres" found` and exits 1 — but the
|
||||||
|
pipeline still "completes", and that message is invisible to a case-sensitive
|
||||||
|
`grep ERROR`. This is the check that turns that into an immediate, obvious stop.
|
||||||
|
- **`grep -v '^CREATE ROLE postgres;$'`.** That is the one statement expected to
|
||||||
|
fail: `pg_dumpall` emits it and initdb has already created the role. The
|
||||||
|
`ALTER ROLE postgres ... PASSWORD` that follows is kept and succeeds, so the
|
||||||
|
superuser ends up with monoco's password hash — the same `123456` anyway.
|
||||||
|
Removing that exact line, rather than tolerating the error, is what allows
|
||||||
|
`ON_ERROR_STOP=1`: any *unexpected* error now aborts the restore and is printed
|
||||||
|
instead of being skipped.
|
||||||
|
- **The database count at the end**, as above.
|
||||||
|
|
||||||
|
The dump was made by `pg_dump` 17.5 and is replayed by `psql` 18: forward
|
||||||
|
restores are the supported direction, and source and target are both the
|
||||||
|
Alpine/musl image, so the `en_US.utf8` locale the databases were created with
|
||||||
|
resolves on both sides.
|
||||||
|
|
||||||
|
If the restore does stop, fix the cause and start over — `ON_ERROR_STOP=1` can
|
||||||
|
leave the cluster half-loaded, and re-running on top of that adds "already
|
||||||
|
exists" noise rather than converging.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# destroys everything in the emily cluster
|
||||||
|
ssh emily 'systemctl --user stop postgres.service
|
||||||
|
rm -rf ~/podman-data/postgres/18'
|
||||||
|
cd prepare-server && ansible-playbook -u tanshu ../postgres/playbook.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
That `rm -rf` needs no `podman unshare` precisely because the role creates the
|
||||||
|
version directory as `tanshu` rather than letting the container strand it as a
|
||||||
|
subuid-owned directory. If it ever fails with permission denied, that is the
|
||||||
|
assumption breaking — check `ls -ln ~/podman-data/postgres/18`.
|
||||||
|
|
||||||
|
## 6. Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# server version, and the databases with their sizes — compare against the table above
|
||||||
|
ssh emily 'podman exec postgres postgres --version'
|
||||||
|
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -tAc \
|
||||||
|
"select datname, pg_size_pretty(pg_database_size(datname)) from pg_database order by pg_database_size(datname) desc"'
|
||||||
|
|
||||||
|
# all three roles made it
|
||||||
|
ssh emily 'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -c "\du"'
|
||||||
|
```
|
||||||
|
|
||||||
|
Then the check that actually matters — the same per-database table count on both
|
||||||
|
sides. Feed the loop to the remote shell over stdin so the quoting stays sane:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh monoco 'bash -s' <<'EOF'
|
||||||
|
EXEC="docker exec -e PGPASSWORD=123456 postgres"
|
||||||
|
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
|
||||||
|
printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
|
||||||
|
done
|
||||||
|
EOF
|
||||||
|
|
||||||
|
ssh emily 'bash -s' <<'EOF'
|
||||||
|
EXEC="podman exec -e PGPASSWORD=123456 postgres"
|
||||||
|
for db in $($EXEC psql -U postgres -tAc "select datname from pg_database where datallowconn and not datistemplate and datname <> 'postgres' order by 1"); do
|
||||||
|
printf '%-16s %s\n' "$db" "$($EXEC psql -U postgres -d "$db" -tAc "select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')")"
|
||||||
|
done
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
Save both outputs and diff them — identical table counts per database is the
|
||||||
|
signal that the restore was complete. Sizes will differ slightly and that is
|
||||||
|
normal (toast/fillfactor are not byte-reproducible).
|
||||||
|
|
||||||
|
Also confirm ownership is what the playbook promises, because a wrong `keep-id`
|
||||||
|
shows up here and nowhere else:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh emily 'ls -ln ~/podman-data/postgres/18/docker | head'
|
||||||
|
# PGDATA and its contents must be owned by tanshu:tanshu, not a 165536+ subuid
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Afterwards
|
||||||
|
|
||||||
|
- **Keep monoco's cluster.** Do not `docker rm postgres` and do not drop
|
||||||
|
databases: a stopped, intact 17 cluster is the fastest rollback there is.
|
||||||
|
Leave the dump in `~/pg-migrate` on both hosts for the same reason.
|
||||||
|
- **Stop the old writers before the new ones start** if both hosts could serve
|
||||||
|
the same app, or you will have two databases diverging behind the same site.
|
||||||
|
- **Apps connect unchanged.** `pg_dumpall` restores each database under its
|
||||||
|
original name with its original role grants, and the app playbooks'
|
||||||
|
`postgres_container: postgres` resolves over the attached app networks — same
|
||||||
|
container name, same `db` alias, same password.
|
||||||
|
- **The dump is not a backup regime.** It is a migration artifact. Per-database
|
||||||
|
`pg_dump` on a schedule is still owed (see the Backups section of
|
||||||
|
[readme.md](readme.md)).
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
There is nothing to reverse on `emily` beyond stopping the new container:
|
||||||
|
`systemctl --user stop postgres.service`. `monoco` was never modified — its
|
||||||
|
container was only read — so rolling back is starting the old stack again.
|
||||||
|
|
||||||
|
The one exception is the quiesce step, if you took it: `docker start` the
|
||||||
|
`WRITERS` list above.
|
||||||
+28
-33
@@ -1,37 +1,32 @@
|
|||||||
#################################################
|
|
||||||
# DO Community Playbooks: Docker
|
|
||||||
#################################################
|
|
||||||
---
|
---
|
||||||
- hosts: all
|
# Deploys PostgreSQL as a rootless Podman Quadlet unit.
|
||||||
become: true
|
#
|
||||||
|
# Connects as the container user and runs everything as that user: the Quadlet
|
||||||
|
# units belong to it, so the connection's PAM session is what makes
|
||||||
|
# `systemctl --user` work without any environment juggling.
|
||||||
|
#
|
||||||
|
# gather_facts is required: podman_user and the home directory come from the
|
||||||
|
# user_id / user_dir facts.
|
||||||
|
#
|
||||||
|
# The rootless-Podman prerequisites (installing podman, enabling lingering, the
|
||||||
|
# low-port sysctl, creating the Quadlet directory) are deliberately NOT applied
|
||||||
|
# here. They are the shared task file at caddy/roles/podman/tasks/main.yaml, and
|
||||||
|
# a host that has already run caddy/playbook.yml has them:
|
||||||
|
#
|
||||||
|
# - name: Rootless Podman prerequisites
|
||||||
|
# ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
|
||||||
|
#
|
||||||
|
# The variables those tasks need are duplicated at the top of vars/default.yml,
|
||||||
|
# so this play stands on its own.
|
||||||
|
#
|
||||||
|
# Run with: ansible-playbook -u tanshu postgres/playbook.yml
|
||||||
|
- name: Deploy PostgreSQL as a rootless Podman Quadlet
|
||||||
|
hosts: emily
|
||||||
|
gather_facts: true
|
||||||
vars_files:
|
vars_files:
|
||||||
- vars/default.yml
|
- vars/default.yml
|
||||||
|
|
||||||
tasks:
|
roles:
|
||||||
- name: Pull default Postgresql image
|
# The base postgres unit. It does NOT know which app networks it serves.
|
||||||
docker_image:
|
# Each service attaches itself via postgres/roles/attach.
|
||||||
name: "{{ postgres_image }}"
|
- postgres
|
||||||
source: pull
|
|
||||||
force_source: yes
|
|
||||||
|
|
||||||
- name: Ensure Nextcloud Directory exists
|
|
||||||
file:
|
|
||||||
path: "/var/lib/{{ host_directory }}"
|
|
||||||
state: directory
|
|
||||||
mode: 0755
|
|
||||||
|
|
||||||
- name: Create postgres container
|
|
||||||
docker_container:
|
|
||||||
name: "{{ postgres_name }}"
|
|
||||||
image: "{{ postgres_image }}"
|
|
||||||
state: started
|
|
||||||
restart_policy: "unless-stopped"
|
|
||||||
env:
|
|
||||||
POSTGRES_USER: "{{ pg_user }}"
|
|
||||||
POSTGRES_PASSWORD: "{{ pg_password }}"
|
|
||||||
POSTGRES_DB: 'postgres'
|
|
||||||
PGDATA: "{{ data_location }}/pgdata"
|
|
||||||
published_ports:
|
|
||||||
- 5432:5432
|
|
||||||
volumes:
|
|
||||||
- "/var/lib/{{ host_directory }}/data:{{ data_location }}"
|
|
||||||
+307
-30
@@ -1,46 +1,323 @@
|
|||||||
# Docker on Ubuntu 18.04
|
# PostgreSQL (rootless Podman + Quadlet)
|
||||||
|
|
||||||
This playbook will install Docker an Ubuntu 18.04 machine, as explained in the guide on
|
Runs PostgreSQL as a **rootless Podman** container managed by **systemd user
|
||||||
[How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
|
units** (Quadlet). Replaces the `docker_container` playbook, which published
|
||||||
A number of containers will be created with the options specified in the `vars/default.yml` variable file.
|
`5432` on `0.0.0.0` and kept its data in `/var/lib/postgresql`.
|
||||||
|
|
||||||
|
**Postgres is the base unit and knows nothing about its clients.** It does not
|
||||||
|
list any app network and does not need editing when a service is added or
|
||||||
|
removed. Each service attaches *itself*. That mirrors [caddy](../caddy/readme.md),
|
||||||
|
which is the reverse proxy end of the same idea.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
postgres/
|
||||||
|
├── playbook.yml
|
||||||
|
├── vars/default.yml
|
||||||
|
└── roles/
|
||||||
|
├── postgres/ # the base postgres.container unit, env file, handlers
|
||||||
|
└── attach/ # includable task file: a service attaches itself to Postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
Everything is installed under `{{ quadlet_dir }}`
|
||||||
|
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
|
||||||
|
(`~/podman-data/postgres/`).
|
||||||
|
|
||||||
|
## How services attach themselves
|
||||||
|
|
||||||
|
Postgres joins the app network of every service that needs a database. A service
|
||||||
|
joins by writing one Quadlet *drop-in* next to `postgres.container`:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.config/containers/systemd/
|
||||||
|
├── postgres.container # base unit, no app networks
|
||||||
|
├── postgres.container.d/ # ← drop-ins, one per client service
|
||||||
|
│ ├── 10-network-aliases.conf # [Container] NetworkAlias=db (shared)
|
||||||
|
│ ├── 50-nextcloud.conf # [Container] Network=nextcloud.network
|
||||||
|
│ ├── 50-gitea.conf
|
||||||
|
│ └── 50-tandoor.conf
|
||||||
|
├── nextcloud.network
|
||||||
|
├── gitea.network
|
||||||
|
└── tandoor.network
|
||||||
|
```
|
||||||
|
|
||||||
|
A service playbook does both halves — its own network, then the attach:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: Register this service with Caddy
|
||||||
|
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
|
||||||
|
vars:
|
||||||
|
network_name: "{{ container_name }}"
|
||||||
|
|
||||||
|
- name: Attach this service to Postgres
|
||||||
|
ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
|
||||||
|
vars:
|
||||||
|
network_name: "{{ container_name }}"
|
||||||
|
```
|
||||||
|
|
||||||
|
The play must also run the `podman` role (for `podman_user` and `quadlet_dir`)
|
||||||
|
and define a `Restart Postgres` handler — see
|
||||||
|
[roles/postgres/handlers/main.yaml](roles/postgres/handlers/main.yaml) for the
|
||||||
|
exact snippet.
|
||||||
|
|
||||||
|
Order matters in one direction only: the `attach` include needs
|
||||||
|
`<name>.network` to already exist, which is why Caddy's `register` comes first.
|
||||||
|
It fails with a readable message if it does not, because a drop-in naming a
|
||||||
|
missing `.network` file makes Quadlet reject the *whole* postgres unit with an
|
||||||
|
error that never mentions the drop-in.
|
||||||
|
|
||||||
|
### Names: `postgres` and `db`
|
||||||
|
|
||||||
|
Both resolve on every attached network. `postgres` comes from
|
||||||
|
`ContainerName=postgres`; `db` is added by `10-network-aliases.conf`. The alias
|
||||||
|
exists because the Docker-era playbooks are not consistent — nextcloud, gitea,
|
||||||
|
tandoor and outline connect to `db`, the rest to `postgres`.
|
||||||
|
|
||||||
|
`NetworkAlias=` applies to *every* network the container joins, which is exactly
|
||||||
|
what a hub wants. It lives in one shared drop-in with a fixed value rather than
|
||||||
|
in each service's `50-*.conf`, for two reasons:
|
||||||
|
|
||||||
|
- it is not per-service — if one service could set a different value, the shared
|
||||||
|
file would be rewritten on every run, and every service writes the same bytes;
|
||||||
|
- with the alias in each service's own file, two attached services would make
|
||||||
|
Quadlet pass `--network-alias db` twice.
|
||||||
|
|
||||||
|
It is *not* in `postgres.container` itself for a duller reason: on a fresh host,
|
||||||
|
before anything has attached, the unit has no `Network=` at all, and asking
|
||||||
|
Podman for an alias on the default network is not a configuration worth
|
||||||
|
depending on.
|
||||||
|
|
||||||
|
### One way to attach, and one rule
|
||||||
|
|
||||||
|
[roles/attach](roles/attach/tasks/main.yaml) is the shared include, and is what
|
||||||
|
every service uses. A service may instead write the same two files from its own
|
||||||
|
role, but then it **must write identical bytes** on purpose: the drop-ins live in
|
||||||
|
one directory, so anything less means whichever play runs last rewrites the
|
||||||
|
other's file on every run. If you add a second form, make it match.
|
||||||
|
|
||||||
|
Whichever form you use, **flush handlers before the app starts on a first
|
||||||
|
attach.** The restart that joins Postgres to the new network is a handler, so by
|
||||||
|
default it runs at the end of the play — after the app has already come up and
|
||||||
|
failed to resolve `db`. `tandoor/roles/postgres` is the working example.
|
||||||
|
|
||||||
|
### Why drop-ins rather than listing the networks
|
||||||
|
|
||||||
|
Listing app networks in `postgres.container` would mean editing this playbook
|
||||||
|
every time a service is added, and again on removal. Drop-ins survive
|
||||||
|
re-templating of the base unit, are the mechanism Quadlet documents for this,
|
||||||
|
and make removal trivial: delete `50-tandoor.conf`, `daemon-reload`, done.
|
||||||
|
|
||||||
|
### The trade-off
|
||||||
|
|
||||||
|
A new drop-in changes the generated unit, so Postgres is **recreated** to join
|
||||||
|
the network — a few seconds of downtime for *every* connected app, not just the
|
||||||
|
new one. Quadlet has no live equivalent of `docker network connect`. If that
|
||||||
|
matters, `podman network connect <net> postgres` takes effect immediately, and
|
||||||
|
the drop-in makes it survive the next recreate; the `attach` role does not do
|
||||||
|
this by default because it would make the play non-idempotent.
|
||||||
|
|
||||||
|
### Why not one shared database network
|
||||||
|
|
||||||
|
Every app on a shared `postgres_net` could reach every other app on it. The
|
||||||
|
generated `<name>.network` units set `Options=isolate=true`, so keeping Postgres
|
||||||
|
as the only common member of otherwise separate networks is what preserves the
|
||||||
|
per-app isolation Caddy's networks give you.
|
||||||
|
|
||||||
|
## Who runs as whom
|
||||||
|
|
||||||
|
`UserNS=keep-id:uid=70,gid=70`. `70` is the uid the *server* writes as: the
|
||||||
|
official Alpine entrypoint starts as root, chowns PGDATA to the `postgres`
|
||||||
|
account, and drops privileges, so `docker exec postgres id -u` reporting `0` is
|
||||||
|
misleading. `keep-id` maps that one uid onto `podman_user`, which means:
|
||||||
|
|
||||||
|
- the whole data root — `~/podman-data/postgres/18/docker` and the version
|
||||||
|
directory above it — is owned by `podman_user`, never by a subuid;
|
||||||
|
- plain `tar`, `pg_dump`, `cp` and `rm -rf` work without `podman unshare` and
|
||||||
|
without `sudo`;
|
||||||
|
- the reverse — chowning the data to `podman_user` before first start — is also
|
||||||
|
correct, which is what makes bringing data across simple.
|
||||||
|
|
||||||
|
PGDATA and its version directory are created by the role rather than left to the
|
||||||
|
container's own `mkdir -p`. The entrypoint runs as container-root — a subuid
|
||||||
|
under `keep-id` — and only chowns PGDATA and its contents, so the version
|
||||||
|
directory would be stranded as subuid-owned: readable but not removable by
|
||||||
|
`podman_user`. Creating the chain up front is what makes the bullets above true.
|
||||||
|
|
||||||
|
Like Caddy, `podman_user` is derived from the connection
|
||||||
|
(`ansible_facts['user_id']`) and the play carries no `become`. This playbook does
|
||||||
|
not run the podman role, so it has no guard against connecting as `root`; the
|
||||||
|
prerequisites are applied separately (see Running).
|
||||||
|
|
||||||
## Settings
|
## Settings
|
||||||
|
|
||||||
- `create_containers`: number of containers to create.
|
| Variable | Purpose |
|
||||||
- `default_container_name`: default name for new containers.
|
|---|---|
|
||||||
- `default_container_image`: default image for new containers.
|
| `podman_user`, `podman_data_dir`, `quadlet_dir` | Rootless-Podman contract, duplicated from [caddy/vars/default.yml](../caddy/vars/default.yml) |
|
||||||
- `default_container_command`: default command to run on new containers.
|
| `postgres_container` | Container / unit name, and the prefix of the drop-in directory |
|
||||||
|
| `postgres_major` | The one knob: pins both the image tag and the PGDATA path |
|
||||||
|
| `postgres_image` | `postgres:<major>-alpine`, fully qualified on purpose (see below) |
|
||||||
|
| `postgres_uid` / `postgres_gid` | The uid the server writes as — `70` in the Alpine image |
|
||||||
|
| `host_directory` | Where state lives (`~/podman-data/postgres`) |
|
||||||
|
| `postgres_mount` / `pgdata` | The image's VOLUME, and PGDATA inside it (`/var/lib/postgresql`, `/var/lib/postgresql/18/docker`) |
|
||||||
|
| `postgres_env_file` | `0600` file holding the credentials, read via `EnvironmentFile=` |
|
||||||
|
| `postgres_dropin_dir` | Where services attach their networks |
|
||||||
|
| `pg_user` / `pg_password` / `pg_database` | Credentials and the default database |
|
||||||
|
| `postgres_timeout_start` / `postgres_timeout_stop` | initdb on first start, clean shutdown on stop |
|
||||||
|
|
||||||
|
### Image and PGDATA layout
|
||||||
|
|
||||||
## Running this Playbook
|
`docker.io/library/postgres:18-alpine` — fully qualified because rootless Podman
|
||||||
|
resolves short names against `unqualified-search-registries`, and a Quadlet unit
|
||||||
|
has no TTY to answer a short-name prompt. `-alpine` still floats within 18.x for
|
||||||
|
security patches.
|
||||||
|
|
||||||
Quick Steps:
|
**18 moved both the image's VOLUME and its default PGDATA**, which is why
|
||||||
|
`postgres_major` exists and why the mount is a *directory* rather than the data
|
||||||
|
directory itself:
|
||||||
|
|
||||||
### 1. Obtain the playbook
|
| | 17 | 18+ |
|
||||||
```shell
|
|---|---|---|
|
||||||
git clone https://github.com/do-community/ansible-playbooks.git
|
| `VOLUME` | `/var/lib/postgresql/data` | `/var/lib/postgresql` |
|
||||||
cd ansible-playbooks/docker_ubuntu1804
|
| `PGDATA` | `/var/lib/postgresql/data/pgdata` | `/var/lib/postgresql/18/docker` |
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Customize Options
|
The 18 entrypoint *refuses to start* if it finds a cluster at the old
|
||||||
|
`/var/lib/postgresql/data` path — that check exists so that bumping the image tag
|
||||||
|
without upgrading the database fails loudly instead of appearing to be data loss.
|
||||||
|
Keeping the major version in the path is also what lets a later
|
||||||
|
`pg_upgrade --link` run without mount-boundary problems.
|
||||||
|
|
||||||
```shell
|
`monoco` runs 17.5, so the data comes across as a **logical dump**, not a copy of
|
||||||
nano vars/default.yml
|
`PGDATA`: 17 data files are unreadable by an 18 server. The one-time move is
|
||||||
```
|
[step-by-step here](migrate-from-monoco.md).
|
||||||
|
|
||||||
```yml
|
### No published port
|
||||||
#vars/default.yml
|
|
||||||
---
|
|
||||||
create_containers: 4
|
|
||||||
default_container_name: docker
|
|
||||||
default_container_image: ubuntu
|
|
||||||
default_container_command: sleep 1d
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Run the Playbook
|
There is deliberately no `PublishPort`. The Docker-era playbook published
|
||||||
|
`5432:5432` on `0.0.0.0` with UFW inactive, so the database was reachable from
|
||||||
|
the internet; it needs no host port at all now that clients reach it over the
|
||||||
|
app networks. Verify with `ss -tlnp | grep 5432` — it should print nothing.
|
||||||
|
|
||||||
|
### Credentials
|
||||||
|
|
||||||
|
`pg_password` is still the shared `123456` from the app playbooks, so a migrated
|
||||||
|
app connects unchanged. It is written to `~/podman-data/postgres/postgres.env`
|
||||||
|
mode `0600` and read with `EnvironmentFile=` rather than as `Environment=` lines,
|
||||||
|
because the unit file itself is world-readable in `~/.config`. Note that
|
||||||
|
`podman inspect postgres` still prints the environment to anyone who can run
|
||||||
|
podman as that user; a real secret store is the next step, and is a
|
||||||
|
one-file change here plus the `postgres_password` vars in the app playbooks.
|
||||||
|
|
||||||
|
## Running
|
||||||
|
|
||||||
|
The play targets `emily` directly (`hosts: emily`):
|
||||||
|
|
||||||
```command
|
```command
|
||||||
ansible-playbook -l [target] -i [inventory file] -u [remote user] playbook.yml
|
ansible-playbook -u tanshu postgres/playbook.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
For more information on how to run this Ansible setup, please check this guide: [How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
|
The inventory that defines `emily` lives in `prepare-server/`, whose `ansible.cfg`
|
||||||
|
names it — run from there, or pass an inventory explicitly.
|
||||||
|
|
||||||
|
This play applies **only** the Postgres unit and its data directory. The
|
||||||
|
rootless-Podman prerequisites — installing podman, `loginctl enable-linger`, the
|
||||||
|
low-port sysctl and the Quadlet directory — are the shared task file at
|
||||||
|
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
|
||||||
|
are not run here. A host that has already run `caddy/playbook.yml` has them. To
|
||||||
|
apply them on their own:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: Rootless Podman prerequisites
|
||||||
|
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
|
||||||
|
```
|
||||||
|
|
||||||
|
The three variables those tasks need are duplicated at the top of
|
||||||
|
[vars/default.yml](vars/default.yml) rather than imported, because a play cannot
|
||||||
|
include Caddy's vars file wholesale — it also sets `host_directory`.
|
||||||
|
|
||||||
|
## Operational notes
|
||||||
|
|
||||||
|
```
|
||||||
|
systemctl --user status postgres
|
||||||
|
systemctl --user restart postgres
|
||||||
|
journalctl --user -u postgres -f
|
||||||
|
podman exec -it postgres psql -U postgres
|
||||||
|
podman exec postgres pg_isready -U postgres
|
||||||
|
podman inspect postgres --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The play waits on `pg_isready` inside the container rather than trusting
|
||||||
|
systemd. `Restart=always` means the unit reports `active` even while the
|
||||||
|
container crash-loops, so a wrong `keep-id` (an entrypoint permission error,
|
||||||
|
over and over) would otherwise look like success.
|
||||||
|
|
||||||
|
`systemctl --user enable postgres` is neither needed nor possible: the unit is
|
||||||
|
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.
|
||||||
|
|
||||||
|
### Debugging a unit that will not generate
|
||||||
|
|
||||||
|
```
|
||||||
|
/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
|
||||||
|
systemd-analyze --user --generators=true verify postgres.service
|
||||||
|
```
|
||||||
|
|
||||||
|
The most common cause is a `Network=` naming a `.network` file that does not
|
||||||
|
exist — which is why `attach` checks first.
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
`--link postgres:db` does not exist in Podman, so every `*-backup.sh` that uses
|
||||||
|
it (`db-`, `bitwarden-`, `hiwikijs-`, `kakawa-`, `nextcloud-`, `tandoor-`) is
|
||||||
|
broken on this host. Dump in place instead — no helper container, no networking,
|
||||||
|
no `--user` juggling:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
podman exec -e PGPASSWORD="..." postgres \
|
||||||
|
pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"
|
||||||
|
```
|
||||||
|
|
||||||
|
Whole-cluster form, which is what a migration uses:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
podman exec -e PGPASSWORD="..." postgres pg_dumpall -U postgres > cluster.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
Because the data is owned by `podman_user`, the `sudo tar` / `chown $(logname)`
|
||||||
|
gymnastics in the old scripts can go away too. The database is also the reason
|
||||||
|
root's crontab cannot drive it any more: root cannot `podman exec` into another
|
||||||
|
user's rootless containers, so anything scheduled belongs in a **systemd user
|
||||||
|
timer** next to the Quadlet units.
|
||||||
|
|
||||||
|
## Bringing data from the Docker host
|
||||||
|
|
||||||
|
The one-time move of monoco's cluster — 12 databases, ~900 MB, and a 17 → 18
|
||||||
|
major-version jump — is written out command by command in
|
||||||
|
**[migrate-from-monoco.md](migrate-from-monoco.md)**: dump, transfer, restore,
|
||||||
|
verify, rollback.
|
||||||
|
|
||||||
|
The short version of why it is a dump rather than a file copy:
|
||||||
|
|
||||||
|
- **17 data files are unreadable by an 18 server**, so `PGDATA` cannot simply be
|
||||||
|
copied across the major version;
|
||||||
|
- **the 18 image moved PGDATA** to `/var/lib/postgresql/18/docker` and refuses to
|
||||||
|
start when it finds a cluster at the old path;
|
||||||
|
- for a cluster well under a gigabyte, a logical dump is both the safer and the
|
||||||
|
faster thing to get right, and it re-creates the three roles
|
||||||
|
(`postgres`, `oc_tanshu`, `zitadel`) in the same pass.
|
||||||
|
|
||||||
|
`pg_dumpall` restores each database under its original name with its original
|
||||||
|
grants, so the app playbooks connect unchanged — same container name, same `db`
|
||||||
|
alias, same shared password.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- [ ] `systemctl --user status postgres` is active, and stays active after a reboot
|
||||||
|
- [ ] `podman exec postgres pg_isready -U postgres` reports accepting connections
|
||||||
|
- [ ] `podman exec postgres psql -U postgres -c '\l'` lists the expected databases
|
||||||
|
- [ ] `ss -tlnp | grep 5432` on the host prints nothing
|
||||||
|
- [ ] `ls -ln ~/podman-data/postgres/18/docker` is owned by `tanshu`, not a `165536+` subuid
|
||||||
|
- [ ] `ls -ln ~/podman-data/postgres/18` is owned by `tanshu` too — if it is not, the
|
||||||
|
container created it as subuid and `rm -rf` will fight you
|
||||||
|
- [ ] From each client app's container: `getent hosts postgres` and `getent hosts db`
|
||||||
|
- [ ] Each app can read **and** write — that is the check that catches a wrong `keep-id`
|
||||||
|
- [ ] A `pg_dump` of every database completes and restores into a scratch database
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
---
|
||||||
|
# Attach a service's network to Postgres.
|
||||||
|
#
|
||||||
|
# Postgres is the *base* unit and knows nothing about its clients, the same way
|
||||||
|
# Caddy knows nothing about its upstreams. A service that needs a database adds
|
||||||
|
# itself by dropping files next to postgres.container, and Quadlet merges them in.
|
||||||
|
#
|
||||||
|
# Include this from a service playbook:
|
||||||
|
#
|
||||||
|
# - name: Attach this service to Postgres
|
||||||
|
# ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
|
||||||
|
# vars:
|
||||||
|
# network_name: "{{ network_name }}"
|
||||||
|
#
|
||||||
|
# The including play must:
|
||||||
|
#
|
||||||
|
# 1. have created {{ network_name }}.network already — caddy/roles/register
|
||||||
|
# does that, and every service here that needs a database is also behind
|
||||||
|
# Caddy, so its register include comes first;
|
||||||
|
# 2. run the `podman` role (or include its task file and define the same four
|
||||||
|
# vars), for podman_user and quadlet_dir;
|
||||||
|
# 3. define a "Restart Postgres" handler — see
|
||||||
|
# roles/postgres/handlers/main.yaml for the exact snippet; and
|
||||||
|
# 4. flush handlers before starting the app, on a first attach. This include
|
||||||
|
# notifies that restart, so by default it runs at the end of the play —
|
||||||
|
# after the app has already started and failed to resolve `db`. A
|
||||||
|
# `meta: flush_handlers` immediately after this include is enough; see
|
||||||
|
# tandoor/roles/postgres for the working example.
|
||||||
|
#
|
||||||
|
# It runs as the connection user, which must be the rootless podman_user.
|
||||||
|
#
|
||||||
|
# A service may instead write these two files from its own role; the bytes have
|
||||||
|
# to match this file exactly, or the two will overwrite each other on every run.
|
||||||
|
|
||||||
|
- name: "Check that the {{ network_name }} network unit exists"
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ quadlet_dir }}/{{ network_name }}.network"
|
||||||
|
register: postgres_attach_network
|
||||||
|
|
||||||
|
# A drop-in naming a .network file that does not exist makes Quadlet fail to
|
||||||
|
# generate the whole postgres unit, with a message that does not mention the
|
||||||
|
# drop-in at all. Fail here instead, where the cause is obvious.
|
||||||
|
- name: "Fail if {{ network_name }} has no network unit"
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that: postgres_attach_network.stat.exists
|
||||||
|
fail_msg: >-
|
||||||
|
{{ quadlet_dir }}/{{ network_name }}.network does not exist, so attaching
|
||||||
|
it to Postgres would break the postgres unit. Create the network first
|
||||||
|
(caddy/roles/register does it), then include this file.
|
||||||
|
|
||||||
|
- name: Ensure the Postgres Quadlet drop-in directory exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ quadlet_dir }}/{{ postgres_container | default('postgres') }}.container.d"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
- name: "Attach {{ network_name }} to Postgres"
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: >-
|
||||||
|
{{ quadlet_dir }}/{{ postgres_container | default('postgres') }}.container.d/50-{{ network_name }}.conf
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0644'
|
||||||
|
content: |
|
||||||
|
# Added by the {{ network_name }} playbook. Do not edit by hand.
|
||||||
|
#
|
||||||
|
# Deleting this file removes the network from {{ postgres_container | default('postgres') }} on
|
||||||
|
# the next daemon-reload; nothing else changes.
|
||||||
|
[Container]
|
||||||
|
Network={{ network_name }}.network
|
||||||
|
notify: Restart Postgres
|
||||||
|
|
||||||
|
# NetworkAlias= applies to every network the container joins, which is exactly
|
||||||
|
# right for a hub: every app in this repo reaches postgres as `db` (nextcloud,
|
||||||
|
# gitea, tandoor and outline use the alias; the rest use the container name).
|
||||||
|
# It is a fixed value in one shared file, not a per-service option, so the same
|
||||||
|
# bytes are written no matter how many services attach — two services writing
|
||||||
|
# their own aliases would make Quadlet pass --network-alias twice.
|
||||||
|
- name: "Give {{ postgres_container | default('postgres') }} the shared 'db' alias"
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: >-
|
||||||
|
{{ quadlet_dir }}/{{ postgres_container | default('postgres') }}.container.d/10-network-aliases.conf
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0644'
|
||||||
|
content: |
|
||||||
|
# Shared container aliases for {{ postgres_container | default('postgres') }}.
|
||||||
|
# Managed by Ansible (the service playbooks that use it). Do not edit.
|
||||||
|
[Container]
|
||||||
|
NetworkAlias=db
|
||||||
|
notify: Restart Postgres
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
---
|
||||||
|
# All handlers run as the connection user ({{ podman_user }}), so systemctl
|
||||||
|
# --user works without setting XDG_RUNTIME_DIR.
|
||||||
|
|
||||||
|
- name: Reload systemd user units
|
||||||
|
ansible.builtin.command: systemctl --user daemon-reload
|
||||||
|
changed_when: true
|
||||||
|
listen: Reload systemd user units
|
||||||
|
|
||||||
|
- name: Restart Postgres
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ postgres_unit }}"
|
||||||
|
scope: user
|
||||||
|
state: restarted
|
||||||
|
daemon_reload: true
|
||||||
|
listen: Restart Postgres
|
||||||
|
|
||||||
|
# Reference implementation for service playbooks. A service that attaches its
|
||||||
|
# network with postgres/roles/attach must notify "Restart Postgres", and defines
|
||||||
|
# its own copy of this handler, e.g.:
|
||||||
|
#
|
||||||
|
# - name: Restart Postgres
|
||||||
|
# ansible.builtin.systemd_service:
|
||||||
|
# name: postgres.service
|
||||||
|
# scope: user
|
||||||
|
# state: restarted
|
||||||
|
# daemon_reload: true
|
||||||
|
# listen: Restart Postgres
|
||||||
|
#
|
||||||
|
# A new drop-in changes the generated unit, so Postgres has to be recreated to
|
||||||
|
# join the network — Quadlet has no live equivalent of `podman network connect`.
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
---
|
||||||
|
# Start PostgreSQL
|
||||||
|
#
|
||||||
|
# Everything here runs as {{ podman_user }} — the connection user. No become,
|
||||||
|
# no XDG_RUNTIME_DIR: SSH gave us a real PAM session, so systemd's user manager
|
||||||
|
# is reachable via the standard environment.
|
||||||
|
|
||||||
|
- name: Ensure the Postgres data directory exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ host_directory }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
# PGDATA and its version directory are created here rather than left to the
|
||||||
|
# container's own `mkdir -p`. The entrypoint runs as container-root, which under
|
||||||
|
# keep-id is a subuid on the host, and it only chowns PGDATA and its contents —
|
||||||
|
# so the version directory would be left owned by 165536 and the host user could
|
||||||
|
# not even `rm -rf` it without podman unshare. Creating the whole chain up front
|
||||||
|
# keeps every directory in the data root owned by podman_user.
|
||||||
|
#
|
||||||
|
# Empty PGDATA is exactly right: the 18 entrypoint reads that as "run initdb".
|
||||||
|
- name: Ensure the versioned data directory exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ pgdata_host_dir | dirname }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
- name: Ensure PGDATA exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ pgdata_host_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0700'
|
||||||
|
|
||||||
|
- name: Ensure the Quadlet drop-in directory exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ postgres_dropin_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
# The unit file is 0644 in ~/.config, so the password does not belong in it.
|
||||||
|
# This file is 0600 and is read by the container through EnvironmentFile=.
|
||||||
|
- name: Write the Postgres environment file
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: postgres.env.j2
|
||||||
|
dest: "{{ postgres_env_file }}"
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0600'
|
||||||
|
notify: Restart Postgres
|
||||||
|
|
||||||
|
- name: Install the base Postgres Quadlet unit
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: postgres.container.j2
|
||||||
|
dest: "{{ quadlet_dir }}/{{ postgres_container }}.container"
|
||||||
|
owner: "{{ podman_user }}"
|
||||||
|
group: "{{ podman_user }}"
|
||||||
|
mode: '0644'
|
||||||
|
notify:
|
||||||
|
- Reload systemd user units
|
||||||
|
- Restart Postgres
|
||||||
|
|
||||||
|
# The wait is part of the block on purpose. `Restart=always` means systemd
|
||||||
|
# reports the unit active even while the container crash-loops — a wrong
|
||||||
|
# keep-id shows up as an entrypoint permission error over and over — so
|
||||||
|
# is-active alone would call a broken database healthy.
|
||||||
|
- name: Start Postgres and wait for it to accept connections
|
||||||
|
block:
|
||||||
|
- name: Start Postgres
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ postgres_unit }}"
|
||||||
|
scope: user
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
|
|
||||||
|
# pg_isready is the server's own answer, and it fails while initdb is still
|
||||||
|
# running on a fresh data directory (it exits 2, so the retry loop holds).
|
||||||
|
- name: Wait for Postgres to accept connections
|
||||||
|
ansible.builtin.command: >-
|
||||||
|
podman exec {{ postgres_container }}
|
||||||
|
pg_isready -U {{ pg_user }} -d {{ pg_database }}
|
||||||
|
register: postgres_ready
|
||||||
|
until: postgres_ready.rc == 0
|
||||||
|
retries: 12
|
||||||
|
delay: 5
|
||||||
|
changed_when: false
|
||||||
|
rescue:
|
||||||
|
# systemd reports "control process exited with error code" and nothing
|
||||||
|
# else, which is useless for the usual failures (a bad uid mapping, an
|
||||||
|
# unwritable PGDATA, a missing image). Pull the real reason out.
|
||||||
|
- name: Collect Postgres diagnostics
|
||||||
|
ansible.builtin.command: "{{ item }}"
|
||||||
|
loop:
|
||||||
|
- "systemctl --user status {{ postgres_unit }} --no-pager"
|
||||||
|
- "journalctl --user -u {{ postgres_unit }} --no-pager -n 40"
|
||||||
|
register: postgres_diag
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Show Postgres diagnostics
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: "{{ postgres_diag.results | map(attribute='stdout') | list }}"
|
||||||
|
|
||||||
|
- name: Fail
|
||||||
|
ansible.builtin.fail:
|
||||||
|
msg: "Postgres did not come up — see the diagnostics above."
|
||||||
|
|
||||||
|
# Quadlet honours [Install] WantedBy= when it generates the unit, so
|
||||||
|
# `systemctl enable` is neither needed nor possible on a generated unit.
|
||||||
|
- name: Verify the Postgres unit is active
|
||||||
|
ansible.builtin.command: "systemctl --user is-active {{ postgres_unit }}"
|
||||||
|
register: postgres_active
|
||||||
|
changed_when: false
|
||||||
|
failed_when: postgres_active.stdout != 'active'
|
||||||
|
|
||||||
|
- name: List the services attached to Postgres
|
||||||
|
ansible.builtin.find:
|
||||||
|
paths: "{{ postgres_dropin_dir }}"
|
||||||
|
patterns: "*.conf"
|
||||||
|
register: postgres_dropins
|
||||||
|
|
||||||
|
- name: Report the networks Postgres is attached to
|
||||||
|
ansible.builtin.command: >-
|
||||||
|
podman inspect {{ postgres_container }}
|
||||||
|
--format '{% raw %}{{ range $k, $v := .NetworkSettings.Networks }}{{ $k }} {{ end }}{% endraw %}'
|
||||||
|
register: postgres_networks_now
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Show attached services
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: >-
|
||||||
|
Postgres networks:
|
||||||
|
{{ postgres_networks_now.stdout | default('(container not running)') }}.
|
||||||
|
Attached by:
|
||||||
|
{{ postgres_dropins.files | map(attribute='path') | map('basename') | join(', ')
|
||||||
|
| default('(nothing yet)', true) }}.
|
||||||
|
|
||||||
|
# Say this out loud rather than let it be discovered by a failing app: with no
|
||||||
|
# drop-in, Postgres has no Network= at all, so it sits on the default Podman
|
||||||
|
# network and no app network can resolve it.
|
||||||
|
- name: Note that no service has attached yet
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: >-
|
||||||
|
No service has attached to Postgres. It is on the default Podman network
|
||||||
|
only, and no app can resolve it until a service playbook includes
|
||||||
|
postgres/roles/attach (see readme.md).
|
||||||
|
when: postgres_dropins.matched == 0
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# PostgreSQL — rootless Podman Quadlet unit.
|
||||||
|
# Managed by Ansible (postgres/roles/postgres). Do not edit by hand.
|
||||||
|
#
|
||||||
|
# Postgres deliberately does NOT list the app networks it serves. Each service
|
||||||
|
# attaches itself by dropping a file into:
|
||||||
|
#
|
||||||
|
# {{ postgres_dropin_dir }}/
|
||||||
|
#
|
||||||
|
# which Quadlet merges into this unit. Removing a service therefore never means
|
||||||
|
# editing this file. See readme.md, "How services attach themselves".
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=PostgreSQL database
|
||||||
|
Documentation=https://hub.docker.com/_/postgres
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Container]
|
||||||
|
ContainerName={{ postgres_container }}
|
||||||
|
Image={{ postgres_image }}
|
||||||
|
|
||||||
|
# Map the container's postgres account (uid {{ postgres_uid }}) onto
|
||||||
|
# {{ podman_user }} on the host, so PGDATA stays owned by {{ podman_user }} and
|
||||||
|
# never by a subuid. The official entrypoint starts as root, chowns PGDATA to
|
||||||
|
# uid {{ postgres_uid }}, then drops privileges — so everything it writes lands
|
||||||
|
# on {{ podman_user }}, and plain tar / pg_dump backups keep working.
|
||||||
|
UserNS=keep-id:uid={{ postgres_uid }},gid={{ postgres_gid }}
|
||||||
|
|
||||||
|
# No PublishPort, on purpose. The Docker-era playbook published 5432 on
|
||||||
|
# 0.0.0.0 (and a UFW rule was never there to narrow it); here Postgres is
|
||||||
|
# reachable only from the app networks that attach themselves to it.
|
||||||
|
#
|
||||||
|
# The mount is the image's VOLUME, which in 18 is the *parent* directory — the
|
||||||
|
# major version is in the path instead ({{ pgdata }}). Mounting
|
||||||
|
# /var/lib/postgresql/data here instead would make the 18 entrypoint refuse to
|
||||||
|
# start, because that path is where it looks for an un-upgraded 17 cluster.
|
||||||
|
Volume={{ host_directory }}:{{ postgres_mount }}
|
||||||
|
|
||||||
|
# POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB / PGDATA live in this 0600
|
||||||
|
# file rather than in Environment= lines here: this unit is world-readable.
|
||||||
|
EnvironmentFile={{ postgres_env_file }}
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Restart=always
|
||||||
|
# The first start runs initdb, and a database needs time to shut down cleanly.
|
||||||
|
TimeoutStartSec={{ postgres_timeout_start }}
|
||||||
|
TimeoutStopSec={{ postgres_timeout_stop }}
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# Environment for postgres.container.
|
||||||
|
# Managed by Ansible (postgres/roles/postgres). Do not edit by hand.
|
||||||
|
#
|
||||||
|
# Mode 0600 on purpose: the Quadlet unit that reads this file is world-readable,
|
||||||
|
# so the password must not live there. Note that `podman inspect` still prints
|
||||||
|
# the resulting environment to anyone who can run podman as this user.
|
||||||
|
POSTGRES_USER={{ pg_user }}
|
||||||
|
POSTGRES_PASSWORD={{ pg_password }}
|
||||||
|
POSTGRES_DB={{ pg_database }}
|
||||||
|
PGDATA={{ pgdata }}
|
||||||
@@ -1,8 +1,94 @@
|
|||||||
---
|
---
|
||||||
postgres_name: postgres
|
# ---------------------------------------------------------------------------
|
||||||
postgres_image: postgres:17-alpine
|
# Rootless Podman / Quadlet
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Duplicated on purpose from caddy/vars/default.yml: a play cannot include that
|
||||||
|
# file wholesale, because it also sets `host_directory` — to the Caddy data
|
||||||
|
# directory — and every playbook defines that for itself. Keep these three in
|
||||||
|
# sync with caddy/vars/default.yml. (Its fourth, podman_unprivileged_port_start,
|
||||||
|
# belongs to the podman tasks, which this play does not run: Postgres listens on
|
||||||
|
# 5432 and publishes nothing.)
|
||||||
|
#
|
||||||
|
# podman_user is derived from the connection: this play carries no become, so
|
||||||
|
# the Quadlet units belong to whoever Ansible connects as. Requires gather_facts
|
||||||
|
# (set in playbook.yml).
|
||||||
|
podman_user: "{{ ansible_facts['user_id'] }}"
|
||||||
|
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
|
||||||
|
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# PostgreSQL
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The name the rest of the repo already uses for this container in its
|
||||||
|
# `postgres_container` vars, and the name every app resolves over the network.
|
||||||
|
postgres_container: postgres
|
||||||
|
postgres_unit: "{{ postgres_container }}.service"
|
||||||
|
|
||||||
|
# The major version is the one knob, because the image tag and the PGDATA
|
||||||
|
# directory have to agree.
|
||||||
|
#
|
||||||
|
# 18 is deliberate. monoco runs 17.5 and the data is brought across with a
|
||||||
|
# logical dump — a PGDATA copy cannot cross a major version. See
|
||||||
|
# migrate-from-monoco.md for the one-time move.
|
||||||
|
#
|
||||||
|
# `-alpine` still floats within 18.x, which is what you want for security
|
||||||
|
# patches; pin a digest here if the image itself must be immutable.
|
||||||
|
postgres_major: '18'
|
||||||
|
|
||||||
|
# Fully qualified on purpose: rootless Podman resolves short names against
|
||||||
|
# unqualified-search-registries, and there is no TTY to answer a prompt when
|
||||||
|
# the systemd unit starts.
|
||||||
|
postgres_image: "docker.io/library/postgres:{{ postgres_major }}-alpine"
|
||||||
|
|
||||||
|
# The uid the server actually writes as — not the image's configured user. The
|
||||||
|
# Alpine image creates the "postgres" account at uid/gid 70 ("70 is the standard
|
||||||
|
# uid/gid for postgres in Alpine"), and the entrypoint starts as root, chowns
|
||||||
|
# PGDATA to it, then drops privileges. keep-id maps that uid onto podman_user,
|
||||||
|
# so PGDATA ends up owned by podman_user on the host and never by a subuid. That
|
||||||
|
# is what keeps backups working without podman unshare or sudo.
|
||||||
|
postgres_uid: 70
|
||||||
|
postgres_gid: 70
|
||||||
|
|
||||||
|
host_directory: "{{ podman_data_dir }}/{{ postgres_container }}"
|
||||||
|
|
||||||
|
# Postgres 18 moved both the image's VOLUME and its default PGDATA:
|
||||||
|
#
|
||||||
|
# 17 Volume /var/lib/postgresql/data PGDATA /var/lib/postgresql/data/pgdata
|
||||||
|
# 18+ Volume /var/lib/postgresql PGDATA /var/lib/postgresql/<major>/docker
|
||||||
|
#
|
||||||
|
# So the mount is now the parent directory and the version lives in the path.
|
||||||
|
# That is what the 18 entrypoint expects: it refuses to start — "there appears
|
||||||
|
# to be PostgreSQL data in ..." — if it finds a cluster in the old layout, and
|
||||||
|
# the versioned subdirectory is what makes a later `pg_upgrade --link` work
|
||||||
|
# without mount-boundary trouble.
|
||||||
|
postgres_mount: /var/lib/postgresql
|
||||||
|
|
||||||
|
# Everything under the mount that identifies this cluster, kept as one value so
|
||||||
|
# the container path and the host path cannot drift apart.
|
||||||
|
postgres_versioned_subdir: "{{ postgres_major }}/docker"
|
||||||
|
|
||||||
|
pgdata: "{{ postgres_mount }}/{{ postgres_versioned_subdir }}"
|
||||||
|
pgdata_host_dir: "{{ host_directory }}/{{ postgres_versioned_subdir }}"
|
||||||
|
|
||||||
|
# Secrets do not go in the unit file: it is world-readable, and `podman inspect`
|
||||||
|
# prints the environment anyway. They go in a 0600 file the unit reads with
|
||||||
|
# EnvironmentFile=.
|
||||||
|
postgres_env_file: "{{ host_directory }}/postgres.env"
|
||||||
|
|
||||||
|
# Quadlet drop-in directory for the Postgres unit. Services attach themselves
|
||||||
|
# here; see the "How services attach themselves" section of readme.md.
|
||||||
|
# Postgres itself never lists its app networks.
|
||||||
|
postgres_dropin_dir: "{{ quadlet_dir }}/{{ postgres_container }}.container.d"
|
||||||
|
|
||||||
|
# Credentials. This is the placeholder the app playbooks already embed in their
|
||||||
|
# db_url / POSTGRES_PASSWORD, kept as-is so a migrated app connects unchanged.
|
||||||
|
# monoco's cluster uses the same one, so the dump restores and the apps connect
|
||||||
|
# with no password reset. Replace it here and in those playbooks together when
|
||||||
|
# the fleet moves to a real secret.
|
||||||
pg_user: postgres
|
pg_user: postgres
|
||||||
pg_password: '123456'
|
pg_password: '123456'
|
||||||
data_location: /var/lib/postgresql/data
|
pg_database: postgres
|
||||||
|
|
||||||
host_directory: postgresql
|
# The first start runs initdb, and a database wants time to shut down cleanly.
|
||||||
|
postgres_timeout_start: 120
|
||||||
|
postgres_timeout_stop: 60
|
||||||
Reference in new issue
Block a user