playbooks/postgres/readme.md

PostgreSQL (rootless Podman + Quadlet)

Runs PostgreSQL as a rootless Podman container managed by systemd user units (Quadlet). Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept its data in /var/lib/postgresql.

Postgres is the base unit and knows nothing about its clients. It does not list any app network and does not need editing when a service is added or removed. Each service attaches itself. That mirrors caddy, which is the reverse proxy end of the same idea.

Layout

postgres/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/   # the base postgres.container unit, env file, handlers
    └── attach/     # includable task file: a service attaches itself to Postgres

Everything is installed under {{ quadlet_dir }} (~/.config/containers/systemd/). State lives under {{ host_directory }} (~/podman-data/postgres/).

How services attach themselves

Postgres joins the app network of every service that needs a database. A service joins by writing one Quadlet drop-in next to postgres.container:

~/.config/containers/systemd/
├── postgres.container                  # base unit, no app networks
├── postgres.container.d/               # ← drop-ins, one per client service
│   ├── 10-network-aliases.conf         # [Container] NetworkAlias=db (shared)
│   ├── 50-nextcloud.conf               # [Container] Network=nextcloud.network
│   ├── 50-gitea.conf
│   └── 50-tandoor.conf
├── nextcloud.network
├── gitea.network
└── tandoor.network

A service playbook does both halves — its own network, then the attach:

- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

- name: Attach this service to Postgres
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

The play must also run the podman role (for podman_user and quadlet_dir) and define a Restart Postgres handler — see roles/postgres/handlers/main.yaml for the exact snippet.

Order matters in one direction only: the attach include needs <name>.network to already exist, which is why Caddy's register comes first. It fails with a readable message if it does not, because a drop-in naming a missing .network file makes Quadlet reject the whole postgres unit with an error that never mentions the drop-in.

Names: postgres and db

Both resolve on every attached network. postgres comes from ContainerName=postgres; db is added by 10-network-aliases.conf. The alias exists because the Docker-era playbooks are not consistent — nextcloud, gitea, tandoor and outline connect to db, the rest to postgres.

NetworkAlias= applies to every network the container joins, which is exactly what a hub wants. It lives in one shared drop-in with a fixed value rather than in each service's 50-*.conf, for two reasons:

• it is not per-service — if one service could set a different value, the shared file would be rewritten on every run, and every service writes the same bytes;
• with the alias in each service's own file, two attached services would make Quadlet pass --network-alias db twice.

It is not in postgres.container itself for a duller reason: on a fresh host, before anything has attached, the unit has no Network= at all, and asking Podman for an alias on the default network is not a configuration worth depending on.

One way to attach, and one rule

roles/attach is the shared include, and is what every service uses. A service may instead write the same two files from its own role, but then it must write identical bytes on purpose: the drop-ins live in one directory, so anything less means whichever play runs last rewrites the other's file on every run. If you add a second form, make it match.

Whichever form you use, flush handlers before the app starts on a first attach. The restart that joins Postgres to the new network is a handler, so by default it runs at the end of the play — after the app has already come up and failed to resolve db. tandoor/roles/postgres is the working example.

Why drop-ins rather than listing the networks

Listing app networks in postgres.container would mean editing this playbook every time a service is added, and again on removal. Drop-ins survive re-templating of the base unit, are the mechanism Quadlet documents for this, and make removal trivial: delete 50-tandoor.conf, daemon-reload, done.

The trade-off

A new drop-in changes the generated unit, so Postgres is recreated to join the network — a few seconds of downtime for every connected app, not just the new one. Quadlet has no live equivalent of docker network connect. If that matters, podman network connect <net> postgres takes effect immediately, and the drop-in makes it survive the next recreate; the attach role does not do this by default because it would make the play non-idempotent.

Why not one shared database network

Every app on a shared postgres_net could reach every other app on it. The generated <name>.network units set Options=isolate=true, so keeping Postgres as the only common member of otherwise separate networks is what preserves the per-app isolation Caddy's networks give you.

Who runs as whom

UserNS=keep-id:uid=70,gid=70. 70 is the uid the server writes as: the official Alpine entrypoint starts as root, chowns PGDATA to the postgres account, and drops privileges, so docker exec postgres id -u reporting 0 is misleading. keep-id maps that one uid onto podman_user, which means:

• the whole data root — ~/podman-data/postgres/18/docker and the version directory above it — is owned by podman_user, never by a subuid;
• plain tar, pg_dump, cp and rm -rf work without podman unshare and without sudo;
• the reverse — chowning the data to podman_user before first start — is also correct, which is what makes bringing data across simple.

PGDATA and its version directory are created by the role rather than left to the container's own mkdir -p. The entrypoint runs as container-root — a subuid under keep-id — and only chowns PGDATA and its contents, so the version directory would be stranded as subuid-owned: readable but not removable by podman_user. Creating the chain up front is what makes the bullets above true.

Like Caddy, podman_user is derived from the connection (ansible_facts['user_id']) and the play carries no become. This playbook does not run the podman role, so it has no guard against connecting as root; the prerequisites are applied separately (see Running).

Settings

Variable	Purpose
podman_user, podman_data_dir, quadlet_dir	Rootless-Podman contract, duplicated from caddy/vars/default.yml
postgres_container	Container / unit name, and the prefix of the drop-in directory
postgres_major	The one knob: pins both the image tag and the PGDATA path
postgres_image	postgres:<major>-alpine, fully qualified on purpose (see below)
postgres_uid / postgres_gid	The uid the server writes as — 70 in the Alpine image
host_directory	Where state lives (~/podman-data/postgres)
postgres_mount / pgdata	The image's VOLUME, and PGDATA inside it (/var/lib/postgresql, /var/lib/postgresql/18/docker)
postgres_env_file	0600 file holding the credentials, read via EnvironmentFile=
postgres_dropin_dir	Where services attach their networks
pg_user / pg_password / pg_database	Credentials and the default database
postgres_timeout_start / postgres_timeout_stop	initdb on first start, clean shutdown on stop

Image and PGDATA layout

docker.io/library/postgres:18-alpine — fully qualified because rootless Podman resolves short names against unqualified-search-registries, and a Quadlet unit has no TTY to answer a short-name prompt. -alpine still floats within 18.x for security patches.

18 moved both the image's VOLUME and its default PGDATA, which is why postgres_major exists and why the mount is a directory rather than the data directory itself:

	17	18+
VOLUME	/var/lib/postgresql/data	/var/lib/postgresql
PGDATA	/var/lib/postgresql/data/pgdata	/var/lib/postgresql/18/docker

The 18 entrypoint refuses to start if it finds a cluster at the old /var/lib/postgresql/data path — that check exists so that bumping the image tag without upgrading the database fails loudly instead of appearing to be data loss. Keeping the major version in the path is also what lets a later pg_upgrade --link run without mount-boundary problems.

monoco runs 17.5, so the data comes across as a logical dump, not a copy of PGDATA: 17 data files are unreadable by an 18 server. The one-time move is step-by-step here.

No published port

There is deliberately no PublishPort. The Docker-era playbook published 5432:5432 on 0.0.0.0 with UFW inactive, so the database was reachable from the internet; it needs no host port at all now that clients reach it over the app networks. Verify with ss -tlnp | grep 5432 — it should print nothing.

Credentials

pg_password is still the shared 123456 from the app playbooks, so a migrated app connects unchanged. It is written to ~/podman-data/postgres/postgres.env mode 0600 and read with EnvironmentFile= rather than as Environment= lines, because the unit file itself is world-readable in ~/.config. Note that podman inspect postgres still prints the environment to anyone who can run podman as that user; a real secret store is the next step, and is a one-file change here plus the postgres_password vars in the app playbooks.

Running

The play targets emily directly (hosts: emily):

ansible-playbook -u tanshu postgres/playbook.yml

The inventory that defines emily lives in prepare-server/, whose ansible.cfg names it — run from there, or pass an inventory explicitly.

This play applies only the Postgres unit and its data directory. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them. To apply them on their own:

- name: Rootless Podman prerequisites
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"

The three variables those tasks need are duplicated at the top of vars/default.yml rather than imported, because a play cannot include Caddy's vars file wholesale — it also sets host_directory.

Operational notes

systemctl --user status postgres
systemctl --user restart postgres
journalctl --user -u postgres -f
podman exec -it postgres psql -U postgres
podman exec postgres pg_isready -U postgres
podman inspect postgres --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'

The play waits on pg_isready inside the container rather than trusting systemd. Restart=always means the unit reports active even while the container crash-loops, so a wrong keep-id (an entrypoint permission error, over and over) would otherwise look like success.

systemctl --user enable postgres is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Debugging a unit that will not generate

/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
systemd-analyze --user --generators=true verify postgres.service

The most common cause is a Network= naming a .network file that does not exist — which is why attach checks first.

Backups

--link postgres:db does not exist in Podman, so every *-backup.sh that uses it (db-, bitwarden-, hiwikijs-, kakawa-, nextcloud-, tandoor-) is broken on this host. Dump in place instead — no helper container, no networking, no --user juggling:

podman exec -e PGPASSWORD="..." postgres \
  pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"

Whole-cluster form, which is what a migration uses:

podman exec -e PGPASSWORD="..." postgres pg_dumpall -U postgres > cluster.sql

Because the data is owned by podman_user, the sudo tar / chown $(logname) gymnastics in the old scripts can go away too. The database is also the reason root's crontab cannot drive it any more: root cannot podman exec into another user's rootless containers, so anything scheduled belongs in a systemd user timer next to the Quadlet units.

Bringing data from the Docker host

The one-time move of monoco's cluster — 12 databases, ~900 MB, and a 17 → 18 major-version jump — is written out command by command in migrate-from-monoco.md: dump, transfer, restore, verify, rollback.

The short version of why it is a dump rather than a file copy:

• 17 data files are unreadable by an 18 server, so PGDATA cannot simply be copied across the major version;
• the 18 image moved PGDATA to /var/lib/postgresql/18/docker and refuses to start when it finds a cluster at the old path;
• for a cluster well under a gigabyte, a logical dump is both the safer and the faster thing to get right, and it re-creates the three roles (postgres, oc_tanshu, zitadel) in the same pass.

pg_dumpall restores each database under its original name with its original grants, so the app playbooks connect unchanged — same container name, same db alias, same shared password.

Verification

• systemctl --user status postgres is active, and stays active after a reboot
• podman exec postgres pg_isready -U postgres reports accepting connections
• podman exec postgres psql -U postgres -c '\l' lists the expected databases
• ss -tlnp | grep 5432 on the host prints nothing
• ls -ln ~/podman-data/postgres/18/docker is owned by tanshu, not a 165536+ subuid
• ls -ln ~/podman-data/postgres/18 is owned by tanshu too — if it is not, the container created it as subuid and rm -rf will fight you
• From each client app's container: getent hosts postgres and getent hosts db
• Each app can read and write — that is the check that catches a wrong keep-id
• A pg_dump of every database completes and restores into a scratch database