playbooks/postgres/readme.md

# PostgreSQL (rootless Podman + Quadlet)

Runs PostgreSQL as a **rootless Podman** container managed by **systemd user
units** (Quadlet). Replaces the `docker_container` playbook, which published
`5432` on `0.0.0.0` and kept its data in `/var/lib/postgresql`.

**Postgres is the base unit and knows nothing about its clients.** It does not
list any app network and does not need editing when a service is added or
removed. Each service attaches *itself*. That mirrors [caddy](../caddy/readme.md),
which is the reverse proxy end of the same idea.

## Layout

```
postgres/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/   # the base postgres.container unit, env file, handlers
    └── attach/     # includable task file: a service attaches itself to Postgres
```

Everything is installed under `{{ quadlet_dir }}`
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
(`~/podman-data/postgres/`).

## How services attach themselves

Postgres joins the app network of every service that needs a database. A service
joins by writing one Quadlet *drop-in* next to `postgres.container`:

```
~/.config/containers/systemd/
├── postgres.container                  # base unit, no app networks
├── postgres.container.d/               # ← drop-ins, one per client service
│   ├── 10-network-aliases.conf         # [Container] NetworkAlias=db (shared)
│   ├── 50-nextcloud.conf               # [Container] Network=nextcloud.network
│   ├── 50-gitea.conf
│   └── 50-tandoor.conf
├── nextcloud.network
├── gitea.network
└── tandoor.network
```

A service playbook does both halves — its own network, then the attach:

```yaml
- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

- name: Attach this service to Postgres
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"
```

The play must also run the `podman` role (for `podman_user` and `quadlet_dir`)
and define a `Restart Postgres` handler — see
[roles/postgres/handlers/main.yaml](roles/postgres/handlers/main.yaml) for the
exact snippet.

Order matters in one direction only: the `attach` include needs
`<name>.network` to already exist, which is why Caddy's `register` comes first.
It fails with a readable message if it does not, because a drop-in naming a
missing `.network` file makes Quadlet reject the *whole* postgres unit with an
error that never mentions the drop-in.

### Names: `postgres` and `db`

Both resolve on every attached network. `postgres` comes from
`ContainerName=postgres`; `db` is added by `10-network-aliases.conf`. The alias
exists because the Docker-era playbooks are not consistent — nextcloud, gitea,
tandoor and outline connect to `db`, the rest to `postgres`.

`NetworkAlias=` applies to *every* network the container joins, which is exactly
what a hub wants. It lives in one shared drop-in with a fixed value rather than
in each service's `50-*.conf`, for two reasons:

- it is not per-service — if one service could set a different value, the shared
  file would be rewritten on every run, and every service writes the same bytes;
- with the alias in each service's own file, two attached services would make
  Quadlet pass `--network-alias db` twice.

It is *not* in `postgres.container` itself for a duller reason: on a fresh host,
before anything has attached, the unit has no `Network=` at all, and asking
Podman for an alias on the default network is not a configuration worth
depending on.

### One way to attach, and one rule

[roles/attach](roles/attach/tasks/main.yaml) is the shared include, and is what
every service uses. A service may instead write the same two files from its own
role, but then it **must write identical bytes** on purpose: the drop-ins live in
one directory, so anything less means whichever play runs last rewrites the
other's file on every run. If you add a second form, make it match.

Whichever form you use, **flush handlers before the app starts on a first
attach.** The restart that joins Postgres to the new network is a handler, so by
default it runs at the end of the play — after the app has already come up and
failed to resolve `db`. `tandoor/roles/postgres` is the working example.

### Why drop-ins rather than listing the networks

Listing app networks in `postgres.container` would mean editing this playbook
every time a service is added, and again on removal. Drop-ins survive
re-templating of the base unit, are the mechanism Quadlet documents for this,
and make removal trivial: delete `50-tandoor.conf`, `daemon-reload`, done.

### The trade-off

A new drop-in changes the generated unit, so Postgres is **recreated** to join
the network — a few seconds of downtime for *every* connected app, not just the
new one. Quadlet has no live equivalent of `docker network connect`. If that
matters, `podman network connect <net> postgres` takes effect immediately, and
the drop-in makes it survive the next recreate; the `attach` role does not do
this by default because it would make the play non-idempotent.

### Why not one shared database network

Every app on a shared `postgres_net` could reach every other app on it. The
generated `<name>.network` units set `Options=isolate=true`, so keeping Postgres
as the only common member of otherwise separate networks is what preserves the
per-app isolation Caddy's networks give you.

## Who runs as whom

`UserNS=keep-id:uid=70,gid=70`. `70` is the uid the *server* writes as: the
official Alpine entrypoint starts as root, chowns PGDATA to the `postgres`
account, and drops privileges, so `docker exec postgres id -u` reporting `0` is
misleading. `keep-id` maps that one uid onto `podman_user`, which means:

- the whole data root — `~/podman-data/postgres/18/docker` and the version
  directory above it — is owned by `podman_user`, never by a subuid;
- plain `tar`, `pg_dump`, `cp` and `rm -rf` work without `podman unshare` and
  without `sudo`;
- the reverse — chowning the data to `podman_user` before first start — is also
  correct, which is what makes bringing data across simple.

PGDATA and its version directory are created by the role rather than left to the
container's own `mkdir -p`. The entrypoint runs as container-root — a subuid
under `keep-id` — and only chowns PGDATA and its contents, so the version
directory would be stranded as subuid-owned: readable but not removable by
`podman_user`. Creating the chain up front is what makes the bullets above true.

Like Caddy, `podman_user` is derived from the connection
(`ansible_facts['user_id']`) and the play carries no `become`. This playbook does
not run the podman role, so it has no guard against connecting as `root`; the
prerequisites are applied separately (see Running).

## Settings

| Variable | Purpose |
|---|---|
| `podman_user`, `podman_data_dir`, `quadlet_dir` | Rootless-Podman contract, duplicated from [caddy/vars/default.yml](../caddy/vars/default.yml) |
| `postgres_container` | Container / unit name, and the prefix of the drop-in directory |
| `postgres_major` | The one knob: pins both the image tag and the PGDATA path |
| `postgres_image` | `postgres:<major>-alpine`, fully qualified on purpose (see below) |
| `postgres_uid` / `postgres_gid` | The uid the server writes as — `70` in the Alpine image |
| `host_directory` | Where state lives (`~/podman-data/postgres`) |
| `postgres_mount` / `pgdata` | The image's VOLUME, and PGDATA inside it (`/var/lib/postgresql`, `/var/lib/postgresql/18/docker`) |
| `postgres_env_file` | `0600` file holding the credentials, read via `EnvironmentFile=` |
| `postgres_dropin_dir` | Where services attach their networks |
| `pg_user` / `pg_password` / `pg_database` | Credentials and the default database |
| `postgres_timeout_start` / `postgres_timeout_stop` | initdb on first start, clean shutdown on stop |

### Image and PGDATA layout

`docker.io/library/postgres:18-alpine` — fully qualified because rootless Podman
resolves short names against `unqualified-search-registries`, and a Quadlet unit
has no TTY to answer a short-name prompt. `-alpine` still floats within 18.x for
security patches.

**18 moved both the image's VOLUME and its default PGDATA**, which is why
`postgres_major` exists and why the mount is a *directory* rather than the data
directory itself:

| | 17 | 18+ |
|---|---|---|
| `VOLUME` | `/var/lib/postgresql/data` | `/var/lib/postgresql` |
| `PGDATA` | `/var/lib/postgresql/data/pgdata` | `/var/lib/postgresql/18/docker` |

The 18 entrypoint *refuses to start* if it finds a cluster at the old
`/var/lib/postgresql/data` path — that check exists so that bumping the image tag
without upgrading the database fails loudly instead of appearing to be data loss.
Keeping the major version in the path is also what lets a later
`pg_upgrade --link` run without mount-boundary problems.

`monoco` runs 17.5, so the data comes across as a **logical dump**, not a copy of
`PGDATA`: 17 data files are unreadable by an 18 server. The one-time move is
[step-by-step here](migrate-from-monoco.md).

### No published port

There is deliberately no `PublishPort`. The Docker-era playbook published
`5432:5432` on `0.0.0.0` with UFW inactive, so the database was reachable from
the internet; it needs no host port at all now that clients reach it over the
app networks. Verify with `ss -tlnp | grep 5432` — it should print nothing.

### Credentials

`pg_password` is still the shared `123456` from the app playbooks, so a migrated
app connects unchanged. It is written to `~/podman-data/postgres/postgres.env`
mode `0600` and read with `EnvironmentFile=` rather than as `Environment=` lines,
because the unit file itself is world-readable in `~/.config`. Note that
`podman inspect postgres` still prints the environment to anyone who can run
podman as that user; a real secret store is the next step, and is a
one-file change here plus the `postgres_password` vars in the app playbooks.

## Running

The play targets `emily` directly (`hosts: emily`):

```command
ansible-playbook -u tanshu postgres/playbook.yml
```

The inventory that defines `emily` lives in `prepare-server/`, whose `ansible.cfg`
names it — run from there, or pass an inventory explicitly.

This play applies **only** the Postgres unit and its data directory. The
rootless-Podman prerequisites — installing podman, `loginctl enable-linger`, the
low-port sysctl and the Quadlet directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are not run here. A host that has already run `caddy/playbook.yml` has them. To
apply them on their own:

```yaml
- name: Rootless Podman prerequisites
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
```

The three variables those tasks need are duplicated at the top of
[vars/default.yml](vars/default.yml) rather than imported, because a play cannot
include Caddy's vars file wholesale — it also sets `host_directory`.

## Operational notes

```
systemctl --user status postgres
systemctl --user restart postgres
journalctl --user -u postgres -f
podman exec -it postgres psql -U postgres
podman exec postgres pg_isready -U postgres
podman inspect postgres --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
```

The play waits on `pg_isready` inside the container rather than trusting
systemd. `Restart=always` means the unit reports `active` even while the
container crash-loops, so a wrong `keep-id` (an entrypoint permission error,
over and over) would otherwise look like success.

`systemctl --user enable postgres` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

### Debugging a unit that will not generate

```
/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
systemd-analyze --user --generators=true verify postgres.service
```

The most common cause is a `Network=` naming a `.network` file that does not
exist — which is why `attach` checks first.

## Backups

`--link postgres:db` does not exist in Podman, so every `*-backup.sh` that uses
it (`db-`, `bitwarden-`, `hiwikijs-`, `kakawa-`, `nextcloud-`, `tandoor-`) is
broken on this host. Dump in place instead — no helper container, no networking,
no `--user` juggling:

```bash
podman exec -e PGPASSWORD="..." postgres \
  pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"
```

Whole-cluster form, which is what a migration uses:

```bash
podman exec -e PGPASSWORD="..." postgres pg_dumpall -U postgres > cluster.sql
```

Because the data is owned by `podman_user`, the `sudo tar` / `chown $(logname)`
gymnastics in the old scripts can go away too. The database is also the reason
root's crontab cannot drive it any more: root cannot `podman exec` into another
user's rootless containers, so anything scheduled belongs in a **systemd user
timer** next to the Quadlet units.

## Bringing data from the Docker host

The one-time move of monoco's cluster — 12 databases, ~900 MB, and a 17 → 18
major-version jump — is written out command by command in
**[migrate-from-monoco.md](migrate-from-monoco.md)**: dump, transfer, restore,
verify, rollback.

The short version of why it is a dump rather than a file copy:

- **17 data files are unreadable by an 18 server**, so `PGDATA` cannot simply be
  copied across the major version;
- **the 18 image moved PGDATA** to `/var/lib/postgresql/18/docker` and refuses to
  start when it finds a cluster at the old path;
- for a cluster well under a gigabyte, a logical dump is both the safer and the
  faster thing to get right, and it re-creates the three roles
  (`postgres`, `oc_tanshu`, `zitadel`) in the same pass.

`pg_dumpall` restores each database under its original name with its original
grants, so the app playbooks connect unchanged — same container name, same `db`
alias, same shared password.

## Verification

- [ ] `systemctl --user status postgres` is active, and stays active after a reboot
- [ ] `podman exec postgres pg_isready -U postgres` reports accepting connections
- [ ] `podman exec postgres psql -U postgres -c '\l'` lists the expected databases
- [ ] `ss -tlnp | grep 5432` on the host prints nothing
- [ ] `ls -ln ~/podman-data/postgres/18/docker` is owned by `tanshu`, not a `165536+` subuid
- [ ] `ls -ln ~/podman-data/postgres/18` is owned by `tanshu` too — if it is not, the
      container created it as subuid and `rm -rf` will fight you
- [ ] From each client app's container: `getent hosts postgres` and `getent hosts db`
- [ ] Each app can read **and** write — that is the check that catches a wrong `keep-id`
- [ ] A `pg_dump` of every database completes and restores into a scratch database