tanshu/playbooks
1
0 Fork 0
Code Issues Pull Requests Releases Wiki Activity

Run Postgres as a rootless Podman Quadlet, and move the 17 cluster across

Browse Source 
Replaces the docker_container playbook, which published 5432 on 0.0.0.0 and kept
its data in /var/lib/postgresql. Postgres is now the base Quadlet unit, the same
shape as Caddy: it knows nothing about its clients, and a service attaches itself
by dropping a file into postgres.container.d/ via postgres/roles/attach. Nothing
here changes when a service is added or removed.

Pinned to 18, which is two decisions rather than one tag. Postgres 18 moved both
the image's VOLUME (/var/lib/postgresql/data -> /var/lib/postgresql) and its
default PGDATA (/var/lib/postgresql/data/pgdata -> /var/lib/postgresql/18/docker),
and the 18 entrypoint refuses to start when it finds a cluster in the old layout.
So postgres_major is the single knob: it picks the image tag and the PGDATA path
together, and the mount is now the parent directory. That also settles the
migration: a PGDATA copy cannot cross a major version, so monoco's 17.5 cluster
comes across as a pg_dumpall.

Three things that bit us and are worth keeping:

- PGDATA and its version directory are created by the role, not left to the
  container. The entrypoint runs as container-root, which under keep-id is a
  subuid on the host, and it only chowns PGDATA and its contents -- so
  ~/podman-data/postgres/18 would have been stranded subuid-owned, and the host
  user could not even rm -rf it without podman unshare.

- NetworkAlias= applies to every network the container joins, which is what a
  database hub wants, but it is one fixed value in a shared drop-in rather than
  per-service: two services writing their own would make Quadlet pass
  --network-alias twice. It is not in postgres.container itself because on a
  fresh host that unit has no Network= at all.

- The credentials live in a 0600 file read with EnvironmentFile=, not in
  Environment= lines: the unit is world-readable in ~/.config.

The play no longer uses become, and does not apply the rootless-Podman
prerequisites -- those are the shared task file at caddy/roles/podman, and the
three variables they need are duplicated at the top of vars/default.yml.

migrate-from-monoco.md records the one-time 17.5 -> 18.6 move: the dump, transfer
and restore commands, the checks that prove the restore is complete (per-database
table counts against the source), and the verified result.

The dead nginx/php-fpm templates in postgres/files/ are removed.
This commit is contained in:
tanshu committed 2026-10-16 06:52:27 +05:30
1 parent 060173db5e
commit 4297e5a71e
11 files changed
+1079 -88
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

postgres/readme.md
+307 -30
View File
@@ -1,46 +1,323 @@
# Docker on Ubuntu 18.04
# PostgreSQL (rootless Podman + Quadlet)
This playbook will install Docker an Ubuntu 18.04 machine, as explained in the guide on
[How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
A number of containers will be created with the options specified in the `vars/default.yml` variable file.
Runs PostgreSQL as a **rootless Podman** container managed by **systemd user
units** (Quadlet). Replaces the `docker_container` playbook, which published
`5432` on `0.0.0.0` and kept its data in `/var/lib/postgresql`.
**Postgres is the base unit and knows nothing about its clients.** It does not
list any app network and does not need editing when a service is added or
removed. Each service attaches *itself*. That mirrors [caddy](../caddy/readme.md),
which is the reverse proxy end of the same idea.
## Layout
```
postgres/
├── playbook.yml
├── vars/default.yml
└── roles/
├── postgres/ # the base postgres.container unit, env file, handlers
└── attach/ # includable task file: a service attaches itself to Postgres
```
Everything is installed under `{{ quadlet_dir }}`
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
(`~/podman-data/postgres/`).
## How services attach themselves
Postgres joins the app network of every service that needs a database. A service
joins by writing one Quadlet *drop-in* next to `postgres.container`:
```
~/.config/containers/systemd/
├── postgres.container # base unit, no app networks
├── postgres.container.d/ # ← drop-ins, one per client service
│ ├── 10-network-aliases.conf # [Container] NetworkAlias=db (shared)
│ ├── 50-nextcloud.conf # [Container] Network=nextcloud.network
│ ├── 50-gitea.conf
│ └── 50-tandoor.conf
├── nextcloud.network
├── gitea.network
└── tandoor.network
```
A service playbook does both halves — its own network, then the attach:
```yaml
- name: Register this service with Caddy
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
vars:
network_name: "{{ container_name }}"
- name: Attach this service to Postgres
ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
vars:
network_name: "{{ container_name }}"
```
The play must also run the `podman` role (for `podman_user` and `quadlet_dir`)
and define a `Restart Postgres` handler — see
[roles/postgres/handlers/main.yaml](roles/postgres/handlers/main.yaml) for the
exact snippet.
Order matters in one direction only: the `attach` include needs
`<name>.network` to already exist, which is why Caddy's `register` comes first.
It fails with a readable message if it does not, because a drop-in naming a
missing `.network` file makes Quadlet reject the *whole* postgres unit with an
error that never mentions the drop-in.
### Names: `postgres` and `db`
Both resolve on every attached network. `postgres` comes from
`ContainerName=postgres`; `db` is added by `10-network-aliases.conf`. The alias
exists because the Docker-era playbooks are not consistent — nextcloud, gitea,
tandoor and outline connect to `db`, the rest to `postgres`.
`NetworkAlias=` applies to *every* network the container joins, which is exactly
what a hub wants. It lives in one shared drop-in with a fixed value rather than
in each service's `50-*.conf`, for two reasons:
- it is not per-service — if one service could set a different value, the shared
file would be rewritten on every run, and every service writes the same bytes;
- with the alias in each service's own file, two attached services would make
Quadlet pass `--network-alias db` twice.
It is *not* in `postgres.container` itself for a duller reason: on a fresh host,
before anything has attached, the unit has no `Network=` at all, and asking
Podman for an alias on the default network is not a configuration worth
depending on.
### One way to attach, and one rule
[roles/attach](roles/attach/tasks/main.yaml) is the shared include, and is what
every service uses. A service may instead write the same two files from its own
role, but then it **must write identical bytes** on purpose: the drop-ins live in
one directory, so anything less means whichever play runs last rewrites the
other's file on every run. If you add a second form, make it match.
Whichever form you use, **flush handlers before the app starts on a first
attach.** The restart that joins Postgres to the new network is a handler, so by
default it runs at the end of the play — after the app has already come up and
failed to resolve `db`. `tandoor/roles/postgres` is the working example.
### Why drop-ins rather than listing the networks
Listing app networks in `postgres.container` would mean editing this playbook
every time a service is added, and again on removal. Drop-ins survive
re-templating of the base unit, are the mechanism Quadlet documents for this,
and make removal trivial: delete `50-tandoor.conf`, `daemon-reload`, done.
### The trade-off
A new drop-in changes the generated unit, so Postgres is **recreated** to join
the network — a few seconds of downtime for *every* connected app, not just the
new one. Quadlet has no live equivalent of `docker network connect`. If that
matters, `podman network connect <net> postgres` takes effect immediately, and
the drop-in makes it survive the next recreate; the `attach` role does not do
this by default because it would make the play non-idempotent.
### Why not one shared database network
Every app on a shared `postgres_net` could reach every other app on it. The
generated `<name>.network` units set `Options=isolate=true`, so keeping Postgres
as the only common member of otherwise separate networks is what preserves the
per-app isolation Caddy's networks give you.
## Who runs as whom
`UserNS=keep-id:uid=70,gid=70`. `70` is the uid the *server* writes as: the
official Alpine entrypoint starts as root, chowns PGDATA to the `postgres`
account, and drops privileges, so `docker exec postgres id -u` reporting `0` is
misleading. `keep-id` maps that one uid onto `podman_user`, which means:
- the whole data root — `~/podman-data/postgres/18/docker` and the version
directory above it — is owned by `podman_user`, never by a subuid;
- plain `tar`, `pg_dump`, `cp` and `rm -rf` work without `podman unshare` and
without `sudo`;
- the reverse — chowning the data to `podman_user` before first start — is also
correct, which is what makes bringing data across simple.
PGDATA and its version directory are created by the role rather than left to the
container's own `mkdir -p`. The entrypoint runs as container-root — a subuid
under `keep-id` — and only chowns PGDATA and its contents, so the version
directory would be stranded as subuid-owned: readable but not removable by
`podman_user`. Creating the chain up front is what makes the bullets above true.
Like Caddy, `podman_user` is derived from the connection
(`ansible_facts['user_id']`) and the play carries no `become`. This playbook does
not run the podman role, so it has no guard against connecting as `root`; the
prerequisites are applied separately (see Running).
## Settings
- `create_containers`: number of containers to create.
- `default_container_name`: default name for new containers.
- `default_container_image`: default image for new containers.
- `default_container_command`: default command to run on new containers.
| Variable | Purpose |
|---|---|
| `podman_user`, `podman_data_dir`, `quadlet_dir` | Rootless-Podman contract, duplicated from [caddy/vars/default.yml](../caddy/vars/default.yml) |
| `postgres_container` | Container / unit name, and the prefix of the drop-in directory |
| `postgres_major` | The one knob: pins both the image tag and the PGDATA path |
| `postgres_image` | `postgres:<major>-alpine`, fully qualified on purpose (see below) |
| `postgres_uid` / `postgres_gid` | The uid the server writes as — `70` in the Alpine image |
| `host_directory` | Where state lives (`~/podman-data/postgres`) |
| `postgres_mount` / `pgdata` | The image's VOLUME, and PGDATA inside it (`/var/lib/postgresql`, `/var/lib/postgresql/18/docker`) |
| `postgres_env_file` | `0600` file holding the credentials, read via `EnvironmentFile=` |
| `postgres_dropin_dir` | Where services attach their networks |
| `pg_user` / `pg_password` / `pg_database` | Credentials and the default database |
| `postgres_timeout_start` / `postgres_timeout_stop` | initdb on first start, clean shutdown on stop |
### Image and PGDATA layout
## Running this Playbook
`docker.io/library/postgres:18-alpine` — fully qualified because rootless Podman
resolves short names against `unqualified-search-registries`, and a Quadlet unit
has no TTY to answer a short-name prompt. `-alpine` still floats within 18.x for
security patches.
Quick Steps:
**18 moved both the image's VOLUME and its default PGDATA**, which is why
`postgres_major` exists and why the mount is a *directory* rather than the data
directory itself:
### 1. Obtain the playbook
```shell
git clone https://github.com/do-community/ansible-playbooks.git
cd ansible-playbooks/docker_ubuntu1804
```
| | 17 | 18+ |
|---|---|---|
| `VOLUME` | `/var/lib/postgresql/data` | `/var/lib/postgresql` |
| `PGDATA` | `/var/lib/postgresql/data/pgdata` | `/var/lib/postgresql/18/docker` |
### 2. Customize Options
The 18 entrypoint *refuses to start* if it finds a cluster at the old
`/var/lib/postgresql/data` path — that check exists so that bumping the image tag
without upgrading the database fails loudly instead of appearing to be data loss.
Keeping the major version in the path is also what lets a later
`pg_upgrade --link` run without mount-boundary problems.
```shell
nano vars/default.yml
```
`monoco` runs 17.5, so the data comes across as a **logical dump**, not a copy of
`PGDATA`: 17 data files are unreadable by an 18 server. The one-time move is
[step-by-step here](migrate-from-monoco.md).
```yml
#vars/default.yml
---
create_containers: 4
default_container_name: docker
default_container_image: ubuntu
default_container_command: sleep 1d
```
### No published port
### 3. Run the Playbook
There is deliberately no `PublishPort`. The Docker-era playbook published
`5432:5432` on `0.0.0.0` with UFW inactive, so the database was reachable from
the internet; it needs no host port at all now that clients reach it over the
app networks. Verify with `ss -tlnp | grep 5432` — it should print nothing.
### Credentials
`pg_password` is still the shared `123456` from the app playbooks, so a migrated
app connects unchanged. It is written to `~/podman-data/postgres/postgres.env`
mode `0600` and read with `EnvironmentFile=` rather than as `Environment=` lines,
because the unit file itself is world-readable in `~/.config`. Note that
`podman inspect postgres` still prints the environment to anyone who can run
podman as that user; a real secret store is the next step, and is a
one-file change here plus the `postgres_password` vars in the app playbooks.
## Running
The play targets `emily` directly (`hosts: emily`):
```command
ansible-playbook -l [target] -i [inventory file] -u [remote user] playbook.yml
ansible-playbook -u tanshu postgres/playbook.yml
```
For more information on how to run this Ansible setup, please check this guide: [How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
The inventory that defines `emily` lives in `prepare-server/`, whose `ansible.cfg`
names it — run from there, or pass an inventory explicitly.
This play applies **only** the Postgres unit and its data directory. The
rootless-Podman prerequisites — installing podman, `loginctl enable-linger`, the
low-port sysctl and the Quadlet directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are not run here. A host that has already run `caddy/playbook.yml` has them. To
apply them on their own:
```yaml
- name: Rootless Podman prerequisites
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/podman/tasks/main.yaml"
```
The three variables those tasks need are duplicated at the top of
[vars/default.yml](vars/default.yml) rather than imported, because a play cannot
include Caddy's vars file wholesale — it also sets `host_directory`.
## Operational notes
```
systemctl --user status postgres
systemctl --user restart postgres
journalctl --user -u postgres -f
podman exec -it postgres psql -U postgres
podman exec postgres pg_isready -U postgres
podman inspect postgres --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
```
The play waits on `pg_isready` inside the container rather than trusting
systemd. `Restart=always` means the unit reports `active` even while the
container crash-loops, so a wrong `keep-id` (an entrypoint permission error,
over and over) would otherwise look like success.
`systemctl --user enable postgres` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.
### Debugging a unit that will not generate
```
/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
systemd-analyze --user --generators=true verify postgres.service
```
The most common cause is a `Network=` naming a `.network` file that does not
exist — which is why `attach` checks first.
## Backups
`--link postgres:db` does not exist in Podman, so every `*-backup.sh` that uses
it (`db-`, `bitwarden-`, `hiwikijs-`, `kakawa-`, `nextcloud-`, `tandoor-`) is
broken on this host. Dump in place instead — no helper container, no networking,
no `--user` juggling:
```bash
podman exec -e PGPASSWORD="..." postgres \
pg_dump -U postgres nextcloud | bzip2 > "$WORKDIR/$DB_DUMP_NAME"
```
Whole-cluster form, which is what a migration uses:
```bash
podman exec -e PGPASSWORD="..." postgres pg_dumpall -U postgres > cluster.sql
```
Because the data is owned by `podman_user`, the `sudo tar` / `chown $(logname)`
gymnastics in the old scripts can go away too. The database is also the reason
root's crontab cannot drive it any more: root cannot `podman exec` into another
user's rootless containers, so anything scheduled belongs in a **systemd user
timer** next to the Quadlet units.
## Bringing data from the Docker host
The one-time move of monoco's cluster — 12 databases, ~900 MB, and a 17 → 18
major-version jump — is written out command by command in
**[migrate-from-monoco.md](migrate-from-monoco.md)**: dump, transfer, restore,
verify, rollback.
The short version of why it is a dump rather than a file copy:
- **17 data files are unreadable by an 18 server**, so `PGDATA` cannot simply be
copied across the major version;
- **the 18 image moved PGDATA** to `/var/lib/postgresql/18/docker` and refuses to
start when it finds a cluster at the old path;
- for a cluster well under a gigabyte, a logical dump is both the safer and the
faster thing to get right, and it re-creates the three roles
(`postgres`, `oc_tanshu`, `zitadel`) in the same pass.
`pg_dumpall` restores each database under its original name with its original
grants, so the app playbooks connect unchanged — same container name, same `db`
alias, same shared password.
## Verification
- [ ] `systemctl --user status postgres` is active, and stays active after a reboot
- [ ] `podman exec postgres pg_isready -U postgres` reports accepting connections
- [ ] `podman exec postgres psql -U postgres -c '\l'` lists the expected databases
- [ ] `ss -tlnp | grep 5432` on the host prints nothing
- [ ] `ls -ln ~/podman-data/postgres/18/docker` is owned by `tanshu`, not a `165536+` subuid
- [ ] `ls -ln ~/podman-data/postgres/18` is owned by `tanshu` too — if it is not, the
container created it as subuid and `rm -rf` will fight you
- [ ] From each client app's container: `getent hosts postgres` and `getent hosts db`
- [ ] Each app can read **and** write — that is the check that catches a wrong `keep-id`
- [ ] A `pg_dump` of every database completes and restores into a scratch database