playbooks/bitwarden/readme.md

# Vaultwarden / Bitwarden (rootless Podman + Quadlet)

Runs [Vaultwarden](https://github.com/dani-garcia/vaultwarden) — the
Bitwarden-compatible server this repo has always called `bitwarden` — on `emily`
as a **rootless Podman** container managed by a **systemd user unit** (Quadlet),
behind the shared [rootless Caddy](../caddy/readme.md), with the vault in the
shared [rootless Postgres](../postgres/readme.md). Replaces the
`docker_container` deployment on `monoco`.

The name is split on purpose: the directory, container, unit, Podman network and
Postgres database are all `bitwarden`, as they have been since the Docker-era
playbook, and only the image is `vaultwarden/server`. Renaming it would touch the
Caddyfile upstream, the database and every client's saved server URL, for no
gain.

**Postgres is authoritative for the vault.** `/data` holds everything else —
attachments, Sends, the `rsa_key*` signing pair, the admin config, the icon
cache — and the `db.sqlite3` that the image would use by default is **stale on
the old host and deliberately not migrated**
([docs/backup-plan.md §3.3](../docs/backup-plan.md)). The playbook warns if it
finds that file in the data directory.

## Ordering

Both shared services are already migrated and running on `emily`:

1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns
   the Caddyfile this site appends to, supplies the `caddy.service` handlers, and
   is the one play that applies the rootless-Podman prerequisites.
2. [postgres/playbook.yml](../postgres/playbook.yml) — deployed, and its cluster
   brought across from monoco. **The `bitwarden` database is already restored**
   (10 MB, one of the 12 databases in
   [postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md)).
3. This playbook — **run it after the file copy below**, not before:
   the app is the thing that writes to `~/podman-data/bitwarden`, and its first
   start generates `rsa_key*` if the copy has not landed yet.

Vaultwarden's environment file points `DATABASE_URL` at `postgres`. Until this
playbook has attached `bitwarden_net` to Postgres, that name does not resolve on
this network, so running this playbook is what makes the app reachable — there is
no separate step.

## Layout

```
bitwarden/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/     # attaches this app's network to Postgres (+ its handler)
    ├── bitwarden/    # data dir, the 0600 env file, the Quadlet unit, start + verify
    └── caddy/        # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/bitwarden.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/bitwarden.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-bitwarden.conf` | attaches Caddy to that network |
| `~/.config/containers/systemd/postgres.container.d/50-bitwarden.conf` | attaches Postgres to it |
| `~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf` | the shared `db` alias |
| `~/podman-data/bitwarden/data/` | the image's `/data`: `attachments/`, `sends/`, `rsa_key*`, `config.json`, `icon_cache/` |
| `~/podman-data/bitwarden/bitwarden.env` | `DATABASE_URL` + `DOMAIN`, mode `0600`, read by systemd |

## What is state, and what is not

Per the upstream wiki's
[Backing up your data](https://github.com/dani-garcia/vaultwarden/wiki/Backing-up-your-vault),
and with Postgres as the database, `data/` holds:

| Path | Keep? | Why |
|---|---|---|
| `attachments/` | **required** | the only important data not in database tables |
| `sends/` | copy it | Send attachments; ephemeral by design, but copying keeps existing Sends working |
| `rsa_key.pem`, `rsa_key.der`, `rsa_key.pub.*` | copy it | signs the JWTs of logged-in users. Regenerating the pair logs everyone out and invalidates open invitation tokens |
| `config.json` | copy it | admin-page configuration; only exists if the admin page was enabled |
| `icon_cache/` | optional | favicon cache, refetched on demand |
| `db.sqlite3`, `-wal`, `-shm` | **no** | stale. Postgres has been authoritative for years; restoring this would silently lose months of vault data |

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Bitwarden's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`bitwarden.container` references `bitwarden.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet. The Postgres attach below depends
on it for the same reason.

`network_name` is `bitwarden`, so the *file* is `bitwarden.network` and the
*Podman network* is `bitwarden_net` (`NetworkName=` in the network unit pins it,
keeping the Docker-era name).

The Docker-era `roles/network` is **deleted rather than ported**. It existed to
attach an already-created container to a network — `docker_network`'s
`connected:`/`appends:` — which `podman_network` has no equivalent for
([docs/podman-compat-audit.md §3.3](../docs/podman-compat-audit.md)). Nothing has
to be connected after the fact now: the app's own unit names its `Network=`, and
Caddy and Postgres learn about the network from the drop-ins written in the same
run.

The site block itself is appended to the shared Caddyfile by
[roles/caddy](roles/caddy/tasks/main.yaml) with `blockinfile`, exactly as the
other services do — only `caddyfile_path` changed, from
`/var/lib/caddy/conf/Caddyfile` to `~/podman-data/caddy/conf/Caddyfile`. Caddy
handles the WebSocket upgrade the Docker-era nginx front end had to configure by
hand, and it imposes no request-body limit, so the `client_max_body_size 525M` in
the deleted `files/nginx.conf.j2` has no equivalent and needs none.

## How it attaches to Postgres

Same idea at the other end: Postgres knows nothing about its clients, and this
playbook uses the shared include in
[postgres/roles/attach](../postgres/roles/attach/tasks/main.yaml) rather than
writing those drop-ins itself. That writes two files, both mergeable into
`postgres.container` by Quadlet:

| File | Content |
|---|---|
| `postgres.container.d/50-bitwarden.conf` | `Network=bitwarden.network` |
| `postgres.container.d/10-network-aliases.conf` | `NetworkAlias=db` |

`DATABASE_URL` uses the host `postgres`, which is what the Docker-era `db_url`
used and what `ContainerName=postgres` provides; the shared `db` alias resolves
too, and the play checks both.

[roles/postgres](roles/postgres/tasks/main.yaml) wraps that include for two
reasons:

- it defines the `Restart Postgres` handler the include notifies; and
- it **flushes handlers** immediately afterwards. On a first attach the drop-in
  is new, so Postgres has to be *recreated* to join `bitwarden_net` — and that
  restart is a handler, which by default runs at the end of the play, after the
  `bitwarden` role has already started the app. The app would then come up
  before `postgres` resolves, fail its own health check, and fail the play on a
  machine where nothing is wrong. Flushing applies the drop-in first; on later
  runs nothing is pending and it is a no-op.

Note the cost stated in [postgres/readme.md](../postgres/readme.md): a new
drop-in recreates Postgres, which is a few seconds of downtime for **every**
attached app, not just this one.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `vault.tanshu.com` — the site block Caddy serves |
| `container_name` | `bitwarden`, and the Quadlet unit / container name |
| `image` | `docker.io/vaultwarden/server:latest` — fully qualified on purpose |
| `network_name` | `bitwarden` → `bitwarden.network` / `bitwarden_net` |
| `data_dir` | `~/podman-data/bitwarden/data` → the image's `/data` |
| `container_port` | `80`, the port inside the container that Caddy proxies to (`ROCKET_PORT`) |
| `env_file` | `~/podman-data/bitwarden/bitwarden.env`, mode `0600` |
| `postgres_container` / `postgres_unit` | The shared database unit this app attaches to |
| `pg_user` / `pg_password` / `pg_database` | Credentials, and the database name — `bitwarden` |
| `database_url` / `domain` | What the environment file contains |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run — Vaultwarden publishes nothing host-side.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

`UserNS=keep-id:uid=0,gid=0` — Vaultwarden's image runs as container-root and
writes `/data` as root, with no entrypoint step that chowns anything
([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). Mapping
container root onto `podman_user` keeps `~/podman-data/bitwarden` owned by
`tanshu` and leaves no subuid-owned files to trip over backups.

## Running

```command
ansible-playbook -u tanshu bitwarden/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

This play applies **only** this app's units and data. Its `pre_tasks` run Caddy's
`register` include and nothing else. The rootless-Podman prerequisites —
installing podman, `loginctl enable-linger`, the low-port sysctl and the Quadlet
directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.

`systemctl --user enable bitwarden` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

## Migrating the file data from monoco

The database already moved — `bitwarden` is one of the 12 databases restored in
[postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md). **This is
the app's own files only**, and the playbook does not move them because they are
on the old host while the playbook runs on the new one. Run this from the machine
that holds the `monoco` and `emily` SSH aliases.

The whole tree is about **4 MB** ([docs/backup-plan.md §2.1](../docs/backup-plan.md)),
so this is quick — the state that matters is the 10 MB Postgres database, and that
is done. Copy it anyway: `attachments/` is the part of the vault that is not in
any database.

```bash
# 1. Look before you copy: size, contents, and whether it is readable without
#    sudo. `ls -ln` shows numeric owners, which is what matters here (the old
#    container ran as root, so these files are uid 0).
ssh monoco 'du -sh /var/lib/bitwarden/data; ls -ln /var/lib/bitwarden/data'
ssh monoco 'find /var/lib/bitwarden/data -type f ! -readable'   # empty = no sudo needed

# 2. Stop the old container so nothing writes while you copy. Leave it stopped:
#    its database is on emily now, and two live copies would diverge.
ssh monoco 'docker stop bitwarden'

# 3. Create the destination (this is the mount target of the image's /data).
ssh emily 'mkdir -p ~/podman-data/bitwarden/data'

# 4. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command.
#
#    `./db.sqlite3*` is excluded — main file, -wal and -shm. It is stale on
#    monoco (see the table above), and with Postgres configured it would be an
#    inert file that looks exactly like a restore source.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
set -o pipefail
ssh monoco 'tar --exclude="./db.sqlite3*" -C /var/lib/bitwarden/data -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/bitwarden/data -xf -'

#    If step 1 listed unreadable files, read them as root instead:
# ssh monoco 'sudo tar --exclude="./db.sqlite3*" -C /var/lib/bitwarden/data -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/bitwarden/data -xf -'

# 5. Verify by content, over the whole tree, with the same exclusion on the
#    source side. The paths are relative on both sides, so the digests are
#    comparable.
ssh monoco 'cd /var/lib/bitwarden/data && find . -type f ! -name "db.sqlite3*" -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
ssh emily  'cd ~/podman-data/bitwarden/data && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/bitwarden ! -user tanshu'    # empty = correct

# 7. Deploy — after the copy, so Vaultwarden finds the real data on first start
#    rather than generating a fresh rsa_key pair.
cd prepare-server && ansible-playbook -u tanshu ../bitwarden/playbook.yml
```

Ownership needs no `chown` pass: the container runs as uid 0 in-container, which
`keep-id` maps to `tanshu`, so files extracted as `tanshu` are already right.
That is also why `--no-same-owner` is explicit on the receiving side — the source
is root-owned, and restoring that ownership would be wrong here.

If the link is slow or drops, do it in two hops instead: resumable, and it shows
progress.

```bash
rsync -aP --exclude='db.sqlite3*' monoco:/var/lib/bitwarden/data/ ~/bitwarden-copy/
rsync -aP ~/bitwarden-copy/ emily:~/podman-data/bitwarden/data/
```

Rolling back means rolling back both halves: `ssh monoco 'docker start bitwarden'`
is only meaningful while monoco's own Postgres 17 cluster is still intact, and it
would then be writing to a database that emily's copy has already moved past.

## Verification

```bash
systemctl --user status bitwarden
journalctl --user -u bitwarden -f
podman logs bitwarden

# What the playbook checks for you:
podman exec bitwarden /healthcheck.sh      # the image's own probe
podman exec bitwarden getent hosts postgres
podman exec bitwarden getent hosts db

# The database really is the migrated one. Run the same query on both hosts and
# compare — equal counts mean the vault came across.
ssh emily  'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d bitwarden -tAc "select count(*) from users"'
ssh monoco 'docker exec -e PGPASSWORD=123456 postgres psql -U postgres -d bitwarden -tAc "select count(*) from users"'

# The data directory is the copied one, and has no SQLite file in it.
ssh emily 'ls -ln ~/podman-data/bitwarden/data'
ssh emily 'test ! -e ~/podman-data/bitwarden/data/db.sqlite3 && echo "no stale SQLite"'

# The site, end to end.
curl -sSI https://vault.tanshu.com
```

Then log in **in a browser** and confirm an existing item decrypts — that is the
check that the Postgres database and the client-facing crypto are both intact,
and a `curl` cannot make it. The play waits on the image's own health check
(`/healthcheck.sh` → `/alive`), which is stronger than `is-active` but still only
says the process answers.

Also exercise a **write** through the app — upload a file attachment — and confirm
on the host that the new file under
`~/podman-data/bitwarden/data/attachments` is owned by `tanshu`. That is the check
that catches a wrong `keep-id`; a read-only smoke test will not.

Then reboot the host and confirm the unit returns unattended — without
`loginctl enable-linger tanshu` it will not, and the failure is silent.

## Notes and gotchas

- **The readiness probe is exec'd, not `podman healthcheck run`.** Podman's
  healthcheck plumbing is unreliable for Quadlet-managed rootless containers: the
  check is registered against the *user* systemd manager, and when that fails the
  check never runs and `podman healthcheck run` reports that the container has no
  healthcheck at all, even though the image declares one
  ([podman#26066](https://github.com/containers/podman/issues/26066),
  [podman#22804](https://github.com/containers/podman/issues/22804)). That is what
  the first deploy here hit — the app was serving and the probe wrapper was the
  only thing failing. The play therefore executes the image's own
  `/healthcheck.sh` directly, which is the same probe with no plumbing in
  between. One cosmetic consequence: `podman inspect bitwarden --format
  '{{.State.Health.Status}}'` can stay `starting` forever, which is not a sign
  that the app is unhealthy.
- **The stale SQLite file.** If `~/podman-data/bitwarden/data/db.sqlite3` exists,
  the play prints a warning on every run rather than failing, because the file is
  inert while `DATABASE_URL` is set. It is not a backup. Delete it once the
  Postgres database has been verified.
- **Signups are open.** The Docker-era container set only `DATABASE_URL` and
  `DOMAIN`, so `SIGNUPS_ALLOWED` is still the image default (`true`), the admin
  panel is off (no `ADMIN_TOKEN`), and SMTP is unconfigured. This playbook keeps
  that behaviour deliberately — tightening it is a decision, not a migration
  step.
- **Client IPs.** Vaultwarden uses the client address for its login
  brute-force protection, and whether the original address survives the rootless
  port forward depends on the forwarder. See
  [caddy/readme.md](../caddy/readme.md), "Client IPs", before relying on it.
- **`latest` floats.** The old playbook used the unqualified
  `vaultwarden/server:latest`; this one fully qualifies the same tag, because
  rootless Podman has no TTY to answer a short-name prompt when the unit starts.
  Neither pins a digest or a version, so an image bump is a normal `podman pull`
  away and may run new Diesel migrations on first start.
- **Registering restarts all of Caddy**, and attaching restarts Postgres for
  everyone — a brief restart of every hosted site, and a few seconds of downtime
  for every attached app. See the trade-offs in
  [caddy/readme.md](../caddy/readme.md) and
  [postgres/readme.md](../postgres/readme.md).
- **If the Caddyfile does not exist yet**, this playbook creates it containing
  only the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a
  missing or zero-length file. Deploy Caddy first on a fresh host.
- **DNS.** `vault.tanshu.com` must resolve to `emily` before Caddy can obtain a
  certificate for it.
- **The backup scripts still assume Docker.** `~/bin/bitwarden-backup.sh` uses
  `--link`, which Podman does not have; it needs the `podman exec` form from
  [postgres/readme.md](../postgres/readme.md), "Backups". `bitwarden-upload.sh`
  is worse than broken — it uploads a `production.sqlite3` that does not exist,
  and the file it would upload is not the authoritative store anyway. Also,
  root's crontab cannot `podman exec` into another user's rootless containers, so
  anything scheduled belongs in a **systemd user timer** next to the Quadlet
  units.
- **Do not restart the container on monoco.** It would write to monoco's old
  cluster, which is being kept intact purely as a rollback.

## Not done here

- Rotating the secrets that are committed in plaintext — `pg_password` is the
  fleet's shared `123456`, and it lives in `bitwarden.env` at mode `0600` rather
  than in the world-readable unit file.
- Closing signups, enabling the admin page, or configuring SMTP (see above).
- Replacing the two broken backup scripts with a systemd user timer.