playbooks/bitwarden/readme.md

Vaultwarden / Bitwarden (rootless Podman + Quadlet)

Runs Vaultwarden — the Bitwarden-compatible server this repo has always called bitwarden — on emily as a rootless Podman container managed by a systemd user unit (Quadlet), behind the shared rootless Caddy, with the vault in the shared rootless Postgres. Replaces the docker_container deployment on monoco.

The name is split on purpose: the directory, container, unit, Podman network and Postgres database are all bitwarden, as they have been since the Docker-era playbook, and only the image is vaultwarden/server. Renaming it would touch the Caddyfile upstream, the database and every client's saved server URL, for no gain.

Postgres is authoritative for the vault. /data holds everything else — attachments, Sends, the rsa_key* signing pair, the admin config, the icon cache — and the db.sqlite3 that the image would use by default is stale on the old host and deliberately not migrated (docs/backup-plan.md §3.3). The playbook warns if it finds that file in the data directory.

Ordering

Both shared services are already migrated and running on emily:

1. caddy/playbook.yml — the reverse proxy, which owns the Caddyfile this site appends to, supplies the caddy.service handlers, and is the one play that applies the rootless-Podman prerequisites.
2. postgres/playbook.yml — deployed, and its cluster brought across from monoco. The bitwarden database is already restored (10 MB, one of the 12 databases in postgres/migrate-from-monoco.md).
3. This playbook — run it after the file copy below, not before: the app is the thing that writes to ~/podman-data/bitwarden, and its first start generates rsa_key* if the copy has not landed yet.

Vaultwarden's environment file points DATABASE_URL at postgres. Until this playbook has attached bitwarden_net to Postgres, that name does not resolve on this network, so running this playbook is what makes the app reachable — there is no separate step.

Layout

bitwarden/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/     # attaches this app's network to Postgres (+ its handler)
    ├── bitwarden/    # data dir, the 0600 env file, the Quadlet unit, start + verify
    └── caddy/        # appends this site's block to the shared Caddyfile

Quadlet units and state:

On the host	What
~/.config/containers/systemd/bitwarden.container	the generated unit (Ansible-templated)
~/.config/containers/systemd/bitwarden.network	the app network, written by caddy/roles/register
~/.config/containers/systemd/caddy.container.d/50-bitwarden.conf	attaches Caddy to that network
~/.config/containers/systemd/postgres.container.d/50-bitwarden.conf	attaches Postgres to it
~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf	the shared db alias
~/podman-data/bitwarden/data/	the image's /data: attachments/, sends/, rsa_key*, config.json, icon_cache/
~/podman-data/bitwarden/bitwarden.env	DATABASE_URL + DOMAIN, mode 0600, read by systemd

What is state, and what is not

Per the upstream wiki's Backing up your data, and with Postgres as the database, data/ holds:

Path	Keep?	Why
attachments/	required	the only important data not in database tables
sends/	copy it	Send attachments; ephemeral by design, but copying keeps existing Sends working
rsa_key.pem, rsa_key.der, rsa_key.pub.*	copy it	signs the JWTs of logged-in users. Regenerating the pair logs everyone out and invalidates open invitation tokens
config.json	copy it	admin-page configuration; only exists if the admin page was enabled
icon_cache/	optional	favicon cache, refetched on demand
db.sqlite3, -wal, -shm	no	stale. Postgres has been authoritative for years; restoring this would silently lose months of vault data

How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls the shared include in caddy/roles/register, which writes the network unit and the drop-in that pulls it into Caddy:

- name: Register Bitwarden's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"

It runs in pre_tasks, before the app unit is started, because bitwarden.container references bitwarden.network: Quadlet fails unit generation — so systemctl --user start reports "unit not found" — when a referenced .network file does not exist yet. The Postgres attach below depends on it for the same reason.

network_name is bitwarden, so the file is bitwarden.network and the Podman network is bitwarden_net (NetworkName= in the network unit pins it, keeping the Docker-era name).

The Docker-era roles/network is deleted rather than ported. It existed to attach an already-created container to a network — docker_network's connected:/appends: — which podman_network has no equivalent for (docs/podman-compat-audit.md §3.3). Nothing has to be connected after the fact now: the app's own unit names its Network=, and Caddy and Postgres learn about the network from the drop-ins written in the same run.

The site block itself is appended to the shared Caddyfile by roles/caddy with blockinfile, exactly as the other services do — only caddyfile_path changed, from /var/lib/caddy/conf/Caddyfile to ~/podman-data/caddy/conf/Caddyfile. Caddy handles the WebSocket upgrade the Docker-era nginx front end had to configure by hand, and it imposes no request-body limit, so the client_max_body_size 525M in the deleted files/nginx.conf.j2 has no equivalent and needs none.

How it attaches to Postgres

Same idea at the other end: Postgres knows nothing about its clients, and this playbook uses the shared include in postgres/roles/attach rather than writing those drop-ins itself. That writes two files, both mergeable into postgres.container by Quadlet:

File	Content
postgres.container.d/50-bitwarden.conf	Network=bitwarden.network
postgres.container.d/10-network-aliases.conf	NetworkAlias=db

DATABASE_URL uses the host postgres, which is what the Docker-era db_url used and what ContainerName=postgres provides; the shared db alias resolves too, and the play checks both.

roles/postgres wraps that include for two reasons:

• it defines the Restart Postgres handler the include notifies; and
• it flushes handlers immediately afterwards. On a first attach the drop-in is new, so Postgres has to be recreated to join bitwarden_net — and that restart is a handler, which by default runs at the end of the play, after the bitwarden role has already started the app. The app would then come up before postgres resolves, fail its own health check, and fail the play on a machine where nothing is wrong. Flushing applies the drop-in first; on later runs nothing is pending and it is a no-op.

Note the cost stated in postgres/readme.md: a new drop-in recreates Postgres, which is a few seconds of downtime for every attached app, not just this one.

Settings

Variable	Value / purpose
podman_user	Derived from the connection (ansible_facts['user_id']), not set by hand
podman_data_dir	~/podman-data
quadlet_dir	~/.config/containers/systemd
host	vault.tanshu.com — the site block Caddy serves
container_name	bitwarden, and the Quadlet unit / container name
image	docker.io/vaultwarden/server:latest — fully qualified on purpose
network_name	bitwarden → bitwarden.network / bitwarden_net
data_dir	~/podman-data/bitwarden/data → the image's /data
container_port	80, the port inside the container that Caddy proxies to (ROCKET_PORT)
env_file	~/podman-data/bitwarden/bitwarden.env, mode 0600
postgres_container / postgres_unit	The shared database unit this app attaches to
pg_user / pg_password / pg_database	Credentials, and the database name — bitwarden
database_url / domain	What the environment file contains
caddy_name / caddyfile_path	The shared Caddy unit and its Caddyfile

The three podman_* / quadlet_dir vars are duplicated from caddy/vars/default.yml, as in every service playbook. That is a deliberate trade: a play cannot include the Caddy vars file wholesale, because it also sets host_directory (to the Caddy data directory) and every service defines that for itself. Keep them in sync. Caddy's fourth, podman_unprivileged_port_start, belongs to the podman tasks this play does not run — Vaultwarden publishes nothing host-side.

Like Caddy, podman_user is derived from the connection and the play carries no become. Because the podman tasks are not run here, there is no guard against connecting as root; the prerequisites, including that check, live in caddy/playbook.yml.

UserNS=keep-id:uid=0,gid=0 — Vaultwarden's image runs as container-root and writes /data as root, with no entrypoint step that chowns anything (docs/podman-compat-audit.md §3.2). Mapping container root onto podman_user keeps ~/podman-data/bitwarden owned by tanshu and leaves no subuid-owned files to trip over backups.

Running

ansible-playbook -u tanshu bitwarden/playbook.yml

The play targets emily directly, so the inventory must contain it — the one in prepare-server/ does, and its ansible.cfg names it.

This play applies only this app's units and data. Its pre_tasks run Caddy's register include and nothing else. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them.

systemctl --user enable bitwarden is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Migrating the file data from monoco

The database already moved — bitwarden is one of the 12 databases restored in postgres/migrate-from-monoco.md. This is the app's own files only, and the playbook does not move them because they are on the old host while the playbook runs on the new one. Run this from the machine that holds the monoco and emily SSH aliases.

The whole tree is about 4 MB (docs/backup-plan.md §2.1), so this is quick — the state that matters is the 10 MB Postgres database, and that is done. Copy it anyway: attachments/ is the part of the vault that is not in any database.

# 1. Look before you copy: size, contents, and whether it is readable without
#    sudo. `ls -ln` shows numeric owners, which is what matters here (the old
#    container ran as root, so these files are uid 0).
ssh monoco 'du -sh /var/lib/bitwarden/data; ls -ln /var/lib/bitwarden/data'
ssh monoco 'find /var/lib/bitwarden/data -type f ! -readable'   # empty = no sudo needed

# 2. Stop the old container so nothing writes while you copy. Leave it stopped:
#    its database is on emily now, and two live copies would diverge.
ssh monoco 'docker stop bitwarden'

# 3. Create the destination (this is the mount target of the image's /data).
ssh emily 'mkdir -p ~/podman-data/bitwarden/data'

# 4. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command.
#
#    `./db.sqlite3*` is excluded — main file, -wal and -shm. It is stale on
#    monoco (see the table above), and with Postgres configured it would be an
#    inert file that looks exactly like a restore source.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
set -o pipefail
ssh monoco 'tar --exclude="./db.sqlite3*" -C /var/lib/bitwarden/data -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/bitwarden/data -xf -'

#    If step 1 listed unreadable files, read them as root instead:
# ssh monoco 'sudo tar --exclude="./db.sqlite3*" -C /var/lib/bitwarden/data -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/bitwarden/data -xf -'

# 5. Verify by content, over the whole tree, with the same exclusion on the
#    source side. The paths are relative on both sides, so the digests are
#    comparable.
ssh monoco 'cd /var/lib/bitwarden/data && find . -type f ! -name "db.sqlite3*" -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
ssh emily  'cd ~/podman-data/bitwarden/data && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 6. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/bitwarden ! -user tanshu'    # empty = correct

# 7. Deploy — after the copy, so Vaultwarden finds the real data on first start
#    rather than generating a fresh rsa_key pair.
cd prepare-server && ansible-playbook -u tanshu ../bitwarden/playbook.yml

Ownership needs no chown pass: the container runs as uid 0 in-container, which keep-id maps to tanshu, so files extracted as tanshu are already right. That is also why --no-same-owner is explicit on the receiving side — the source is root-owned, and restoring that ownership would be wrong here.

If the link is slow or drops, do it in two hops instead: resumable, and it shows progress.

rsync -aP --exclude='db.sqlite3*' monoco:/var/lib/bitwarden/data/ ~/bitwarden-copy/
rsync -aP ~/bitwarden-copy/ emily:~/podman-data/bitwarden/data/

Rolling back means rolling back both halves: ssh monoco 'docker start bitwarden' is only meaningful while monoco's own Postgres 17 cluster is still intact, and it would then be writing to a database that emily's copy has already moved past.

Verification

systemctl --user status bitwarden
journalctl --user -u bitwarden -f
podman logs bitwarden

# What the playbook checks for you:
podman exec bitwarden /healthcheck.sh      # the image's own probe
podman exec bitwarden getent hosts postgres
podman exec bitwarden getent hosts db

# The database really is the migrated one. Run the same query on both hosts and
# compare — equal counts mean the vault came across.
ssh emily  'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d bitwarden -tAc "select count(*) from users"'
ssh monoco 'docker exec -e PGPASSWORD=123456 postgres psql -U postgres -d bitwarden -tAc "select count(*) from users"'

# The data directory is the copied one, and has no SQLite file in it.
ssh emily 'ls -ln ~/podman-data/bitwarden/data'
ssh emily 'test ! -e ~/podman-data/bitwarden/data/db.sqlite3 && echo "no stale SQLite"'

# The site, end to end.
curl -sSI https://vault.tanshu.com

Then log in in a browser and confirm an existing item decrypts — that is the check that the Postgres database and the client-facing crypto are both intact, and a curl cannot make it. The play waits on the image's own health check (/healthcheck.sh → /alive), which is stronger than is-active but still only says the process answers.

Also exercise a write through the app — upload a file attachment — and confirm on the host that the new file under ~/podman-data/bitwarden/data/attachments is owned by tanshu. That is the check that catches a wrong keep-id; a read-only smoke test will not.

Then reboot the host and confirm the unit returns unattended — without loginctl enable-linger tanshu it will not, and the failure is silent.

Notes and gotchas

• The readiness probe is exec'd, not podman healthcheck run. Podman's healthcheck plumbing is unreliable for Quadlet-managed rootless containers: the check is registered against the user systemd manager, and when that fails the check never runs and podman healthcheck run reports that the container has no healthcheck at all, even though the image declares one (podman#26066, podman#22804). That is what the first deploy here hit — the app was serving and the probe wrapper was the only thing failing. The play therefore executes the image's own /healthcheck.sh directly, which is the same probe with no plumbing in between. One cosmetic consequence: podman inspect bitwarden --format '{{.State.Health.Status}}' can stay starting forever, which is not a sign that the app is unhealthy.
• The stale SQLite file. If ~/podman-data/bitwarden/data/db.sqlite3 exists, the play prints a warning on every run rather than failing, because the file is inert while DATABASE_URL is set. It is not a backup. Delete it once the Postgres database has been verified.
• Signups are open. The Docker-era container set only DATABASE_URL and DOMAIN, so SIGNUPS_ALLOWED is still the image default (true), the admin panel is off (no ADMIN_TOKEN), and SMTP is unconfigured. This playbook keeps that behaviour deliberately — tightening it is a decision, not a migration step.
• Client IPs. Vaultwarden uses the client address for its login brute-force protection, and whether the original address survives the rootless port forward depends on the forwarder. See caddy/readme.md, "Client IPs", before relying on it.
• latest floats. The old playbook used the unqualified vaultwarden/server:latest; this one fully qualifies the same tag, because rootless Podman has no TTY to answer a short-name prompt when the unit starts. Neither pins a digest or a version, so an image bump is a normal podman pull away and may run new Diesel migrations on first start.
• Registering restarts all of Caddy, and attaching restarts Postgres for everyone — a brief restart of every hosted site, and a few seconds of downtime for every attached app. See the trade-offs in caddy/readme.md and postgres/readme.md.
• If the Caddyfile does not exist yet, this playbook creates it containing only the site block. That is a valid Caddyfile; the Caddy playbook's seeded global-options block is then not added, because its seed only fires on a missing or zero-length file. Deploy Caddy first on a fresh host.
• DNS. vault.tanshu.com must resolve to emily before Caddy can obtain a certificate for it.
• The backup scripts still assume Docker. ~/bin/bitwarden-backup.sh uses --link, which Podman does not have; it needs the podman exec form from postgres/readme.md, "Backups". bitwarden-upload.sh is worse than broken — it uploads a production.sqlite3 that does not exist, and the file it would upload is not the authoritative store anyway. Also, root's crontab cannot podman exec into another user's rootless containers, so anything scheduled belongs in a systemd user timer next to the Quadlet units.
• Do not restart the container on monoco. It would write to monoco's old cluster, which is being kept intact purely as a rollback.

Not done here

• Rotating the secrets that are committed in plaintext — pg_password is the fleet's shared 123456, and it lives in bitwarden.env at mode 0600 rather than in the world-readable unit file.
• Closing signups, enabling the admin page, or configuring SMTP (see above).
• Replacing the two broken backup scripts with a systemd user timer.