Run Bitwarden as a rootless Podman Quadlet

Replaces the docker_container role on monoco with a Quadlet systemd user unit on
emily, and moves state from /var/lib/bitwarden to ~/podman-data/bitwarden.
Postgres is authoritative for the vault and already moved with the cluster
restore, so only the file data comes across -- and the stale db.sqlite3 that has
sat beside it for years is deliberately excluded from that copy. The role warns
if it finds the file in the data directory rather than failing, because with
DATABASE_URL set it is inert but looks exactly like a restore source.

The image runs as container-root and writes /data as root, so
UserNS=keep-id:uid=0,gid=0 maps that onto the connecting user and leaves nothing
owned by a subuid.

Both integrations go through the shared includes rather than local copies --
caddy/roles/register for the network and the Caddy drop-in, postgres/roles/attach
for the database -- so this app writes the same drop-in bytes as every other
service and a new service never means editing Caddy or Postgres. That is also
what lets the Docker-era roles/network go: it existed to attach an
already-created container to a network (docker_network's connected:/appends:),
which podman_network has no equivalent for. The dead files/nginx.conf.j2 goes
with it -- Caddy handles the WebSocket upgrade and imposes no request-body limit,
so its 525M client_max_body_size needs no replacement.

Three ordering details are load-bearing:

- The register include runs in pre_tasks, before the unit is started. Quadlet
  fails unit *generation*, not merely startup, when a referenced .network file
  does not exist yet, so starting first reports "unit not found".
- roles/postgres flushes handlers immediately after attaching. The restart that
  joins Postgres to this network is a handler, so by default it runs at the end
  of the play, after the app has already started and failed to resolve the
  database.
- The play waits on the image's own health check (podman healthcheck run) rather
  than on systemctl is-active. Vaultwarden runs its Diesel migrations before
  Rocket answers, and Restart=always means even a crash-looping container is
  reported active.

DATABASE_URL and DOMAIN move into a 0600 environment file read through
EnvironmentFile=, since the Quadlet unit is world-readable and DATABASE_URL
carries the password. The credentials are unchanged, so the restored database
connects with no reset. readme.md carries the monoco -> emily file copy command
by command.

bitwarden/roles/bitwarden/handlers/main.yaml

@@ -0,0 +1,20 @@
+---
+# All handlers run as the connection user ({{ podman_user }}), so systemctl
+# --user works without setting XDG_RUNTIME_DIR.
+
+- name: Reload systemd user units
+  ansible.builtin.command: systemctl --user daemon-reload
+  changed_when: true
+  listen: Reload systemd user units
+
+# A changed environment file or unit file changes the generated unit, so
+# Vaultwarden is recreated. The vault itself is in Postgres and the attachments
+# are on a bind mount, so a recreate loses nothing — but it is a few seconds of
+# downtime for the one site.
+- name: Restart Bitwarden
+  ansible.builtin.systemd_service:
+    name: "{{ unit_name }}"
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Bitwarden

bitwarden/roles/bitwarden/tasks/main.yaml

@@ -1,27 +1,138 @@
 ---
-# Tasks for Bitwarden role
-- name: Pull Bitwarden Vault Warden image
-  docker_image:
-    name: "{{ image_name }}"
-    source: pull
-    force_source: yes
+# Vaultwarden (the Bitwarden-compatible server) as a rootless Podman Quadlet
+# unit.
+#
+# Everything here runs as {{ podman_user }} — the connection user. No become, no
+# XDG_RUNTIME_DIR: the SSH connection gave us a real PAM session, so systemd's
+# user manager is reachable through the standard environment.
 
-- name: Ensure Bitwarden Directory exists
-  file:
-      path: "/var/lib/{{ host_directory }}"
-      state: directory
-      mode: 0755
+# /data is the image's VOLUME, and it holds everything Vaultwarden owns that is
+# not in Postgres: attachments/, sends/, the rsa_key signing pair, the admin
+# config and the icon cache. Postgres is authoritative for the vault itself
+# (docs/backup-plan.md §3.3).
+- name: Ensure the Vaultwarden data directory exists
+  ansible.builtin.file:
+    path: "{{ data_dir }}"
+    state: directory
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0755'
 
-- name: Create bitwarden container
-  docker_container:
-    name: "{{ docker_container }}"
-    image: "{{ image_name }}"
-    state: started
-    restart_policy: "unless-stopped"
-    env:
-      DATABASE_URL: "{{ db_url }}"
-      DOMAIN: "https://{{ host }}"
-    volumes:
-      - "/var/lib/{{ host_directory }}/data:/data/"
-    networks:
-      - name: "{{ docker_network }}"
+# The app's secrets and database coordinates are consumed by systemd
+# (EnvironmentFile= in the Quadlet unit), not by Ansible, so this has to be
+# written before the unit starts. Mode 0600: DATABASE_URL carries the Postgres
+# password.
+- name: Write the environment file
+  ansible.builtin.template:
+    src: bitwarden.env.j2
+    dest: "{{ env_file }}"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0600'
+  notify: Restart Bitwarden
+
+- name: Install the Vaultwarden Quadlet unit
+  ansible.builtin.template:
+    src: bitwarden.container.j2
+    dest: "{{ quadlet_dir }}/{{ container_name }}.container"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+  notify:
+    - Reload systemd user units
+    - Restart Bitwarden
+
+- name: Start Vaultwarden
+  block:
+    - name: Start Vaultwarden
+      ansible.builtin.systemd_service:
+        name: "{{ unit_name }}"
+        scope: user
+        state: started
+        daemon_reload: true
+  rescue:
+    # systemd reports "control process exited with error code" and nothing
+    # else, which is useless for the usual failures (an unattached network, an
+    # unreachable database, a bad volume). Pull the real reason out.
+    - name: Collect Vaultwarden diagnostics
+      ansible.builtin.command: "{{ item }}"
+      loop:
+        - "systemctl --user status {{ unit_name }} --no-pager"
+        - "journalctl --user -u {{ unit_name }} --no-pager -n 40"
+        - "podman logs --tail 40 {{ container_name }}"
+      register: bitwarden_diag
+      changed_when: false
+      failed_when: false
+
+    - name: Show Vaultwarden diagnostics
+      ansible.builtin.debug:
+        msg: "{{ bitwarden_diag.results | map(attribute='stdout') | list }}"
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: "Vaultwarden did not start — see the diagnostics above."
+
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl --user enable` is neither needed nor possible on a generated unit.
+- name: Verify Vaultwarden is running
+  ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
+  register: bitwarden_active
+  changed_when: false
+  failed_when: bitwarden_active.stdout != 'active'
+
+# `is-active` says the container is running; it does not say the app can serve.
+# Vaultwarden runs its Diesel migrations against Postgres before Rocket starts
+# answering, and `Restart=always` means even a crash-looping container is
+# reported active.
+#
+# `podman healthcheck run` executes the image's *own* HEALTHCHECK — which is
+# /healthcheck.sh curling http://localhost:80/alive — so this depends on nothing
+# the play has to know about: no curl, wget or python3 inside the image, and no
+# hardcoded path that a later image tag could move.
+- name: Wait for Vaultwarden to pass its own health check
+  ansible.builtin.command: "podman healthcheck run {{ container_name }}"
+  register: bitwarden_health
+  until: bitwarden_health.rc == 0
+  retries: 24
+  delay: 5
+  changed_when: false
+
+# The database is the whole vault, and it arrives over the network that
+# roles/postgres attached Postgres to. Check the name the environment file
+# actually asks for, plus the shared `db` alias.
+- name: Confirm Vaultwarden can resolve its database
+  ansible.builtin.command: >-
+    podman exec {{ container_name }} getent hosts {{ item }}
+  loop:
+    - "{{ postgres_container }}"
+    - db
+  register: bitwarden_db_lookup
+  changed_when: false
+  failed_when: false
+
+- name: Report database resolution
+  ansible.builtin.debug:
+    msg: >-
+      {{ item.item }} resolves as
+      {{ item.stdout | default('(no answer)') }}
+  loop: "{{ bitwarden_db_lookup.results }}"
+
+- name: Check for a stale SQLite database in the data directory
+  ansible.builtin.stat:
+    path: "{{ data_dir }}/db.sqlite3"
+  register: bitwarden_stale_sqlite
+
+# SQLite in /data is the image's own default, and the Docker-era deployment left
+# a stale db.sqlite3 (with -wal/-shm sidecars) behind long after DATABASE_URL had
+# been pointed at Postgres. This deployment ignores the file entirely, which is
+# exactly why it is dangerous: it looks like a restore source, and restoring it
+# would silently lose months of vault data (docs/backup-plan.md §3.3). Warn
+# rather than fail, because the file is inert while DATABASE_URL is set.
+- name: Warn about the stale SQLite database
+  ansible.builtin.debug:
+    msg: >-
+      {{ data_dir }}/db.sqlite3 exists, but Postgres is the authoritative store
+      for this deployment and Vaultwarden is not reading it. It is not a backup
+      and must not be restored. Delete it once the Postgres database has been
+      verified — see readme.md, "Migrating the file data".
+  when: bitwarden_stale_sqlite.stat.exists

bitwarden/roles/bitwarden/templates/bitwarden.container.j2

@@ -0,0 +1,45 @@
+# Vaultwarden (Bitwarden-compatible server) — rootless Podman Quadlet unit.
+# Managed by Ansible (bitwarden/roles/bitwarden). Do not edit by hand.
+#
+# This unit joins {{ network_name }}.network, which is created by
+# caddy/roles/register (the same include that attaches Caddy to it). The
+# postgres container is joined to that network by postgres/roles/attach, which
+# is what makes the host in DATABASE_URL resolve.
+
+[Unit]
+Description=Vaultwarden (Bitwarden-compatible server)
+Documentation=https://github.com/dani-garcia/vaultwarden/wiki
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName={{ container_name }}
+Image={{ image }}
+
+# Vaultwarden's image runs as container-root and writes /data as root, with no
+# entrypoint step that chowns it (see docs/podman-compat-audit.md §3.2).
+# Mapping container root onto {{ podman_user }} keeps {{ data_dir }} owned by
+# {{ podman_user }} on the host, so plain tar/rsync backups keep working and no
+# subuid-owned files appear.
+UserNS=keep-id:uid=0,gid=0
+
+# DATABASE_URL and DOMAIN live here rather than as Environment= lines above:
+# this unit is world-readable in ~/.config, and DATABASE_URL carries the
+# Postgres password. Read by systemd, not by Ansible, so the file has to exist
+# before this unit starts.
+EnvironmentFile={{ env_file }}
+
+Network={{ network_name }}.network
+
+# The image's VOLUME. Everything Vaultwarden owns outside Postgres — attachments,
+# sends, the rsa_key signing pair, the admin config — lives here.
+Volume={{ data_dir }}:/data
+
+[Service]
+Restart=always
+# The first start runs Diesel migrations against Postgres before Rocket begins
+# answering on port {{ container_port }}.
+TimeoutStartSec=120
+
+[Install]
+WantedBy=default.target

bitwarden/roles/bitwarden/templates/bitwarden.env.j2

@@ -0,0 +1,19 @@
+# Environment for {{ container_name }}.container.
+# Managed by Ansible (bitwarden/roles/bitwarden). Do not edit by hand.
+#
+# Mode 0600 on purpose: the Quadlet unit that reads this file is world-readable,
+# and DATABASE_URL carries the Postgres password. Note that `podman inspect`
+# still prints the resulting environment to anyone who can run podman as this
+# user.
+#
+# Postgres is authoritative for the vault. The SQLite file the image would use
+# by default ({{ data_dir }}/db.sqlite3) is stale on the old host and is
+# deliberately not migrated — see readme.md, "Migrating the file data".
+DATABASE_URL={{ database_url }}
+DOMAIN={{ domain }}
+
+# Deliberately not set, so this deployment behaves exactly as the Docker-era
+# container did: SIGNUPS_ALLOWED is still the image default (true), the admin
+# panel is off (no ADMIN_TOKEN), and SMTP is unconfigured. Tightening any of
+# those is a behaviour change, not a migration step — see readme.md,
+# "Notes and gotchas".

bitwarden/roles/caddy/handlers/main.yaml

@@ -1,5 +1,31 @@
 ---
-# Handlers for Caddy role
+# Caddy is its own playbook (caddy/playbook.yml) and a rootless Quadlet unit
+# there. These handlers run as the connection user, so `systemctl --user` and
+# `podman exec` work without any XDG_RUNTIME_DIR juggling.
+
+# Notified by this role when the Caddyfile changes. A reload keeps existing
+# connections; a restart would briefly drop every hosted site.
+#
+# Note the config path: /etc/caddy/Caddyfile is the mount inside the container,
+# not {{ caddyfile_path }}, which is the host side of it.
 - name: Reload Caddy configuration
-  ansible.builtin.command: "docker exec -w /etc/caddy {{ caddy_container }} caddy reload"
-  listen: "Reload Caddy"
+  ansible.builtin.command: >-
+    podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
+  changed_when: true
+  listen: "Reload Caddy"
+
+# Notified by caddy/roles/register when this service's network unit, or the
+# drop-in that attaches Caddy to it, is written. Quadlet has no live equivalent
+# of `docker network connect`, so the container has to be recreated — a brief
+# restart of every hosted site, not just this one. See caddy/readme.md,
+# "Trade-off worth knowing".
+#
+# Defined after the reload above so that when both are notified the cheap
+# config reload happens first.
+- name: Restart Caddy
+  ansible.builtin.systemd_service:
+    name: "{{ caddy_name }}.service"
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Caddy

bitwarden/roles/caddy/tasks/main.yaml

@@ -1,22 +1,22 @@
 ---
-- name: Read current Caddyfile
-  ansible.builtin.slurp:
-    path: "{{ caddyfile_path }}"
-  register: caddyfile_raw
+# Append this site's block to the shared Caddyfile.
+#
+# The Caddyfile is a bind mount into the Caddy container, so Caddy sees the
+# change as soon as the file is written; the handler only asks Caddy to re-read
+# it. That is a reload, not a restart, so the other hosted sites are untouched.
+#
+# blockinfile is idempotent on its own via the marker, so there is no need to
+# slurp the file and compare first — which also matters because slurp *fails*
+# when the Caddyfile does not exist yet, on a host where the Caddy playbook has
+# not run.
 
-- name: Decode Caddyfile content
-  set_fact:
-    caddyfile_content: "{{ caddyfile_raw['content'] | b64decode }}"
-
-- name: Check if snippet already exists
-  set_fact:
-    snippet_present: "{{ snippet_block in caddyfile_content }}"
-
-- name: Add snippet if missing
+- name: "Add the {{ host }} site block to the Caddyfile"
   ansible.builtin.blockinfile:
     path: "{{ caddyfile_path }}"
     marker: "# {mark} Ansible managed Caddy snippet for {{ host }}"
     block: "{{ snippet_block }}"
     create: yes
-  when: not snippet_present
-  notify: "Reload Caddy"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+  notify: Reload Caddy

bitwarden/roles/network/tasks/main.yaml

@@ -1,9 +0,0 @@
----
-- name: "Ensure {{ docker_container }} Docker network exists"
-  docker_network:
-    name: "{{ docker_network }}"
-    state: present
-    connected:
-      - "{{ caddy_container }}"
-      - "{{ postgres_container }}"
-    appends: yes

bitwarden/roles/postgres/handlers/main.yaml

@@ -0,0 +1,17 @@
+---
+# Postgres is its own playbook (postgres/playbook.yml) and a rootless Quadlet
+# unit there. This handler runs as the connection user, so `systemctl --user`
+# works without setting XDG_RUNTIME_DIR.
+#
+# Required by postgres/roles/attach, which notifies it whenever this app's
+# drop-in changes: a new drop-in changes the generated unit, so Postgres must be
+# recreated to join the network — Quadlet has no live equivalent of
+# `podman network connect`. That is a few seconds of downtime for every attached
+# app, not just this one. See postgres/readme.md, "The trade-off".
+- name: Restart Postgres
+  ansible.builtin.systemd_service:
+    name: "{{ postgres_unit }}"
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Postgres

bitwarden/roles/postgres/tasks/main.yaml

@@ -0,0 +1,26 @@
+---
+# Attach this app's network to the shared Postgres unit.
+#
+# Postgres knows nothing about its clients, exactly like Caddy: a service joins
+# by dropping 50-bitwarden.conf and the shared 10-network-aliases.conf next to
+# postgres.container. The canonical way to do that is the shared include below —
+# it writes the same bytes every service writes, asserts that bitwarden.network
+# already exists, and notifies the "Restart Postgres" handler defined in this
+# role.
+#
+# This runs in the roles phase, which is after the register include in
+# playbook.yml's pre_tasks has created bitwarden.network.
+
+- name: Attach Bitwarden to Postgres
+  ansible.builtin.include_tasks: "{{ playbook_dir }}/../postgres/roles/attach/tasks/main.yaml"
+
+# On a first attach both drop-ins are new, so Postgres has to be recreated to
+# join bitwarden_net — and that restart is a *handler*, which by default runs at
+# the end of the play, after the bitwarden role has already started the app. The
+# app would then come up before DATABASE_URL's host resolves, fail its own
+# health check, and fail the play on a machine where nothing is actually wrong.
+#
+# Flushing applies the new drop-in before the app starts. On later runs the
+# drop-ins are unchanged, nothing is pending, and this is a no-op.
+- name: Apply the Postgres network change before starting Vaultwarden
+  ansible.builtin.meta: flush_handlers