Run Caddy as a rootless Podman Quadlet

Replaces the docker_container role with Quadlet systemd user units under
~/.config/containers/systemd, and moves state from /var/lib/caddy to
~/podman-data/caddy.

Caddy is now the base unit and no longer lists its upstream networks. Each
service registers itself by writing its own <name>.network unit plus a drop-in
into caddy.container.d/, so adding or removing a service never means editing the
Caddy playbook. Drop-ins are used rather than blockinfile because this playbook
re-templates caddy.container on every run, which would wipe appended blocks.

The play runs as the connecting user instead of using become: the rootless units
belong to it, so the connection's PAM session is what makes systemctl --user
work without setting XDG_RUNTIME_DIR. podman_user and the data directory are
derived from facts, so the play fails loudly if that resolves to root.

Two things that bit us and are worth keeping:

- The privileged-port sysctl is applied inline with sysctl -w, not via a
  handler. Handlers only run at the end of the play, by which point Caddy has
  already failed to bind 80/443.
- The Caddyfile is seeded with a minimal valid config. Caddy exits 1 with a
  bare "EOF" on an empty config file, so touching it into existence made the
  service crash-loop.

caddy/roles/caddy/handlers/main.yaml

@@ -0,0 +1,32 @@
+---
+# All handlers run as the connection user ({{ podman_user }}), so systemctl
+# --user works without setting XDG_RUNTIME_DIR.
+
+- name: Reload systemd user units
+  ansible.builtin.command: systemctl --user daemon-reload
+  changed_when: true
+  listen: Reload systemd user units
+
+- name: Restart Caddy
+  ansible.builtin.systemd_service:
+    name: "{{ caddy_unit }}"
+    scope: user
+    state: restarted
+    daemon_reload: true
+  listen: Restart Caddy
+
+# Reference implementation for service playbooks. A service that appends a site
+# block to {{ caddyfile_path }} should notify this instead of restarting the
+# container, and defines its own copy of the handler in its caddy role, e.g.:
+#
+#   - name: Reload Caddy configuration
+#     ansible.builtin.command: podman exec {{ caddy_name }} caddy reload --config /etc/caddy/Caddyfile
+#     listen: "Reload Caddy"
+#
+# Note the container path is /etc/caddy/Caddyfile (the mount), not caddyfile_path.
+- name: Reload Caddy configuration
+  ansible.builtin.command: >-
+    podman exec {{ caddy_name }}
+    caddy reload --config /etc/caddy/Caddyfile
+  changed_when: true
+  listen: Reload Caddy

caddy/roles/caddy/tasks/main.yaml

@@ -0,0 +1,115 @@
+---
+# Start Caddy
+#
+# Everything here runs as {{ podman_user }} — the connection user. No become,
+# no XDG_RUNTIME_DIR: SSH gave us a real PAM session, so systemd's user manager
+# is reachable via the standard environment.
+
+- name: Ensure Caddy directories exist
+  ansible.builtin.file:
+    path: "{{ host_directory }}/{{ item }}"
+    state: directory
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0755'
+  loop:
+    - conf
+    - site
+    - data
+    - config
+
+- name: Check whether the Caddyfile already has content
+  ansible.builtin.stat:
+    path: "{{ caddyfile_path }}"
+  register: caddyfile_stat
+
+# Caddy exits 1 with a bare "EOF" on an empty config file, so an empty
+# Caddyfile means caddy.service crash-loops. Seed a minimal valid config.
+#
+# Only acts when the file is missing or still empty, so it never fights the
+# site blocks that service playbooks append via blockinfile. The global
+# options block must come first, which is why it is seeded rather than
+# appended.
+- name: Seed the Caddyfile with a valid empty config
+  ansible.builtin.copy:
+    dest: "{{ caddyfile_path }}"
+    content: |
+      # Managed by Ansible (caddy/roles/caddy).
+      # Global options go in the block below. Service playbooks append their
+      # site blocks after it.
+      {
+      }
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+  when: not caddyfile_stat.stat.exists or caddyfile_stat.stat.size == 0
+
+- name: Ensure the Quadlet drop-in directory exists
+  ansible.builtin.file:
+    path: "{{ caddy_dropin_dir }}"
+    state: directory
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0755'
+
+- name: Install the base Caddy Quadlet unit
+  ansible.builtin.template:
+    src: caddy.container.j2
+    dest: "{{ quadlet_dir }}/{{ caddy_name }}.container"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+  notify:
+    - Reload systemd user units
+    - Restart Caddy
+
+- name: Start Caddy
+  block:
+    - name: Start Caddy
+      ansible.builtin.systemd_service:
+        name: "{{ caddy_unit }}"
+        scope: user
+        state: started
+        daemon_reload: true
+  rescue:
+    # systemd reports "control process exited with error code" and nothing
+    # else, which is useless for the usual failures (a port already bound, a
+    # missing image, a bad volume). Pull the real reason out.
+    - name: Collect Caddy diagnostics
+      ansible.builtin.command: "{{ item }}"
+      loop:
+        - "systemctl --user status {{ caddy_unit }} --no-pager"
+        - "journalctl --user -u {{ caddy_unit }} --no-pager -n 40"
+      register: caddy_diag
+      changed_when: false
+      failed_when: false
+
+    - name: Show Caddy diagnostics
+      ansible.builtin.debug:
+        msg: "{{ caddy_diag.results | map(attribute='stdout') | list }}"
+
+    - name: Fail
+      ansible.builtin.fail:
+        msg: "Caddy did not start — see the diagnostics above."
+
+# Quadlet honours [Install] WantedBy= when it generates the unit, so
+# `systemctl enable` is neither needed nor possible on a generated unit.
+- name: Verify Caddy is running
+  ansible.builtin.command: "systemctl --user is-active {{ caddy_unit }}"
+  register: caddy_active
+  changed_when: false
+  failed_when: caddy_active.stdout != 'active'
+
+- name: Show the networks Caddy is currently attached to
+  ansible.builtin.command: >-
+    podman inspect {{ caddy_name }}
+    --format '{% raw %}{{ range $k, $v := .NetworkSettings.Networks }}{{ $k }} {{ end }}{% endraw %}'
+  register: caddy_networks_now
+  changed_when: false
+  failed_when: false
+
+- name: Report registered networks
+  ansible.builtin.debug:
+    msg: >-
+      Caddy networks:
+      {{ caddy_networks_now.stdout | default('(container not running)') }}

caddy/roles/caddy/templates/caddy.container.j2

@@ -0,0 +1,40 @@
+# Caddy reverse proxy — rootless Podman Quadlet unit.
+# Managed by Ansible (caddy/roles/caddy). Do not edit by hand.
+#
+# Caddy deliberately does NOT list its upstream networks here. Each service
+# registers its own network by dropping a file into
+# {{ caddy_dropin_dir }}/, which Quadlet merges into this unit.
+# See readme.md, "How services register themselves".
+#
+# The Caddyfile is likewise not templated here: each service appends its own
+# site block to {{ caddyfile_path }} and then reloads Caddy.
+
+[Unit]
+Description=Caddy reverse proxy
+Documentation=https://caddyserver.com/docs/
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName={{ caddy_name }}
+Image={{ caddy_image }}
+
+# Map the container's root (uid 0) to {{ podman_user }} on the host, so
+# certificates and the Caddyfile stay owned by {{ podman_user }} and never by
+# a subuid. Caddy can still bind 80/443 inside its own network namespace.
+UserNS=keep-id:uid=0,gid=0
+
+{% for port in caddy_ports %}
+PublishPort={{ port }}
+{% endfor %}
+Volume={{ host_directory }}/conf:/etc/caddy
+Volume={{ host_directory }}/site:/srv
+Volume={{ host_directory }}/data:/data
+Volume={{ host_directory }}/config:/config
+
+[Service]
+Restart=always
+TimeoutStartSec=120
+
+[Install]
+WantedBy=default.target

caddy/roles/podman/tasks/main.yaml

@@ -0,0 +1,80 @@
+---
+# Rootless Podman prerequisites. Shared by every container playbook.
+#
+# This playbook runs AS the container user; it does not use become to reach it.
+# Root-only steps elevate individually. Keeping the connection's PAM session
+# intact is what lets `systemctl --user` work without setting XDG_RUNTIME_DIR.
+
+- name: Fail if the connecting user resolved to root
+  ansible.builtin.assert:
+    that: podman_user != 'root'
+    fail_msg: >-
+      podman_user resolved to root, but this playbook configures *rootless*
+      Podman and writes Quadlet units into the user's home. Connect as the
+      unprivileged user that should own the containers, e.g.
+      `ansible-playbook -u tanshu ...`. If this appeared because become was
+      re-enabled at play level, drop it: the root-only tasks below elevate
+      individually.
+
+- name: Install Podman and the rootless runtime pieces
+  ansible.builtin.apt:
+    name:
+      - podman
+      # Everything below is only a Recommends, not a Depends, of podman or
+      # netavark. apt installs Recommends by default, so these would normally
+      # arrive on their own; listing them makes the requirement explicit and
+      # keeps it true under --no-install-recommends.
+      # - aardvark-dns   # container-name DNS: Caddy resolves every upstream by name
+      # - nftables       # netavark's firewall backend; Podman 6 dropped iptables
+      # - passt          # pasta, the rootless network backend
+      # - uidmap         # newuidmap/newgidmap, required for rootless user namespaces
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  become: true
+
+# Without lingering, the user's systemd manager is torn down when the last
+# session ends: Quadlet units stop, and they never start at boot.
+- name: Enable systemd lingering for {{ podman_user }}
+  ansible.builtin.command: "loginctl enable-linger {{ podman_user }}"
+  args:
+    creates: "/var/lib/systemd/linger/{{ podman_user }}"
+  become: true
+
+# Rootless containers cannot bind host ports below 1024 unless the kernel
+# floor is lowered. Caddy publishes 80/443.
+#
+# This is applied inline rather than via a handler on purpose: handlers only
+# run at the end of the play, by which point Caddy has already tried (and
+# failed) to bind port 80. sysctl -w touches the running kernel; the file
+# makes it survive a reboot.
+- name: Read the current unprivileged port floor
+  ansible.builtin.command: sysctl -n net.ipv4.ip_unprivileged_port_start
+  register: podman_port_floor
+  changed_when: false
+
+- name: Persist the unprivileged port floor
+  ansible.builtin.copy:
+    dest: /etc/sysctl.d/99-podman.conf
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by Ansible (podman role).
+      net.ipv4.ip_unprivileged_port_start = {{ podman_unprivileged_port_start }}
+  become: true
+
+- name: Apply the unprivileged port floor
+  ansible.builtin.command: >-
+    sysctl -w net.ipv4.ip_unprivileged_port_start={{ podman_unprivileged_port_start }}
+  become: true
+  when: podman_port_floor.stdout | int > podman_unprivileged_port_start
+  changed_when: true
+
+- name: Ensure the Quadlet unit directory exists
+  ansible.builtin.file:
+    path: "{{ quadlet_dir }}"
+    state: directory
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: "0755"

caddy/roles/register/tasks/main.yaml

@@ -0,0 +1,66 @@
+---
+# Register a service's logical network with Caddy.
+#
+# Caddy is the *base* unit and knows nothing about its upstreams. Each service
+# adds itself by writing two files into the Quadlet directory:
+#
+#   1. its own <name>.network unit, and
+#   2. a drop-in next to caddy.container that pulls that network into Caddy.
+#
+# Caddy's own playbook never changes when a service is added or removed.
+#
+# Include this from a service playbook:
+#
+#   - name: Register this service with Caddy
+#     ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
+#     vars:
+#       network_name: "{{ container_name }}"
+#
+# The including play must also run the `podman` role (for podman_user and
+# quadlet_dir) and define a "Restart Caddy" handler:
+#
+#   - name: Restart Caddy
+#     ansible.builtin.systemd_service:
+#       name: caddy.service
+#       scope: user
+#       state: restarted
+#       daemon_reload: true
+#     listen: Restart Caddy
+#
+# It runs as the connection user, which must be {{ podman_user }}.
+
+- name: "Create the {{ network_name }} network unit"
+  ansible.builtin.copy:
+    dest: "{{ quadlet_dir }}/{{ network_name }}.network"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+    content: |
+      # {{ network_name }} — logical app network.
+      # Managed by Ansible. Do not edit by hand.
+      #
+      # NetworkName pins the Podman network name. Without it Quadlet would
+      # create "systemd-{{ network_name }}".
+      #
+      # isolate blocks traffic between separate Podman networks, so a container
+      # on this network cannot reach one that is only on another. Caddy and any
+      # shared service still reach it, because they are members of it.
+      [Network]
+      NetworkName={{ network_name }}_net
+      Options=isolate=true
+  notify: Restart Caddy
+
+- name: "Register {{ network_name }} with Caddy"
+  ansible.builtin.copy:
+    dest: "{{ quadlet_dir }}/{{ caddy_name | default('caddy') }}.container.d/50-{{ network_name }}.conf"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+    content: |
+      # Added by the {{ network_name }} playbook. Do not edit by hand.
+      #
+      # Quadlet merges this into caddy.container. Deleting this file removes
+      # the network from Caddy on the next daemon-reload.
+      [Container]
+      Network={{ network_name }}.network
+  notify: Restart Caddy