# Caddy (rootless Podman + Quadlet) Runs the Caddy reverse proxy as a **rootless Podman** container managed by **systemd user units** (Quadlet). Replaces the previous `docker_container` role. **Caddy is the base unit and knows nothing about its upstreams.** It does not list any app network and does not need editing when you add a service. Each service registers *itself* with Caddy. That is the whole design. ## Layout ``` caddy/ ├── playbook.yml ├── vars/default.yml └── roles/ ├── podman/ # rootless prerequisites: podman, linger, port floor, Quadlet dir ├── caddy/ # the base caddy.container unit + handlers └── register/ # includable task file: a service registers itself with Caddy ``` Everything is installed under `{{ quadlet_dir }}` (`~/.config/containers/systemd/`). State lives under `{{ host_directory }}` (`~/podman-data/caddy/`). ## How services register themselves Caddy's unit is minimal. A service joins it by writing **two files** — its own network unit, and a Quadlet *drop-in* next to `caddy.container`: ``` ~/.config/containers/systemd/ ├── caddy.container # base unit, no app networks ├── caddy.container.d/ # ← drop-ins, one per service │ ├── 50-gitea.conf # [Container] Network=gitea.network │ ├── 50-nextcloud.conf # [Container] Network=nextcloud.network │ └── 50-mozimo.conf ├── gitea.network ├── nextcloud.network └── mozimo.network ``` Each service playbook does: ```yaml - name: Register this service with Caddy ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml" vars: network_name: "{{ container_name }}" ``` The play must also run the `podman` role (for `podman_user`, `quadlet_dir` and the host prerequisites) and define a `Restart Caddy` handler — see [roles/register/tasks/main.yaml](roles/register/tasks/main.yaml) for the exact snippet. ### Why drop-ins rather than editing caddy.container Editing the base file directly (e.g. with `blockinfile`) **breaks on the next run of the Caddy playbook**, because that playbook re-templates `caddy.container` and would wipe every service's appended block. Drop-ins sit beside the base file, survive re-templating, and are the mechanism Quadlet documents for exactly this. They also make removal trivial — delete `50-mozimo.conf`, `daemon-reload`, and Caddy drops that network. Nothing else changes. ### Trade-off worth knowing A new drop-in changes the generated unit, so Caddy must be **recreated** to pick up the network. That is a brief restart of *every* hosted site, not just the new one. Quadlet has no live equivalent of `docker network connect`. If that matters, connect live first and persist the drop-in in the same task: `podman network connect caddy` takes effect immediately, and the drop-in makes it survive the next recreate. The `register` role does not do this by default because it makes the play non-idempotent. ### `isolate` The generated network units set `Options=isolate=true`, which blocks traffic between separate Podman networks — an app on `nextcloud_net` cannot reach an app that is only on `gitea_net`. Caddy and shared services still reach both, because they are *members* of those networks. This is set explicitly on purpose: Podman 6.0 changed the default to `isolate=strict`, and older Podman defaulted differently. Being explicit means the isolation you get does not depend on which Podman version is installed. On Podman 6 you may prefer `strict`, which is stronger. ## Multi-network mechanics (the parts that bite) **A container joins several networks by repeating `Network=`.** Quadlet turns each `Network=.network` into a dependency on the generated network unit: ```ini [Container] Network=nextcloud.network Network=gitea.network ``` **You cannot use the `:alias=` form with a `.network` reference.** Quadlet only applies its `.network` mapping when the value *ends with* `.network`. So `Network=nextcloud.network:alias=db` is passed to Podman verbatim and fails, because no network is actually named `nextcloud.network`. Use the separate key: ```ini NetworkAlias=db # applies to every network the container joins ``` **Naming.** `nextcloud.network` would otherwise create a Podman network called `systemd-nextcloud`. `NetworkName=` in the `[Network]` section overrides that, so we keep the Docker-era `nextcloud_net` names. Note the indirection: the *file* is `.network`, the *Podman network* is `_net`. ## Settings | Variable | Purpose | |---|---| | `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`) — not set by hand | | `podman_data_dir` | Where state lives (`~/podman-data`), from `ansible_facts['user_dir']` | | `quadlet_dir` | Where `.container` / `.network` units are written | | `caddy_dropin_dir` | Where services register their networks | | `podman_unprivileged_port_start` | Lowers the privileged-port floor so rootless Caddy can publish 80/443 | | `caddy_image` | Fully qualified on purpose | | `caddy_ports` | Host port mappings, including 443/udp for HTTP/3 | | `caddy_name` | Container / unit name, and the prefix of the drop-in directory | ## Running The play targets `emily` directly (`hosts: emily`), so: ```command ansible-playbook -u tanshu caddy/playbook.yml ``` The `podman` role installs the `podman` package, enables lingering, lowers the privileged-port floor to 80, and creates the Quadlet directory. Target is Ubuntu 26.04 or later (Podman 5.7). Only `podman` is named explicitly. `aardvark-dns`, `nftables`, `passt` and `uidmap` arrive as **Recommends** of podman/netavark, which a normal `apt install` installs. That is fine by default, but it does mean they would go missing under `--no-install-recommends`. `aardvark-dns` is the one that matters most: it is the container-name DNS every `reverse_proxy :` depends on, and nothing else in the dependency graph guarantees it. If you ever build Podman from source, this role would also pull in the distro package — pin or skip it deliberately at that point. Order does not matter: services can be deployed before or after Caddy. A drop-in referencing a `.network` that does not exist yet will fail unit generation, so deploy the network unit and the drop-in together — which the `register` role does. ## The Caddyfile `caddyfile_path` (`~/podman-data/caddy/conf/Caddyfile`) is **not** templated here. It is seeded once and appended to by each service's `caddy` role via `blockinfile`, exactly as before — only the path changed from `/var/lib/caddy/conf/Caddyfile`. The seed matters: **Caddy exits 1 with a bare `EOF` on an empty config file**, so an empty Caddyfile makes the service crash-loop. The role therefore writes a minimal valid config when the file is missing or zero-length: ``` # Managed by Ansible (caddy/roles/caddy). # Global options go in the block below. Service playbooks append their # site blocks after it. { } ``` It only acts when the file is empty, so it never fights the appended site blocks. The global options block has to come *first* in a Caddyfile, which is why it is seeded up front rather than appended. Service roles must also update their reload handler, which currently runs `docker exec -w /etc/caddy caddy caddy reload`: ```yaml - name: Reload Caddy configuration ansible.builtin.command: >- podman exec caddy caddy reload --config /etc/caddy/Caddyfile listen: "Reload Caddy" ``` A reference handler is in [roles/caddy/handlers/main.yaml](roles/caddy/handlers/main.yaml). Note the container path is `/etc/caddy/Caddyfile` (the mount), not `caddyfile_path`. ## Who runs as whom `podman_user` is **derived from the connection**, not configured: ```yaml podman_user: "{{ ansible_facts['user_id'] }}" podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data" quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd" ``` The play carries no `become`, so the modules run as — and the Quadlet units belong to — whoever Ansible connects as. Running as that user is what gives us a real PAM session, which is what makes `systemctl --user` work, with `XDG_RUNTIME_DIR` set by `pam_systemd`. No `become_user`, no environment block. ``` ansible-playbook -u tanshu caddy/playbook.yml ``` The home directory comes from `ansible_facts['user_dir']`, so this is correct even for a user whose home is not `/home/`. This is why `gather_facts: true` is set explicitly on the play — the two facts above are the only configuration. Because the user is now implied rather than declared, the `podman` role **fails loudly if it resolves to `root`**. That is the one case where deriving it would silently produce something wrong: rootless units written into `/root`, and a `systemctl --user` operating on root's user manager. The usual cause is re-enabling `become` at play level, so the message says so. Using a *different* user is deliberately not supported — it would require `become_user` for the user-facing tasks, which is the thing this design removes. Only genuinely root-only steps elevate, each with its own `become: true`: | Task | Why root | |---|---| | `apt` install of podman + runtime pieces | package management | | `loginctl enable-linger` | modifies system-wide linger state | | `/etc/sysctl.d/99-podman.conf` + `sysctl -w …` | kernel setting | Everything else — directories, Quadlet files, `systemctl --user`, `podman exec` — runs unprivileged as the connection user. ## Operational notes Caddy is a **systemd user unit**: ``` systemctl --user status caddy systemctl --user restart caddy journalctl --user -u caddy -f podman logs caddy podman exec caddy caddy reload --config /etc/caddy/Caddyfile podman inspect caddy --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' ``` From a plain root shell, user units need the runtime dir — `sudo -u` does not give you a PAM session, which is exactly why the playbook avoids `become`: ``` sudo -u tanshu XDG_RUNTIME_DIR=/run/user/$(id -u tanshu) systemctl --user status caddy ``` Two prerequisites the `podman` role sets up, both easy to forget on a fresh instance: - **`loginctl enable-linger tanshu`** — otherwise user units stop when the last session ends and never start at boot. - **`net.ipv4.ip_unprivileged_port_start = 80`** — rootless containers cannot bind host ports below 1024 without it. The role writes `/etc/sysctl.d/99-podman.conf` **and** applies it with `sysctl -w` in the same run. It deliberately does not use a handler: handlers only fire at the end of the play, by which point Caddy has already tried and failed to bind 80/443. `systemctl --user enable caddy` is neither needed nor possible: the unit is *generated*, and Quadlet applies `[Install] WantedBy=default.target` itself. ### Debugging a unit that will not generate ``` /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun systemd-analyze --user --generators=true verify caddy.service ``` The most common cause is a `Network=` naming a `.network` file that does not exist. ## Client IPs (verify before relying on them) Whether the original client address survives the host-to-container port forward depends on the forwarding mechanism: - `rootlessport` — a userspace proxy that rewrites the source address. - `pasta` — kernel-level forwarding that preserves it. The upstream switch, `rootless_port_forwarder = "pasta"` in `containers.conf`, is Podman 6 only, so it is unavailable on Ubuntu 26.04's 5.7. **Not yet verified on emily.** Podman 5.7 uses pasta as the rootless network backend, and pasta is what binds the published ports — visible in the journal as `pasta[...]: Listen failed for HOST UDP port */443` when the port floor was still 1024. That shows pasta is in the path, but does not by itself prove the address is preserved end-to-end. Check it once a site is serving — the Caddy access log should show a real client address, not a gateway or `10.0.2.x`: ``` podman logs caddy | tail ``` If it is wrong and it matters (Nextcloud and Vaultwarden use client IPs for brute-force protection; Vince tracks them for analytics), there is a second route that works on stock Podman: **systemd socket activation**. systemd creates the listening socket and passes it in, which preserves the source address natively and at native speed. Caddy supports socket activation and `sd_notify`. It needs a `caddy.socket` unit, `Notify=true` on the container, and a `bind fd/3` directive in the Caddyfile — which is why it is not enabled here. ## Migrating from the Docker version 1. Deploy the services first, or at least their `.network` units. 2. Move state: `/var/lib/caddy` → `~/podman-data/caddy`, then `chown -R tanshu:tanshu ~/podman-data/caddy`. **Do not skip this** — `data/` holds the ACME account key and every issued certificate. Losing it means re-issuing ~20 certificates against Let's Encrypt rate limits. 3. Run this playbook. 4. Re-register each service (their `register` includes), then verify: `podman exec caddy getent hosts nextcloud` resolves, and `curl -sSI https://` works for each site block. Image drift to note: the running Docker container was `caddy:latest` while the old vars said `caddy:alpine`. This role uses `caddy:latest` to match reality.