playbooks/caddy/readme.md

Caddy (rootless Podman + Quadlet)

Runs the Caddy reverse proxy as a rootless Podman container managed by systemd user units (Quadlet). Replaces the previous docker_container role.

Caddy is the base unit and knows nothing about its upstreams. It does not list any app network and does not need editing when you add a service. Each service registers itself with Caddy. That is the whole design.

Layout

caddy/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── podman/     # rootless prerequisites: podman, linger, port floor, Quadlet dir
    ├── caddy/      # the base caddy.container unit + handlers
    └── register/   # includable task file: a service registers itself with Caddy

Everything is installed under {{ quadlet_dir }} (~/.config/containers/systemd/). State lives under {{ host_directory }} (~/podman-data/caddy/).

How services register themselves

Caddy's unit is minimal. A service joins it by writing two files — its own network unit, and a Quadlet drop-in next to caddy.container:

~/.config/containers/systemd/
├── caddy.container                  # base unit, no app networks
├── caddy.container.d/               # ← drop-ins, one per service
│   ├── 50-gitea.conf                # [Container] Network=gitea.network
│   ├── 50-nextcloud.conf            # [Container] Network=nextcloud.network
│   └── 50-mozimo.conf
├── gitea.network
├── nextcloud.network
└── mozimo.network

Each service playbook does:

- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

The play must also run the podman role (for podman_user, quadlet_dir and the host prerequisites) and define a Restart Caddy handler — see roles/register/tasks/main.yaml for the exact snippet.

Why drop-ins rather than editing caddy.container

Editing the base file directly (e.g. with blockinfile) breaks on the next run of the Caddy playbook, because that playbook re-templates caddy.container and would wipe every service's appended block. Drop-ins sit beside the base file, survive re-templating, and are the mechanism Quadlet documents for exactly this.

They also make removal trivial — delete 50-mozimo.conf, daemon-reload, and Caddy drops that network. Nothing else changes.

Trade-off worth knowing

A new drop-in changes the generated unit, so Caddy must be recreated to pick up the network. That is a brief restart of every hosted site, not just the new one. Quadlet has no live equivalent of docker network connect.

If that matters, connect live first and persist the drop-in in the same task: podman network connect <net> caddy takes effect immediately, and the drop-in makes it survive the next recreate. The register role does not do this by default because it makes the play non-idempotent.

isolate

The generated network units set Options=isolate=true, which blocks traffic between separate Podman networks — an app on nextcloud_net cannot reach an app that is only on gitea_net. Caddy and shared services still reach both, because they are members of those networks.

This is set explicitly on purpose: Podman 6.0 changed the default to isolate=strict, and older Podman defaulted differently. Being explicit means the isolation you get does not depend on which Podman version is installed. On Podman 6 you may prefer strict, which is stronger.

Multi-network mechanics (the parts that bite)

A container joins several networks by repeating Network=. Quadlet turns each Network=<name>.network into a dependency on the generated network unit:

[Container]
Network=nextcloud.network
Network=gitea.network

You cannot use the :alias= form with a .network reference. Quadlet only applies its .network mapping when the value ends with .network. So Network=nextcloud.network:alias=db is passed to Podman verbatim and fails, because no network is actually named nextcloud.network. Use the separate key:

NetworkAlias=db     # applies to every network the container joins

Naming. nextcloud.network would otherwise create a Podman network called systemd-nextcloud. NetworkName= in the [Network] section overrides that, so we keep the Docker-era nextcloud_net names. Note the indirection: the file is <name>.network, the Podman network is <name>_net.

Settings

Variable	Purpose
podman_user	Derived from the connection (ansible_facts['user_id']) — not set by hand
podman_data_dir	Where state lives (~/podman-data), from ansible_facts['user_dir']
quadlet_dir	Where .container / .network units are written
caddy_dropin_dir	Where services register their networks
podman_unprivileged_port_start	Lowers the privileged-port floor so rootless Caddy can publish 80/443
caddy_image	Fully qualified on purpose
caddy_ports	Host port mappings, including 443/udp for HTTP/3
caddy_name	Container / unit name, and the prefix of the drop-in directory

Running

The play targets emily directly (hosts: emily), so:

ansible-playbook -u tanshu caddy/playbook.yml

The podman role installs the podman package, enables lingering, lowers the privileged-port floor to 80, and creates the Quadlet directory. Target is Ubuntu 26.04 or later (Podman 5.7).

Only podman is named explicitly. aardvark-dns, nftables, passt and uidmap arrive as Recommends of podman/netavark, which a normal apt install installs. That is fine by default, but it does mean they would go missing under --no-install-recommends. aardvark-dns is the one that matters most: it is the container-name DNS every reverse_proxy <name>:<port> depends on, and nothing else in the dependency graph guarantees it.

If you ever build Podman from source, this role would also pull in the distro package — pin or skip it deliberately at that point.

Order does not matter: services can be deployed before or after Caddy. A drop-in referencing a .network that does not exist yet will fail unit generation, so deploy the network unit and the drop-in together — which the register role does.

The Caddyfile

caddyfile_path (~/podman-data/caddy/conf/Caddyfile) is not templated here. It is seeded once and appended to by each service's caddy role via blockinfile, exactly as before — only the path changed from /var/lib/caddy/conf/Caddyfile.

The seed matters: Caddy exits 1 with a bare EOF on an empty config file, so an empty Caddyfile makes the service crash-loop. The role therefore writes a minimal valid config when the file is missing or zero-length:

# Managed by Ansible (caddy/roles/caddy).
# Global options go in the block below. Service playbooks append their
# site blocks after it.
{
}

It only acts when the file is empty, so it never fights the appended site blocks. The global options block has to come first in a Caddyfile, which is why it is seeded up front rather than appended.

Service roles must also update their reload handler, which currently runs docker exec -w /etc/caddy caddy caddy reload:

- name: Reload Caddy configuration
  ansible.builtin.command: >-
    podman exec caddy caddy reload --config /etc/caddy/Caddyfile
  listen: "Reload Caddy"

A reference handler is in roles/caddy/handlers/main.yaml. Note the container path is /etc/caddy/Caddyfile (the mount), not caddyfile_path.

Who runs as whom

podman_user is derived from the connection, not configured:

podman_user: "{{ ansible_facts['user_id'] }}"
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"

The play carries no become, so the modules run as — and the Quadlet units belong to — whoever Ansible connects as. Running as that user is what gives us a real PAM session, which is what makes systemctl --user work, with XDG_RUNTIME_DIR set by pam_systemd. No become_user, no environment block.

ansible-playbook -u tanshu caddy/playbook.yml

The home directory comes from ansible_facts['user_dir'], so this is correct even for a user whose home is not /home/<name>. This is why gather_facts: true is set explicitly on the play — the two facts above are the only configuration.

Because the user is now implied rather than declared, the podman role fails loudly if it resolves to root. That is the one case where deriving it would silently produce something wrong: rootless units written into /root, and a systemctl --user operating on root's user manager. The usual cause is re-enabling become at play level, so the message says so.

Using a different user is deliberately not supported — it would require become_user for the user-facing tasks, which is the thing this design removes. Only genuinely root-only steps elevate, each with its own become: true:

Task	Why root
apt install of podman + runtime pieces	package management
loginctl enable-linger	modifies system-wide linger state
/etc/sysctl.d/99-podman.conf + sysctl -w …	kernel setting

Everything else — directories, Quadlet files, systemctl --user, podman exec — runs unprivileged as the connection user.

Operational notes

Caddy is a systemd user unit:

systemctl --user status caddy
systemctl --user restart caddy
journalctl --user -u caddy -f
podman logs caddy
podman exec caddy caddy reload --config /etc/caddy/Caddyfile
podman inspect caddy --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'

From a plain root shell, user units need the runtime dir — sudo -u does not give you a PAM session, which is exactly why the playbook avoids become:

sudo -u tanshu XDG_RUNTIME_DIR=/run/user/$(id -u tanshu) systemctl --user status caddy

Two prerequisites the podman role sets up, both easy to forget on a fresh instance:

• loginctl enable-linger tanshu — otherwise user units stop when the last session ends and never start at boot.
• net.ipv4.ip_unprivileged_port_start = 80 — rootless containers cannot bind host ports below 1024 without it. The role writes /etc/sysctl.d/99-podman.conf and applies it with sysctl -w in the same run. It deliberately does not use a handler: handlers only fire at the end of the play, by which point Caddy has already tried and failed to bind 80/443.

systemctl --user enable caddy is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Debugging a unit that will not generate

/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
systemd-analyze --user --generators=true verify caddy.service

The most common cause is a Network= naming a .network file that does not exist.

Client IPs (verify before relying on them)

Whether the original client address survives the host-to-container port forward depends on the forwarding mechanism:

• rootlessport — a userspace proxy that rewrites the source address.
• pasta — kernel-level forwarding that preserves it.

The upstream switch, rootless_port_forwarder = "pasta" in containers.conf, is Podman 6 only, so it is unavailable on Ubuntu 26.04's 5.7.

Not yet verified on emily. Podman 5.7 uses pasta as the rootless network backend, and pasta is what binds the published ports — visible in the journal as pasta[...]: Listen failed for HOST UDP port */443 when the port floor was still 1024. That shows pasta is in the path, but does not by itself prove the address is preserved end-to-end.

Check it once a site is serving — the Caddy access log should show a real client address, not a gateway or 10.0.2.x:

podman logs caddy | tail

If it is wrong and it matters (Nextcloud and Vaultwarden use client IPs for brute-force protection; Vince tracks them for analytics), there is a second route that works on stock Podman: systemd socket activation. systemd creates the listening socket and passes it in, which preserves the source address natively and at native speed. Caddy supports socket activation and sd_notify. It needs a caddy.socket unit, Notify=true on the container, and a bind fd/3 directive in the Caddyfile — which is why it is not enabled here.

Migrating from the Docker version

1. Deploy the services first, or at least their .network units.
2. Move state: /var/lib/caddy → ~/podman-data/caddy, then chown -R tanshu:tanshu ~/podman-data/caddy. Do not skip this — data/ holds the ACME account key and every issued certificate. Losing it means re-issuing ~20 certificates against Let's Encrypt rate limits.
3. Run this playbook.
4. Re-register each service (their register includes), then verify: podman exec caddy getent hosts nextcloud resolves, and curl -sSI https://<vhost> works for each site block.

Image drift to note: the running Docker container was caddy:latest while the old vars said caddy:alpine. This role uses caddy:latest to match reality.