playbooks/caddy/readme.md

# Caddy (rootless Podman + Quadlet)

Runs the Caddy reverse proxy as a **rootless Podman** container managed by
**systemd user units** (Quadlet). Replaces the previous `docker_container` role.

**Caddy is the base unit and knows nothing about its upstreams.** It does not
list any app network and does not need editing when you add a service. Each
service registers *itself* with Caddy. That is the whole design.

## Layout

```
caddy/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── podman/     # rootless prerequisites: podman, linger, port floor, Quadlet dir
    ├── caddy/      # the base caddy.container unit + handlers
    └── register/   # includable task file: a service registers itself with Caddy
```

Everything is installed under `{{ quadlet_dir }}`
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
(`~/podman-data/caddy/`).

## How services register themselves

Caddy's unit is minimal. A service joins it by writing **two files** — its own
network unit, and a Quadlet *drop-in* next to `caddy.container`:

```
~/.config/containers/systemd/
├── caddy.container                  # base unit, no app networks
├── caddy.container.d/               # ← drop-ins, one per service
│   ├── 50-gitea.conf                # [Container] Network=gitea.network
│   ├── 50-nextcloud.conf            # [Container] Network=nextcloud.network
│   └── 50-mozimo.conf
├── gitea.network
├── nextcloud.network
└── mozimo.network
```

Each service playbook does:

```yaml
- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"
```

The play must also run the `podman` role (for `podman_user`, `quadlet_dir` and
the host prerequisites) and define a `Restart Caddy` handler — see
[roles/register/tasks/main.yaml](roles/register/tasks/main.yaml) for the exact
snippet.

### Why drop-ins rather than editing caddy.container

Editing the base file directly (e.g. with `blockinfile`) **breaks on the next run
of the Caddy playbook**, because that playbook re-templates `caddy.container` and
would wipe every service's appended block. Drop-ins sit beside the base file,
survive re-templating, and are the mechanism Quadlet documents for exactly this.

They also make removal trivial — delete `50-mozimo.conf`, `daemon-reload`, and
Caddy drops that network. Nothing else changes.

### Trade-off worth knowing

A new drop-in changes the generated unit, so Caddy must be **recreated** to pick
up the network. That is a brief restart of *every* hosted site, not just the new
one. Quadlet has no live equivalent of `docker network connect`.

If that matters, connect live first and persist the drop-in in the same task:
`podman network connect <net> caddy` takes effect immediately, and the drop-in
makes it survive the next recreate. The `register` role does not do this by
default because it makes the play non-idempotent.

### `isolate`

The generated network units set `Options=isolate=true`, which blocks traffic
between separate Podman networks — an app on `nextcloud_net` cannot reach an app
that is only on `gitea_net`. Caddy and shared services still reach both, because
they are *members* of those networks.

This is set explicitly on purpose: Podman 6.0 changed the default to
`isolate=strict`, and older Podman defaulted differently. Being explicit means the
isolation you get does not depend on which Podman version is installed. On Podman
6 you may prefer `strict`, which is stronger.

## Multi-network mechanics (the parts that bite)

**A container joins several networks by repeating `Network=`.** Quadlet turns each
`Network=<name>.network` into a dependency on the generated network unit:

```ini
[Container]
Network=nextcloud.network
Network=gitea.network
```

**You cannot use the `:alias=` form with a `.network` reference.** Quadlet only
applies its `.network` mapping when the value *ends with* `.network`. So
`Network=nextcloud.network:alias=db` is passed to Podman verbatim and fails,
because no network is actually named `nextcloud.network`. Use the separate key:

```ini
NetworkAlias=db     # applies to every network the container joins
```

**Naming.** `nextcloud.network` would otherwise create a Podman network called
`systemd-nextcloud`. `NetworkName=` in the `[Network]` section overrides that, so
we keep the Docker-era `nextcloud_net` names. Note the indirection: the *file* is
`<name>.network`, the *Podman network* is `<name>_net`.

## Settings

| Variable | Purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`) — not set by hand |
| `podman_data_dir` | Where state lives (`~/podman-data`), from `ansible_facts['user_dir']` |
| `quadlet_dir` | Where `.container` / `.network` units are written |
| `caddy_dropin_dir` | Where services register their networks |
| `podman_unprivileged_port_start` | Lowers the privileged-port floor so rootless Caddy can publish 80/443 |
| `caddy_image` | Fully qualified on purpose |
| `caddy_ports` | Host port mappings, including 443/udp for HTTP/3 |
| `caddy_name` | Container / unit name, and the prefix of the drop-in directory |

## Running

The play targets `emily` directly (`hosts: emily`), so:

```command
ansible-playbook -u tanshu caddy/playbook.yml
```

The `podman` role installs the `podman` package, enables lingering, lowers the
privileged-port floor to 80, and creates the Quadlet directory. Target is Ubuntu
26.04 or later (Podman 5.7).

Only `podman` is named explicitly. `aardvark-dns`, `nftables`, `passt` and
`uidmap` arrive as **Recommends** of podman/netavark, which a normal
`apt install` installs. That is fine by default, but it does mean they would go
missing under `--no-install-recommends`. `aardvark-dns` is the one that matters
most: it is the container-name DNS every `reverse_proxy <name>:<port>` depends
on, and nothing else in the dependency graph guarantees it.

If you ever build Podman from source, this role would also pull in the distro
package — pin or skip it deliberately at that point.

Order does not matter: services can be deployed before or after Caddy. A drop-in
referencing a `.network` that does not exist yet will fail unit generation, so
deploy the network unit and the drop-in together — which the `register` role does.

## The Caddyfile

`caddyfile_path` (`~/podman-data/caddy/conf/Caddyfile`) is **not** templated here.
It is seeded once and appended to by each service's `caddy` role via
`blockinfile`, exactly as before — only the path changed from
`/var/lib/caddy/conf/Caddyfile`.

The seed matters: **Caddy exits 1 with a bare `EOF` on an empty config file**, so
an empty Caddyfile makes the service crash-loop. The role therefore writes a
minimal valid config when the file is missing or zero-length:

```
# Managed by Ansible (caddy/roles/caddy).
# Global options go in the block below. Service playbooks append their
# site blocks after it.
{
}
```

It only acts when the file is empty, so it never fights the appended site blocks.
The global options block has to come *first* in a Caddyfile, which is why it is
seeded up front rather than appended.

Service roles must also update their reload handler, which currently runs
`docker exec -w /etc/caddy caddy caddy reload`:

```yaml
- name: Reload Caddy configuration
  ansible.builtin.command: >-
    podman exec caddy caddy reload --config /etc/caddy/Caddyfile
  listen: "Reload Caddy"
```

A reference handler is in [roles/caddy/handlers/main.yaml](roles/caddy/handlers/main.yaml).
Note the container path is `/etc/caddy/Caddyfile` (the mount), not `caddyfile_path`.

## Who runs as whom

`podman_user` is **derived from the connection**, not configured:

```yaml
podman_user: "{{ ansible_facts['user_id'] }}"
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
```

The play carries no `become`, so the modules run as — and the Quadlet units belong
to — whoever Ansible connects as. Running as that user is what gives us a real PAM
session, which is what makes `systemctl --user` work, with `XDG_RUNTIME_DIR` set by
`pam_systemd`. No `become_user`, no environment block.

```
ansible-playbook -u tanshu caddy/playbook.yml
```

The home directory comes from `ansible_facts['user_dir']`, so this is correct even for a
user whose home is not `/home/<name>`. This is why `gather_facts: true` is set
explicitly on the play — the two facts above are the only configuration.

Because the user is now implied rather than declared, the `podman` role **fails
loudly if it resolves to `root`**. That is the one case where deriving it would
silently produce something wrong: rootless units written into `/root`, and a
`systemctl --user` operating on root's user manager. The usual cause is
re-enabling `become` at play level, so the message says so.

Using a *different* user is deliberately not supported — it would require
`become_user` for the user-facing tasks, which is the thing this design removes.
Only genuinely root-only steps elevate, each with its own `become: true`:

| Task | Why root |
|---|---|
| `apt` install of podman + runtime pieces | package management |
| `loginctl enable-linger` | modifies system-wide linger state |
| `/etc/sysctl.d/99-podman.conf` + `sysctl -w …` | kernel setting |

Everything else — directories, Quadlet files, `systemctl --user`, `podman exec` —
runs unprivileged as the connection user.

## Operational notes

Caddy is a **systemd user unit**:

```
systemctl --user status caddy
systemctl --user restart caddy
journalctl --user -u caddy -f
podman logs caddy
podman exec caddy caddy reload --config /etc/caddy/Caddyfile
podman inspect caddy --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
```

From a plain root shell, user units need the runtime dir — `sudo -u` does not
give you a PAM session, which is exactly why the playbook avoids `become`:

```
sudo -u tanshu XDG_RUNTIME_DIR=/run/user/$(id -u tanshu) systemctl --user status caddy
```

Two prerequisites the `podman` role sets up, both easy to forget on a fresh
instance:

- **`loginctl enable-linger tanshu`** — otherwise user units stop when the last
  session ends and never start at boot.
- **`net.ipv4.ip_unprivileged_port_start = 80`** — rootless containers cannot bind
  host ports below 1024 without it. The role writes
  `/etc/sysctl.d/99-podman.conf` **and** applies it with `sysctl -w` in the same
  run. It deliberately does not use a handler: handlers only fire at the end of
  the play, by which point Caddy has already tried and failed to bind 80/443.

`systemctl --user enable caddy` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

### Debugging a unit that will not generate

```
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
systemd-analyze --user --generators=true verify caddy.service
```

The most common cause is a `Network=` naming a `.network` file that does not exist.

## Client IPs (verify before relying on them)

Whether the original client address survives the host-to-container port forward
depends on the forwarding mechanism:

- `rootlessport` — a userspace proxy that rewrites the source address.
- `pasta` — kernel-level forwarding that preserves it.

The upstream switch, `rootless_port_forwarder = "pasta"` in `containers.conf`, is
Podman 6 only, so it is unavailable on Ubuntu 26.04's 5.7.

**Not yet verified on emily.** Podman 5.7 uses pasta as the rootless network
backend, and pasta is what binds the published ports — visible in the journal as
`pasta[...]: Listen failed for HOST UDP port */443` when the port floor was still
1024. That shows pasta is in the path, but does not by itself prove the address is
preserved end-to-end.

Check it once a site is serving — the Caddy access log should show a real client
address, not a gateway or `10.0.2.x`:

```
podman logs caddy | tail
```

If it is wrong and it matters (Nextcloud and Vaultwarden use client IPs for
brute-force protection; Vince tracks them for analytics), there is a second route
that works on stock Podman: **systemd socket activation**. systemd creates the
listening socket and passes it in, which preserves the source address natively and
at native speed. Caddy supports socket activation and `sd_notify`. It needs a
`caddy.socket` unit, `Notify=true` on the container, and a `bind fd/3` directive in
the Caddyfile — which is why it is not enabled here.

## Migrating from the Docker version

1. Deploy the services first, or at least their `.network` units.
2. Move state: `/var/lib/caddy` → `~/podman-data/caddy`, then
   `chown -R tanshu:tanshu ~/podman-data/caddy`.
   **Do not skip this** — `data/` holds the ACME account key and every issued
   certificate. Losing it means re-issuing ~20 certificates against Let's Encrypt
   rate limits.
3. Run this playbook.
4. Re-register each service (their `register` includes), then verify:
   `podman exec caddy getent hosts nextcloud` resolves, and
   `curl -sSI https://<vhost>` works for each site block.

Image drift to note: the running Docker container was `caddy:latest` while the old
vars said `caddy:alpine`. This role uses `caddy:latest` to match reality.