tanshu/playbooks
1
0 Fork 0
Code Issues Pull Requests Releases Wiki Activity

Run Caddy as a rootless Podman Quadlet

Browse Source 
Replaces the docker_container role with Quadlet systemd user units under
~/.config/containers/systemd, and moves state from /var/lib/caddy to
~/podman-data/caddy.

Caddy is now the base unit and no longer lists its upstream networks. Each
service registers itself by writing its own <name>.network unit plus a drop-in
into caddy.container.d/, so adding or removing a service never means editing the
Caddy playbook. Drop-ins are used rather than blockinfile because this playbook
re-templates caddy.container on every run, which would wipe appended blocks.

The play runs as the connecting user instead of using become: the rootless units
belong to it, so the connection's PAM session is what makes systemctl --user
work without setting XDG_RUNTIME_DIR. podman_user and the data directory are
derived from facts, so the play fails loudly if that resolves to root.

Two things that bit us and are worth keeping:

- The privileged-port sysctl is applied inline with sysctl -w, not via a
  handler. Handlers only run at the end of the play, by which point Caddy has
  already failed to bind 80/443.
- The Caddyfile is seeded with a minimal valid config. Caddy exits 1 with a
  bare "EOF" on an empty config file, so touching it into existence made the
  service crash-loop.
This commit is contained in:
tanshu committed 2026-10-15 22:12:29 +05:30
1 parent 21f848340e
commit 060173db5e
8 files changed
+708 -79
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

caddy/readme.md
+311 -36
View File
@@ -1,46 +1,321 @@
# Docker on Ubuntu 18.04
# Caddy (rootless Podman + Quadlet)
This playbook will install Docker an Ubuntu 18.04 machine, as explained in the guide on
[How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
A number of containers will be created with the options specified in the `vars/default.yml` variable file.
Runs the Caddy reverse proxy as a **rootless Podman** container managed by
**systemd user units** (Quadlet). Replaces the previous `docker_container` role.
**Caddy is the base unit and knows nothing about its upstreams.** It does not
list any app network and does not need editing when you add a service. Each
service registers *itself* with Caddy. That is the whole design.
## Layout
```
caddy/
├── playbook.yml
├── vars/default.yml
└── roles/
├── podman/ # rootless prerequisites: podman, linger, port floor, Quadlet dir
├── caddy/ # the base caddy.container unit + handlers
└── register/ # includable task file: a service registers itself with Caddy
```
Everything is installed under `{{ quadlet_dir }}`
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
(`~/podman-data/caddy/`).
## How services register themselves
Caddy's unit is minimal. A service joins it by writing **two files** — its own
network unit, and a Quadlet *drop-in* next to `caddy.container`:
```
~/.config/containers/systemd/
├── caddy.container # base unit, no app networks
├── caddy.container.d/ # ← drop-ins, one per service
│ ├── 50-gitea.conf # [Container] Network=gitea.network
│ ├── 50-nextcloud.conf # [Container] Network=nextcloud.network
│ └── 50-mozimo.conf
├── gitea.network
├── nextcloud.network
└── mozimo.network
```
Each service playbook does:
```yaml
- name: Register this service with Caddy
ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
vars:
network_name: "{{ container_name }}"
```
The play must also run the `podman` role (for `podman_user`, `quadlet_dir` and
the host prerequisites) and define a `Restart Caddy` handler — see
[roles/register/tasks/main.yaml](roles/register/tasks/main.yaml) for the exact
snippet.
### Why drop-ins rather than editing caddy.container
Editing the base file directly (e.g. with `blockinfile`) **breaks on the next run
of the Caddy playbook**, because that playbook re-templates `caddy.container` and
would wipe every service's appended block. Drop-ins sit beside the base file,
survive re-templating, and are the mechanism Quadlet documents for exactly this.
They also make removal trivial — delete `50-mozimo.conf`, `daemon-reload`, and
Caddy drops that network. Nothing else changes.
### Trade-off worth knowing
A new drop-in changes the generated unit, so Caddy must be **recreated** to pick
up the network. That is a brief restart of *every* hosted site, not just the new
one. Quadlet has no live equivalent of `docker network connect`.
If that matters, connect live first and persist the drop-in in the same task:
`podman network connect <net> caddy` takes effect immediately, and the drop-in
makes it survive the next recreate. The `register` role does not do this by
default because it makes the play non-idempotent.
### `isolate`
The generated network units set `Options=isolate=true`, which blocks traffic
between separate Podman networks — an app on `nextcloud_net` cannot reach an app
that is only on `gitea_net`. Caddy and shared services still reach both, because
they are *members* of those networks.
This is set explicitly on purpose: Podman 6.0 changed the default to
`isolate=strict`, and older Podman defaulted differently. Being explicit means the
isolation you get does not depend on which Podman version is installed. On Podman
6 you may prefer `strict`, which is stronger.
## Multi-network mechanics (the parts that bite)
**A container joins several networks by repeating `Network=`.** Quadlet turns each
`Network=<name>.network` into a dependency on the generated network unit:
```ini
[Container]
Network=nextcloud.network
Network=gitea.network
```
**You cannot use the `:alias=` form with a `.network` reference.** Quadlet only
applies its `.network` mapping when the value *ends with* `.network`. So
`Network=nextcloud.network:alias=db` is passed to Podman verbatim and fails,
because no network is actually named `nextcloud.network`. Use the separate key:
```ini
NetworkAlias=db # applies to every network the container joins
```
**Naming.** `nextcloud.network` would otherwise create a Podman network called
`systemd-nextcloud`. `NetworkName=` in the `[Network]` section overrides that, so
we keep the Docker-era `nextcloud_net` names. Note the indirection: the *file* is
`<name>.network`, the *Podman network* is `<name>_net`.
## Settings
- `create_containers`: number of containers to create.
- `default_container_name`: default name for new containers.
- `default_container_image`: default image for new containers.
- `default_container_command`: default command to run on new containers.
| Variable | Purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`) — not set by hand |
| `podman_data_dir` | Where state lives (`~/podman-data`), from `ansible_facts['user_dir']` |
| `quadlet_dir` | Where `.container` / `.network` units are written |
| `caddy_dropin_dir` | Where services register their networks |
| `podman_unprivileged_port_start` | Lowers the privileged-port floor so rootless Caddy can publish 80/443 |
| `caddy_image` | Fully qualified on purpose |
| `caddy_ports` | Host port mappings, including 443/udp for HTTP/3 |
| `caddy_name` | Container / unit name, and the prefix of the drop-in directory |
## Running
## Running this Playbook
Quick Steps:
### 1. Obtain the playbook
```shell
git clone https://github.com/do-community/ansible-playbooks.git
cd ansible-playbooks/docker_ubuntu1804
```
### 2. Customize Options
```shell
nano vars/default.yml
```
```yml
#vars/default.yml
---
create_containers: 4
default_container_name: docker
default_container_image: ubuntu
default_container_command: sleep 1d
```
### 3. Run the Playbook
The play targets `emily` directly (`hosts: emily`), so:
```command
ansible-playbook -l [target] -i [inventory file] -u [remote user] playbook.yml
ansible-playbook -u tanshu caddy/playbook.yml
```
For more information on how to run this Ansible setup, please check this guide: [How to Use Ansible to Install and Set Up Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04).
The `podman` role installs the `podman` package, enables lingering, lowers the
privileged-port floor to 80, and creates the Quadlet directory. Target is Ubuntu
26.04 or later (Podman 5.7).
Only `podman` is named explicitly. `aardvark-dns`, `nftables`, `passt` and
`uidmap` arrive as **Recommends** of podman/netavark, which a normal
`apt install` installs. That is fine by default, but it does mean they would go
missing under `--no-install-recommends`. `aardvark-dns` is the one that matters
most: it is the container-name DNS every `reverse_proxy <name>:<port>` depends
on, and nothing else in the dependency graph guarantees it.
If you ever build Podman from source, this role would also pull in the distro
package — pin or skip it deliberately at that point.
Order does not matter: services can be deployed before or after Caddy. A drop-in
referencing a `.network` that does not exist yet will fail unit generation, so
deploy the network unit and the drop-in together — which the `register` role does.
## The Caddyfile
`caddyfile_path` (`~/podman-data/caddy/conf/Caddyfile`) is **not** templated here.
It is seeded once and appended to by each service's `caddy` role via
`blockinfile`, exactly as before — only the path changed from
`/var/lib/caddy/conf/Caddyfile`.
The seed matters: **Caddy exits 1 with a bare `EOF` on an empty config file**, so
an empty Caddyfile makes the service crash-loop. The role therefore writes a
minimal valid config when the file is missing or zero-length:
```
# Managed by Ansible (caddy/roles/caddy).
# Global options go in the block below. Service playbooks append their
# site blocks after it.
{
}
```
It only acts when the file is empty, so it never fights the appended site blocks.
The global options block has to come *first* in a Caddyfile, which is why it is
seeded up front rather than appended.
Service roles must also update their reload handler, which currently runs
`docker exec -w /etc/caddy caddy caddy reload`:
```yaml
- name: Reload Caddy configuration
ansible.builtin.command: >-
podman exec caddy caddy reload --config /etc/caddy/Caddyfile
listen: "Reload Caddy"
```
A reference handler is in [roles/caddy/handlers/main.yaml](roles/caddy/handlers/main.yaml).
Note the container path is `/etc/caddy/Caddyfile` (the mount), not `caddyfile_path`.
## Who runs as whom
`podman_user` is **derived from the connection**, not configured:
```yaml
podman_user: "{{ ansible_facts['user_id'] }}"
podman_data_dir: "{{ ansible_facts['user_dir'] }}/podman-data"
quadlet_dir: "{{ ansible_facts['user_dir'] }}/.config/containers/systemd"
```
The play carries no `become`, so the modules run as — and the Quadlet units belong
to — whoever Ansible connects as. Running as that user is what gives us a real PAM
session, which is what makes `systemctl --user` work, with `XDG_RUNTIME_DIR` set by
`pam_systemd`. No `become_user`, no environment block.
```
ansible-playbook -u tanshu caddy/playbook.yml
```
The home directory comes from `ansible_facts['user_dir']`, so this is correct even for a
user whose home is not `/home/<name>`. This is why `gather_facts: true` is set
explicitly on the play — the two facts above are the only configuration.
Because the user is now implied rather than declared, the `podman` role **fails
loudly if it resolves to `root`**. That is the one case where deriving it would
silently produce something wrong: rootless units written into `/root`, and a
`systemctl --user` operating on root's user manager. The usual cause is
re-enabling `become` at play level, so the message says so.
Using a *different* user is deliberately not supported — it would require
`become_user` for the user-facing tasks, which is the thing this design removes.
Only genuinely root-only steps elevate, each with its own `become: true`:
| Task | Why root |
|---|---|
| `apt` install of podman + runtime pieces | package management |
| `loginctl enable-linger` | modifies system-wide linger state |
| `/etc/sysctl.d/99-podman.conf` + `sysctl -w …` | kernel setting |
Everything else — directories, Quadlet files, `systemctl --user`, `podman exec` —
runs unprivileged as the connection user.
## Operational notes
Caddy is a **systemd user unit**:
```
systemctl --user status caddy
systemctl --user restart caddy
journalctl --user -u caddy -f
podman logs caddy
podman exec caddy caddy reload --config /etc/caddy/Caddyfile
podman inspect caddy --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
```
From a plain root shell, user units need the runtime dir — `sudo -u` does not
give you a PAM session, which is exactly why the playbook avoids `become`:
```
sudo -u tanshu XDG_RUNTIME_DIR=/run/user/$(id -u tanshu) systemctl --user status caddy
```
Two prerequisites the `podman` role sets up, both easy to forget on a fresh
instance:
- **`loginctl enable-linger tanshu`** — otherwise user units stop when the last
session ends and never start at boot.
- **`net.ipv4.ip_unprivileged_port_start = 80`** — rootless containers cannot bind
host ports below 1024 without it. The role writes
`/etc/sysctl.d/99-podman.conf` **and** applies it with `sysctl -w` in the same
run. It deliberately does not use a handler: handlers only fire at the end of
the play, by which point Caddy has already tried and failed to bind 80/443.
`systemctl --user enable caddy` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.
### Debugging a unit that will not generate
```
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
systemd-analyze --user --generators=true verify caddy.service
```
The most common cause is a `Network=` naming a `.network` file that does not exist.
## Client IPs (verify before relying on them)
Whether the original client address survives the host-to-container port forward
depends on the forwarding mechanism:
- `rootlessport` — a userspace proxy that rewrites the source address.
- `pasta` — kernel-level forwarding that preserves it.
The upstream switch, `rootless_port_forwarder = "pasta"` in `containers.conf`, is
Podman 6 only, so it is unavailable on Ubuntu 26.04's 5.7.
**Not yet verified on emily.** Podman 5.7 uses pasta as the rootless network
backend, and pasta is what binds the published ports — visible in the journal as
`pasta[...]: Listen failed for HOST UDP port */443` when the port floor was still
1024. That shows pasta is in the path, but does not by itself prove the address is
preserved end-to-end.
Check it once a site is serving — the Caddy access log should show a real client
address, not a gateway or `10.0.2.x`:
```
podman logs caddy | tail
```
If it is wrong and it matters (Nextcloud and Vaultwarden use client IPs for
brute-force protection; Vince tracks them for analytics), there is a second route
that works on stock Podman: **systemd socket activation**. systemd creates the
listening socket and passes it in, which preserves the source address natively and
at native speed. Caddy supports socket activation and `sd_notify`. It needs a
`caddy.socket` unit, `Notify=true` on the container, and a `bind fd/3` directive in
the Caddyfile — which is why it is not enabled here.
## Migrating from the Docker version
1. Deploy the services first, or at least their `.network` units.
2. Move state: `/var/lib/caddy` → `~/podman-data/caddy`, then
`chown -R tanshu:tanshu ~/podman-data/caddy`.
**Do not skip this** — `data/` holds the ACME account key and every issued
certificate. Losing it means re-issuing ~20 certificates against Let's Encrypt
rate limits.
3. Run this playbook.
4. Re-register each service (their `register` includes), then verify:
`podman exec caddy getent hosts nextcloud` resolves, and
`curl -sSI https://<vhost>` works for each site block.
Image drift to note: the running Docker container was `caddy:latest` while the old
vars said `caddy:alpine`. This role uses `caddy:latest` to match reality.