playbooks/nextcloud/readme.md

# Nextcloud (rootless Podman + Quadlet)

Runs Nextcloud as a **rootless Podman** container managed by a **systemd user
unit** (Quadlet), behind the shared [rootless Caddy](../caddy/readme.md), with its
database in the shared [rootless Postgres](../postgres/readme.md) and its cache in
the shared [rootless Valkey](../valkey/readme.md).

Standalone and idempotent, and host-agnostic apart from `hosts:` in
[playbook.yml](playbook.yml) and `host` in [vars/default.yml](vars/default.yml):

- **First run** creates the webroot, the environment file, the unit and the
  background-job timer, starts the container, and leaves a fresh instance ready
  for its [setup wizard](#first-run) — or installs it unattended, if you set the
  two optional credentials first.
- **Every run after that** converges the same things, and Quadlet recreates the
  container when the unit, the environment or the image changed. There is no
  update step to remember and nothing to copy in by hand.

Nextcloud is the one service here whose webroot is *also* its code directory: the
image's `VOLUME`, `config/config.php`, `data/`, `custom_apps/` and every PHP file
live under one mount. That single fact drives the ownership story and the backup
guidance below — see [What is state, and what is not](#what-is-state-and-what-is-not).

## Deploy order

Three shared services have to exist on the host first, each of them standalone:

1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns the
   Caddyfile this site appends to, supplies the `caddy.service` handlers, and is
   the one play that applies the rootless-Podman prerequisites.
2. [postgres/playbook.yml](../postgres/playbook.yml) — the cluster. **This play
   attaches to it and creates its own `nextcloud` database in it**, so the cluster
   can be brand new and empty.
3. [valkey/playbook.yml](../valkey/playbook.yml) — the cache. It has to exist
   before this playbook runs: the attach include only writes drop-ins next to
   `valkey.container`, so a `Restart Valkey` handler against an undeployed unit
   fails.

Then this playbook. `config.php` connects to the hosts `db` and `redis`; until it
has attached `nextcloud_net` to Postgres and to Valkey, neither name resolves on
that network, so running this playbook is what makes the app reachable.

## Layout

```
nextcloud/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/     # the database, and this app's network on the cluster
    ├── valkey/       # this app's network on the cache
    ├── nextcloud/    # webroot, env file, Quadlet unit, session config,
    │                 # background-job timer, start + verify
    └── caddy/        # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/nextcloud.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/nextcloud.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-nextcloud.conf` | attaches Caddy to that network |
| `~/.config/containers/systemd/postgres.container.d/50-nextcloud.conf` | attaches Postgres to it |
| `~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf` | the shared `db` alias |
| `~/.config/containers/systemd/valkey.container.d/50-nextcloud.conf` | attaches Valkey to it |
| `~/.config/containers/systemd/valkey.container.d/10-network-aliases.conf` | the shared `redis` alias |
| `~/.config/systemd/user/nextcloud-cron.{service,timer}` | the background-job timer (not Quadlet) |
| `~/podman-data/nextcloud/` | the whole `/var/www/html`: code, `config/config.php`, `data/`, `custom_apps/` |
| `~/podman-data/nextcloud/nextcloud.env` | `POSTGRES_*` + `REDIS_HOST`, mode `0600`, read by systemd |
| `~/podman-data/nextcloud/php-conf.d/redis-session.ini` | PHP's session handler, mounted into the image |
| `~/podman-data/nextcloud/nextcloud-init-sync.lock` | the entrypoint's init lock, pre-created so it stays host-owned |

## What is state, and what is not

The webroot is about 1.2 GB on a working instance, and most of it is not worth
backing up:

| Path | Keep? | Why |
|---|---|---|
| `data/` | **required** | user files, `appdata_*`, versions, thumbnails, the log |
| `config/` | **required** | `config.php` — without it the instance is not restorable, and neither is the encryption material |
| `custom_apps/` | **required** | apps installed from the app store |
| `themes/` | if themed | only exists when a theme was installed |
| `version.php`, `core/`, `lib/`, `apps/`, `3rdparty/`, `dist/` | regenerable | shipped by the image, rsynced into the webroot on start |

The database is separate and is not in this directory at all. A backup that
matters is `config/`, `data/`, `custom_apps/` **and** a `pg_dump` of the
`nextcloud` database taken together — files without the database restore to an
instance that does not know about its own contents, and the reverse is just as
useless. Nextcloud's own guidance is `occ maintenance:mode --on` around the pair;
the play waits for maintenance mode to clear on the next run, so a forgotten
`--on` is reported rather than silently 503ing.

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Nextcloud's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`nextcloud.container` references `nextcloud.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet. Both attaches below depend on it
for the same reason.

`network_name` is `nextcloud`, so the *file* is `nextcloud.network` and the
*Podman network* is `nextcloud_net`.

The site block is appended to the shared Caddyfile by
[roles/caddy](roles/caddy/tasks/main.yaml) with `blockinfile`, exactly as the
other services do — plus the HSTS header Nextcloud asks for. There is no
request-body limit to set: Caddy streams request bodies and imposes none by
default.

## How it attaches to Postgres

Postgres knows nothing about its clients, and this playbook uses the shared
include in [postgres/roles/attach](../postgres/roles/attach/tasks/main.yaml)
rather than writing the drop-ins itself. That writes two files, both mergeable
into `postgres.container` by Quadlet:

| File | Content |
|---|---|
| `postgres.container.d/50-nextcloud.conf` | `Network=nextcloud.network` |
| `postgres.container.d/10-network-aliases.conf` | `NetworkAlias=db` |

`db` is the alias `config.php` connects to; the `postgres` container name resolves
too, and the play checks both.

[roles/postgres](roles/postgres/tasks/main.yaml) wraps that include for three
reasons:

- it defines the `Restart Postgres` handler the include notifies;
- it **flushes handlers** immediately afterwards. On a first attach the drop-in is
  new, so Postgres has to be *recreated* to join `nextcloud_net` — and that restart
  is a handler, which by default runs at the end of the play, after the `nextcloud`
  role has already started the app. The app would then come up before `db`
  resolves, fail its readiness check, and fail the play on a machine where nothing
  is wrong. The flush also fires the `Restart Caddy` queued by the register
  pre_task, which is what attaches Caddy to the network. On later runs nothing is
  pending and it is a no-op;
- it **creates the `nextcloud` database** if the cluster does not have it yet.
  Postgres is a shared unit that ships one default database, and nothing else in
  this repo knows what this app's database is called, so without that task a
  first run on a fresh host would reach the setup wizard only to fail on
  "database nextcloud does not exist". It is a `createdb` guarded by a lookup, so
  it is a no-op on every run after the first.

Note the cost stated in [postgres/readme.md](../postgres/readme.md): a new drop-in
recreates Postgres, which is a few seconds of downtime for **every** attached app,
not just this one.

## How it attaches to Valkey

The same idea at the cache end, via
[valkey/roles/attach](../valkey/roles/attach/tasks/main.yaml):

| File | Content |
|---|---|
| `valkey.container.d/50-nextcloud.conf` | `Network=nextcloud.network` |
| `valkey.container.d/10-network-aliases.conf` | `NetworkAlias=redis` |

`redis` is not a name this playbook chose: the image writes the Redis host into
`config.php` and into PHP's session config from `REDIS_HOST`, which is `redis`
here. Renaming it would mean editing a live instance's configuration for nothing.

It is also **not optional**. Go through the setup wizard and Nextcloud puts
`memcache.locking` on Redis; the image's own install does too when the unattended
credentials are set. An unresolvable `redis` therefore does not merely make things
slower — uploads fail, and they fail at write time rather than at startup. That is
why `roles/valkey` flushes handlers before the app starts (the same reason as
Postgres) and why `REDIS_HOST` is in the environment file at all.

The cost of the attach is a recreate of Valkey: no keys are lost (RDB persistence
is on by default, the stop is graceful, and the new container loads the snapshot
back off the bind mount), but for the seconds it is down, PHP sessions and
`memcache.locking` are unreachable. See [valkey/readme.md](../valkey/readme.md),
"The trade-off".

## Background jobs

Nextcloud expects `cron.php` every five minutes. It cannot be a crontab entry:
root cannot `podman exec` into another user's rootless containers, and there is no
container user to own a crontab. The role installs a **systemd user** pair next to
the Quadlet units instead:

| Unit | What it does |
|---|---|
| `~/.config/systemd/user/nextcloud-cron.service` | `Type=oneshot`, `podman exec -u www-data nextcloud php /var/www/html/cron.php` |
| `~/.config/systemd/user/nextcloud-cron.timer` | `OnCalendar=*:0/5`, `Persistent=true` |

Unlike the Quadlet units these are plain units, so they really are `enable`d —
that is what brings the timer back after a reboot. `journalctl --user -u
nextcloud-cron` shows every run and every failure.

The background-job *mode* is left alone. Both this timer and AJAX polling run
jobs, and job locking keeps them from colliding; switching AJAX off is
`occ background:cron`, a deliberate change rather than something this play
imposes.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `cloud.hopsngrains.com` — the hostname Caddy serves, and the trusted domain for an unattended install |
| `container_name` | `nextcloud`, and the Quadlet unit / container name |
| `image` | `docker.io/library/nextcloud:latest` — fully qualified on purpose |
| `network_name` | `nextcloud` → `nextcloud.network` / `nextcloud_net` |
| `host_directory` / `webroot` | `~/podman-data/nextcloud` → `/var/www/html` |
| `app_uid` / `app_gid` | `33` — `www-data`, the uid the webroot is owned by |
| `container_port` | `80`, the port inside the container that Caddy proxies to |
| `env_file` | `~/podman-data/nextcloud/nextcloud.env`, mode `0600` |
| `php_conf_dir` / `php_session_ini` | PHP's session config on the host, and the path in the image it is mounted over |
| `postgres_container` / `postgres_unit` | The shared database unit this app attaches to |
| `pg_user` / `pg_password` / `pg_database` | Credentials, and the database the play creates and the app uses — `nextcloud` |
| `db_host` | `db` — the alias `config.php` connects to |
| `valkey_container` / `valkey_unit` | The shared cache unit this app attaches to |
| `redis_host` | `redis` — the alias `config.php` and PHP sessions connect to |
| `nextcloud_admin_user` / `nextcloud_admin_password` | Empty by default; set both to install unattended |
| `cron_unit` / `cron_schedule` / `user_unit_dir` | The background-job timer |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run — Nextcloud publishes nothing host-side.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

`UserNS=keep-id:uid=33,gid=33`. `33` is `www-data`, the uid the webroot is owned
by: the entrypoint rsyncs the code in with `--chown www-data:www-data` and Apache
runs its workers as that account. `keep-id` maps that one uid onto the connection
user, so the whole webroot stays owned by `tanshu` and plain `tar` and `rm -rf`
keep working on it.

Two paths the entrypoint writes fall outside that arrangement, and both are
handled explicitly rather than left to chance — each is somewhere a naive
deployment ends up with a file owned by a `165536+` subuid, or with no running
container at all:

- **`/var/www/html/nextcloud-init-sync.lock`**, which the entrypoint creates with
  a root-owned shell redirect. The role **pre-creates it** as `podman_user`: the
  redirect truncates the file rather than recreating it, so its owner never
  changes afterwards.
- **`/usr/local/etc/php/conf.d/redis-session.ini`**, in the image, which is not
  writable at all. It has its own section below.

`POSTGRES_PASSWORD` is committed in plaintext, like the other playbooks' shared
`123456`. It lives in `nextcloud.env` at mode `0600` rather than in the
world-readable unit file; a real secret store is the next step, and it is a
one-file change here plus the `pg_password` vars in the app playbooks.

## The PHP session file

The webroot is not the only thing the entrypoint writes. With `REDIS_HOST` set —
and it is, because PHP sessions should survive a redeploy — the entrypoint
generates that session configuration on **every** start:

```sh
/entrypoint.sh:121
    } > /usr/local/etc/php/conf.d/redis-session.ini
```

That directory belongs to the image's root, created by the php base image with no
`chown`:

```dockerfile
RUN set -eux; \
	mkdir -p "$PHP_INI_DIR/conf.d"; \
	...
```

so the uid this container runs as has no write bit on it, and the redirect fails:

```
cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied
```

The entrypoint exits `2` before Apache is ever reached, `Restart=always` retries
it, systemd hits its start limit, and the unit ends up **`failed`** — with the
whole thing looking like a network or database problem, because that message only
appears in the journal.

The fix is a mount: `nextcloud.container` binds
`~/podman-data/nextcloud/php-conf.d/redis-session.ini` over that exact path. A
bind-mounted file replaces the directory entry, so the directory's ownership stops
mattering and the write lands on a file owned by `podman_user`. The role also
templates the file with the five lines the entrypoint would have written, so a
start where the write is somehow refused still gets working Redis sessions instead
of a crash loop.

Apache's *own* runtime directories need none of this — the same base image chowns
`/var/run/apache2`, `/var/lock/apache2` and `/var/log/apache2` to `www-data` and
chmods them `1777`:

```dockerfile
	for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR" \
		"$APACHE_RUN_DIR/socks"; do
		rm -rvf "$dir"; mkdir -p "$dir"; \
		chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir"; \
		chmod 1777 "$dir"; \
	done
```

so the session file is the only write that was ever going to fail.

## First run

```command
ansible-playbook -u tanshu nextcloud/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

On a host with no webroot yet, this is what happens, in order: the webroot and the
PHP session directory are created; the `0600` environment file, the session config
and the Quadlet unit are written; the unit is started; anything the run queued (a
new image, a changed unit or environment) is applied *before* verification; and
then the play waits for Apache to answer `status.php` inside the container.

At that point there is a running container and **no instance yet** — the image
initialises the code, writes no `config.php` (it has no admin credentials to
install with), and serves the setup wizard. The play says so and succeeds:

```
The container is running, but no instance is installed in
/home/tanshu/podman-data/nextcloud yet. Open https://cloud.hopsngrains.com and
finish the setup wizard with the database credentials in .../nextcloud.env —
database nextcloud, user postgres, host db. ...
```

Three things to know about that wizard:

- **DNS first.** `host` must resolve to this host before Caddy can obtain a
  certificate for it, and before the wizard is reachable at all.
- **The database already exists.** [roles/postgres](roles/postgres/tasks/main.yaml)
  created it, or verified it, before the container started. Point the wizard at
  host `db`, database `nextcloud`, user `postgres`, and the password in
  `nextcloud.env`.
- **`trusted_domains`.** The wizard installs with `localhost` as the only trusted
  domain unless you tell it otherwise, so opening the finished instance on `host`
  can answer "Access through untrusted domain". Add it before or after:

  ```bash
  podman exec -u www-data nextcloud php occ config:system:set trusted_domains 1 --value=cloud.hopsngrains.com
  ```

To skip the wizard entirely, set `nextcloud_admin_user` and
`nextcloud_admin_password` in [vars/default.yml](vars/default.yml) before the first
run. The entrypoint then runs `occ maintenance:install` itself and sets
`trusted_domains` from `host` in the same pass, and the play reports the installed
version instead of the wizard note. Both variables are inert once `config.php`
exists, so they can stay set (or be removed) afterwards.

## Updates

Nothing special is required: re-run the playbook. What happens depends on what
changed.

| Changed | What the play does |
|---|---|
| The unit, the environment file or the session config | re-templates it, and Quadlet recreates the container |
| A new image behind the tag | pulls it, notices the ID changed, and recreates the container |
| Nothing | nothing — no restart, no churn |

The image is pulled on every run precisely because a Quadlet unit alone cannot see
a tag move: the unit names `nextcloud:latest`, `podman run` reuses whatever image
is local, and nothing fetches a newer one. The role reads the image ID either side
of the pull, so an unchanged tag reports no change and does not restart a running
site. A pull that fails (a host with no route to the registry) is reported and the
run continues with the local image.

**Version drift is the one irreversible step.** The entrypoint compares
`version.php` in the webroot against the image. A newer image rsyncs its code in
and runs `occ upgrade` before Apache answers — and it will only step **one major
version at a time**, so several majors behind means several runs with an
intermediate image. An image *older* than the installed version makes the
container exit with "downgrading is not supported". So: pin the tag if you want
no surprises, and take a database dump before a run that is going to upgrade.

`occ upgrade` sets maintenance mode for its duration, and the play **waits for it
to clear** rather than asserting it is off — so a long upgrade is tolerated, and an
instance left in maintenance mode (a backup that set `--on` and never finished) is
reported with the one-line command to clear it. That distinction matters because
`status.php` is exempt from maintenance mode: a site in that state looks healthy
to a probe while returning `503` to every real request.

## Verification

```bash
systemctl --user status nextcloud
journalctl --user -u nextcloud -f
podman logs nextcloud

# What the playbook checks for you:
podman exec -u www-data nextcloud php occ status          # installed, version, maintenance
podman exec nextcloud getent hosts db                     # config.php's database host
podman exec nextcloud getent hosts redis                  # config.php's cache host
podman exec nextcloud getent hosts postgres valkey        # the container names

# The background job, and that it is actually scheduled:
systemctl --user list-timers nextcloud-cron.timer
systemctl --user start nextcloud-cron.service             # run one by hand
journalctl --user -u nextcloud-cron -n 20

# The database this play created, and its size:
podman exec -e PGPASSWORD=123456 postgres psql -U postgres -l | grep nextcloud

# The site, end to end:
curl -sSI https://cloud.hopsngrains.com
```

Then open it **in a browser** and do two things a `curl` cannot: log in, and
**upload a file**. Uploading is the check that matters — it exercises the uid-33
mapping (a wrong `keep-id` shows up as a permissions error), the Redis file
locking in `config.php`, and PHP sessions, which the image configures from
`REDIS_HOST`.

After the upload, confirm on the host that the new file is owned by `tanshu`:

```bash
find ~/podman-data/nextcloud/data -newermt "-10 minutes" -type f ! -user tanshu
# empty = the write path lands on the right uid
```

Then reboot the host and confirm both the container and the timer return
unattended — without `loginctl enable-linger tanshu` neither will, and the failure
is silent.

### Debugging a unit that will not start

A unit that has hit systemd's start limit reports only a state, and `Restart=always`
means the interesting output is in the journal, not in `podman logs` (the container
may already be gone — Quadlet passes `--rm`). The play collects all of this itself
when a start fails; by hand:

```bash
systemctl --user status nextcloud.service --no-pager -l | head -25
systemctl --user cat nextcloud.service            # what systemd is running
journalctl --user -u nextcloud.service --no-pager -n 60
podman ps -a --filter name=nextcloud
podman logs --tail 40 nextcloud
```

Read it in this order:

1. **`journalctl`** — the entrypoint's own output. `cannot create ...: Permission
   denied` for a path in `/usr/local/etc/php/conf.d` is the session-file problem
   above; anything about `db`/`redis` is the network attach; `downgrading is not
   supported` is version drift.
2. **`podman ps -a`** — a container left behind under this name makes `podman run`
   refuse to start.
3. **`systemctl --user cat`** — the generated unit, if a drop-in or a `Network=`
   is in question (the usual failure is a `.network` file that does not exist).

A `failed` unit needs its start-limit counter cleared before it will try again:

```bash
systemctl --user reset-failed nextcloud.service
systemctl --user start nextcloud.service
```

## Notes and gotchas

- **`REDIS_HOST` makes the entrypoint write into the image, which it is not
  allowed to do.** `/usr/local/etc/php/conf.d` belongs to the image's root, so the
  crash is a `Permission denied` on `redis-session.ini`, the unit ends up `failed`
  via systemd's start limit, and nothing about it looks like a permissions
  problem. The unit mounts a host-owned file over that path; see
  ["The PHP session file"](#the-php-session-file).
- **`activating` is not a healthy state.** A crash-looping container spends its
  time `activating` between restarts, so the play's wait retries only while the
  state can still change on its own and otherwise hands over to a rescue that has
  the logs. A bare `is-active` check would report either state as a failure with no
  explanation.
- **A new drop-in restarts Postgres for everyone** (a few seconds without a
  database for every attached app), **registering restarts all of Caddy** (a brief
  restart of every hosted site) and **attaching restarts Valkey** (seconds without
  sessions or locks, though the keys survive the graceful stop). See the
  trade-offs in [postgres/readme.md](../postgres/readme.md),
  [caddy/readme.md](../caddy/readme.md) and [valkey/readme.md](../valkey/readme.md).
- **If the Caddyfile does not exist yet**, this playbook creates it containing only
  the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a missing
  or zero-length file. Deploy Caddy first on a fresh host.
- **Client IPs.** Nextcloud uses the client address for brute-force protection and
  for the `X-Forwarded-For` it logs, so the reverse proxy has to be trusted — see
  [caddy/readme.md](../caddy/readme.md), "Client IPs", which also covers whether
  the original address survives the rootless port forward at all.
- **The webroot holds the code.** That is the image's design, not this playbook's:
  an image bump edits `~/podman-data/nextcloud`. It is also why `apps/`, `core/`
  and `lib/` are listed as regenerable above — they are, but only from an image
  that is the same version or newer.
- **`occ` must run as `www-data`.** It refuses to run as any other user, and under
  `keep-id:uid=33` that is the same user the host sees as `tanshu`. Every
  `podman exec` in this file therefore carries `-u www-data`; the plain form works
  for `getent`, which does not care.
- **The backup scripts are not this playbook's business.** Whatever dumps the
  database and archives the webroot has to run as a **systemd user timer** next to
  the Quadlet units: root cannot `podman exec` into another user's rootless
  containers. Dump in place rather than through a helper container — `--link` does
  not exist in Podman:

  ```bash
  podman exec -e PGPASSWORD="..." postgres pg_dump -U postgres nextcloud | bzip2 > nextcloud.sql.bz2
  ```

## Not done here

- Rotating the secrets that are committed in plaintext — `pg_password` is the
  fleet's shared `123456`, and it lives in `nextcloud.env` at mode `0600` rather
  than in the world-readable unit file.
- Switching the background-job mode to `cron` (`occ background:cron`).
- Pinning the image to a digest, so that a `latest` bump is always deliberate.
- A backup timer for the webroot and the database.