playbooks/nextcloud/readme.md

# Nextcloud (rootless Podman + Quadlet)

Runs Nextcloud on `emily` as a **rootless Podman** container managed by a
**systemd user unit** (Quadlet), behind the shared [rootless Caddy](../caddy/readme.md),
with its database in the shared [rootless Postgres](../postgres/readme.md) and its
cache in the shared [rootless Valkey](../valkey/readme.md). Replaces the
`docker_container` deployment on `monoco`.

The container name, network, unit, Caddyfile upstream and database are all
`nextcloud`, as they have been since the Docker-era playbook. The **host is
unchanged**: `cloud.hopsngrains.com`.

Nextcloud is the one service here whose webroot is *also* its code directory.
The Docker-era playbook bind-mounted `/var/lib/nextcloud` at `/var/www/html`, and
this one keeps that shape: the image's `VOLUME`, `config/config.php`, `data/`,
`custom_apps/` and every PHP file live under one mount. That single fact drives
both the [migration](#migrating-the-data-from-monoco) and the ownership story
below — see [What is state, and what is not](#what-is-state-and-what-is-not).

## Ordering

The three shared services are already migrated and running on `emily`, and this
playbook is the last step:

1. [caddy/playbook.yml](../caddy/playbook.yml) — the reverse proxy, which owns
   the Caddyfile this site appends to, supplies the `caddy.service` handlers, and
   is the one play that applies the rootless-Podman prerequisites.
2. [postgres/playbook.yml](../postgres/playbook.yml) — deployed, and its cluster
   brought across from monoco. **The `nextcloud` database is already restored**
   (320 MB, the largest of the 12 databases in
   [postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md)).
3. [valkey/playbook.yml](../valkey/playbook.yml) — **new**, and this is the first
   service that needs it. It has to exist before this playbook runs: the attach
   include only writes drop-ins next to `valkey.container`, so a `Restart Valkey`
   handler against an undeployed unit fails.
4. This playbook — **run it after the file copy below**, not before. The app is
   what writes to `~/podman-data/nextcloud`, and its first start is when the
   entrypoint runs its version check against whatever is in the webroot.

`config.php` connects to the hosts `db` and `redis`. Until this playbook has
attached `nextcloud_net` to Postgres and to Valkey, neither name resolves on this
network, so running this playbook is what makes the app reachable — there is no
separate step.

## Layout

```
nextcloud/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/     # attaches this app's network to Postgres (+ its handler)
    ├── valkey/       # attaches this app's network to Valkey (+ its handler)
    ├── nextcloud/    # webroot, the 0600 env file, the Quadlet unit, the cron
    │                 # timer, start + verify
    └── caddy/        # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/nextcloud.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/nextcloud.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-nextcloud.conf` | attaches Caddy to that network |
| `~/.config/containers/systemd/postgres.container.d/50-nextcloud.conf` | attaches Postgres to it |
| `~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf` | the shared `db` alias |
| `~/.config/containers/systemd/valkey.container.d/50-nextcloud.conf` | attaches Valkey to it |
| `~/.config/containers/systemd/valkey.container.d/10-network-aliases.conf` | the shared `redis` alias |
| `~/.config/systemd/user/nextcloud-cron.{service,timer}` | the background-job timer (not Quadlet) |
| `~/podman-data/nextcloud/` | the whole `/var/www/html`: code, `config/config.php`, `data/`, `custom_apps/` |
| `~/podman-data/nextcloud/nextcloud.env` | `POSTGRES_*` + `REDIS_HOST`, mode `0600`, read by systemd |
| `~/podman-data/nextcloud/php-conf.d/redis-session.ini` | PHP's session handler, mounted into the image at `/usr/local/etc/php/conf.d/` |
| `~/podman-data/nextcloud/nextcloud-init-sync.lock` | the entrypoint's init lock, pre-created so it stays host-owned |

## What is state, and what is not

Per [docs/backup-plan.md §2.1](../docs/backup-plan.md), the instance is 1.2 GB
and only a fraction of it is irreplaceable:

| Path | Keep? | Why |
|---|---|---|
| `data/` | **required** | user files, `appdata_*`, versions, thumbnails, the log |
| `config/` | **required** | `config.php` — without it the instance is not restorable, and neither is the encryption material |
| `custom_apps/` | **required** | apps installed from the app store |
| `themes/` | if themed | only exists when a theme was installed |
| `version.php`, `core/`, `lib/`, `apps/`, `3rdparty/`, `dist/` | regenerable | shipped by the image, rsynced into the webroot on start |

**The migration copies all of it anyway**, and that is deliberate. Because the
mount holds the code, the entrypoint decides what to do by reading
`/var/www/html/version.php` **from the mount**: copy the whole tree and the first
start is close to a no-op; copy only the state and `version.php` is missing, so
every start looks like a brand-new instance — no `occ upgrade`, no version
tracking — and Nextcloud would then have to be talked through a manual upgrade.

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Nextcloud's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`nextcloud.container` references `nextcloud.network`: Quadlet fails *unit
generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet. Both attaches below depend on it
for the same reason.

`network_name` is `nextcloud`, so the *file* is `nextcloud.network` and the
*Podman network* is `nextcloud_net` (`NetworkName=` in the network unit pins it,
keeping the Docker-era name).

The Docker-era `roles/network` is **deleted rather than ported**. It existed to
attach an already-created container to a network — `docker_network`'s
`connected:`/`appends:`, with the `redis` alias for the old `valkey` container —
which `podman_network` has no equivalent for
([docs/podman-compat-audit.md §3.3](../docs/podman-compat-audit.md)). Nothing has
to be connected after the fact now: the app's own unit names its `Network=`, and
Caddy, Postgres and Valkey learn about the network from drop-ins written in the
same run.

The site block is appended to the shared Caddyfile by
[roles/caddy](roles/caddy/tasks/main.yaml) with `blockinfile`, exactly as the
other services do. **Only `caddyfile_path` changed** — from
`/var/lib/caddy/conf/Caddyfile` to `~/podman-data/caddy/conf/Caddyfile` — and the
block itself is the Docker-era one, HSTS header included:

```
cloud.hopsngrains.com {
    reverse_proxy nextcloud:80

    header {
        Strict-Transport-Security "max-age=15552000; includeSubDomains; preload"
    }
}
```

The `marker` in the role is unchanged from the Docker era, so an existing block
on a reused Caddyfile is replaced in place rather than duplicated. There is no
request-body limit to carry over: Caddy streams request bodies and imposes none
by default, so the `client_max_body_size 525M` the dead nginx front end carried
has no equivalent and needs none.

## How it attaches to Postgres

Postgres knows nothing about its clients, and this playbook uses the shared
include in [postgres/roles/attach](../postgres/roles/attach/tasks/main.yaml)
rather than writing the drop-ins itself. That writes two files, both mergeable
into `postgres.container` by Quadlet:

| File | Content |
|---|---|
| `postgres.container.d/50-nextcloud.conf` | `Network=nextcloud.network` |
| `postgres.container.d/10-network-aliases.conf` | `NetworkAlias=db` |

`config.php`'s `dbhost` is `db`, which is what the Docker-era container's
`POSTGRES_HOST=db` wrote; the `postgres` container name resolves too, and the play
checks both.

[roles/postgres](roles/postgres/tasks/main.yaml) wraps that include for two
reasons:

- it defines the `Restart Postgres` handler the include notifies; and
- it **flushes handlers** immediately afterwards. On a first attach the drop-in
  is new, so Postgres has to be *recreated* to join `nextcloud_net` — and that
  restart is a handler, which by default runs at the end of the play, after the
  `nextcloud` role has already started the app. The app would then come up before
  `db` resolves, fail its own readiness check, and fail the play on a machine
  where nothing is wrong. The flush also fires the `Restart Caddy` queued by the
  register pre_task, which is what attaches Caddy to the network. On later runs
  nothing is pending and it is a no-op.

Note the cost stated in [postgres/readme.md](../postgres/readme.md): a new
drop-in recreates Postgres, which is a few seconds of downtime for **every**
attached app, not just this one.

## How it attaches to Valkey

The same idea at the cache end, via
[valkey/roles/attach](../valkey/roles/attach/tasks/main.yaml):

| File | Content |
|---|---|
| `valkey.container.d/50-nextcloud.conf` | `Network=nextcloud.network` |
| `valkey.container.d/10-network-aliases.conf` | `NetworkAlias=redis` |

`redis` is not a name this playbook chose. The image writes `config.php`'s Redis
host from `REDIS_HOST`, the Docker-era `env:` block set it to `redis`, and the
same variable rewrites PHP's session-handler config on **every** start. Renaming
it would mean editing `config.php` on a live instance for nothing.

It is also **not optional**. `config.php` carries `memcache.locking` pointed at
Redis, so an unresolvable `redis` does not merely make things slower — uploads
fail, and they fail at write time rather than at startup. That is why
`roles/valkey` flushes handlers before the app starts (same reason as Postgres),
and why `REDIS_HOST` is in the environment file at all.

The cost of the attach is a recreate of Valkey, and it is **not** the same story
as Postgres's. No keys are lost — RDB persistence is on by default, the stop is
graceful, and the new container loads the snapshot back off the bind mount — but
for the seconds it is down, Nextcloud's PHP sessions and its `memcache.locking`
are unreachable, so logins and uploads stall rather than fail cleanly. See
[valkey/readme.md](../valkey/readme.md), "The trade-off".

## Background jobs

Nextcloud expects `cron.php` to run every five minutes, and the Docker-era
playbook scheduled it in **root's crontab**:

```
*/5 * * * * docker exec -u www-data nextcloud php cron.php
```

That cannot survive the move: root cannot `podman exec` into another user's
rootless containers, and there is no container user to own a crontab
([docs/podman-migration.md §8(b)](../docs/podman-migration.md)). The role
installs a **systemd user** pair next to the Quadlet units instead:

| Unit | What it does |
|---|---|
| `~/.config/systemd/user/nextcloud-cron.service` | `Type=oneshot`, `podman exec -u www-data nextcloud php /var/www/html/cron.php` |
| `~/.config/systemd/user/nextcloud-cron.timer` | `OnCalendar=*:0/5`, `Persistent=true` |

Unlike the Quadlet units these are plain units, so they really are `enable`d —
that is what brings the timer back after a reboot. No `sudo`, no crontab, and
`journalctl --user -u nextcloud-cron` shows every run and every failure, which the
old crontab never did.

The background-job *mode* is left alone: `config.php` came from monoco with
whatever it had, and this playbook does not switch it to `cron`. Both the timer
and AJAX polling run jobs; job locking is what keeps them from colliding. If you
want AJAX disabled, that is `occ background:cron`, a deliberate change rather
than a migration step.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `cloud.hopsngrains.com` — the site block Caddy serves |
| `container_name` | `nextcloud`, and the Quadlet unit / container name |
| `image` | `docker.io/library/nextcloud:latest` — fully qualified on purpose |
| `network_name` | `nextcloud` → `nextcloud.network` / `nextcloud_net` |
| `host_directory` / `webroot` | `~/podman-data/nextcloud` → `/var/www/html` |
| `app_uid` / `app_gid` | `33` — `www-data`, the uid the server writes as |
| `container_port` | `80`, the port inside the container that Caddy proxies to |
| `env_file` | `~/podman-data/nextcloud/nextcloud.env`, mode `0600` |
| `postgres_container` / `postgres_unit` | The shared database unit this app attaches to |
| `pg_user` / `pg_password` / `pg_database` | Credentials, and the database name — `nextcloud` |
| `db_host` | `db` — the alias `config.php` connects to |
| `valkey_container` / `valkey_unit` | The shared cache unit this app attaches to |
| `redis_host` | `redis` — the alias `config.php` and PHP sessions connect to |
| `cron_unit` / `cron_schedule` / `user_unit_dir` | The background-job timer |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run — Nextcloud publishes nothing host-side.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

`UserNS=keep-id:uid=33,gid=33` — `33` is the uid the server writes as. Whatever
the entrypoint's *own* starting uid under this mapping, the writes that matter land
on `podman_user`: the code is rsynced with `--chown www-data:www-data`, and Apache's
workers run as `www-data`
([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `33` is also
what the data is owned by on monoco, which is what makes the copy below need no
`chown` pass.

Two paths the entrypoint writes fall outside that arrangement, and both are handled
explicitly rather than left to chance — each one is somewhere a naive deployment
ends up with a file owned by a `165536+` subuid, or with no deployment at all:

- **`/var/www/html/nextcloud-init-sync.lock`**, which the entrypoint creates with a
  root-owned shell redirect. The role **pre-creates it** as `podman_user`: the
  redirect truncates the file rather than recreating it, so its owner never changes
  afterwards.
- **`/usr/local/etc/php/conf.d/redis-session.ini`**, in the image, which is not
  writable at all — it is covered in its own section below.

### The PHP session file

The webroot is not the only thing the entrypoint writes. With `REDIS_HOST` set —
which it is, because `config.php` puts `memcache.locking` in Redis and PHP
sessions should survive a redeploy — the entrypoint generates that session
configuration on **every** start:

```sh
/entrypoint.sh:121
    } > /usr/local/etc/php/conf.d/redis-session.ini
```

That directory belongs to the image's root, created by the php base image with no
`chown`:

```dockerfile
RUN set -eux; \
	mkdir -p "$PHP_INI_DIR/conf.d"; \
	...
```

so the uid this container runs as has no write bit on it, and the redirect fails:

```
cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied
```

The entrypoint exits `2` before Apache is ever reached, `Restart=always` retries
it, systemd hits its start limit, and the unit ends up **`failed`** — with the
whole thing looking like a network or database problem, because that message only
appears in the journal.

The fix is a mount: `nextcloud.container` binds
`~/podman-data/nextcloud/php-conf.d/redis-session.ini` over that exact path. A
bind-mounted file replaces the directory entry, so the directory's ownership stops
mattering and the write lands on a file owned by `podman_user`. The role also
templates the file with the five lines the entrypoint would have written, so a
start where the write is somehow refused still gets working Redis sessions instead
of a crash loop.

Apache's *own* runtime directories need none of this — the same base image chowns
`/var/run/apache2`, `/var/lock/apache2` and `/var/log/apache2` to `www-data` and
chmods them `1777`:

```dockerfile
	for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR" \
		"$APACHE_RUN_DIR/socks"; do
		rm -rvf "$dir"; mkdir -p "$dir"; \
		chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir"; \
		chmod 1777 "$dir"; \
	done
```

`POSTGRES_PASSWORD` is the fleet's shared `123456`, which matches
`pg_password` in [postgres/vars/default.yml](../postgres/vars/default.yml) and
what `config.php` already carries — a migrated app connects unchanged. It lives
in `nextcloud.env` at mode `0600` rather than in the world-readable unit file.
Both are committed in plaintext as before; flagged as a follow-up, not changed
here.

## Running

```command
ansible-playbook -u tanshu nextcloud/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

This play applies **only** this app's units and data. The rootless-Podman
prerequisites — installing podman, `loginctl enable-linger`, the low-port sysctl
and the Quadlet directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.

`systemctl --user enable nextcloud` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself. The
cron **timer** is a plain unit and is enabled explicitly, as described above.

## Migrating the data from monoco

The database and the files have to correspond, so this is a freeze, not a
best-effort copy. Run it from the machine that holds the `monoco` and `emily` SSH
aliases, as in [postgres/migrate-from-monoco.md](../postgres/migrate-from-monoco.md).

Two things are already done: the `nextcloud` database is restored on `emily`, and
the Caddyfile and certificates live there too. What is left is
`/var/lib/nextcloud`.

### 0. Look first

```bash
# What is there, how big, and whether it is readable without sudo. Nextcloud's
# data/ is normally 0770 www-data:www-data, so this says "no" — the sudo form
# below is the expected one, not the exception.
ssh monoco 'du -sh /var/lib/nextcloud; ls -ln /var/lib/nextcloud'
ssh monoco 'find /var/lib/nextcloud -type f ! -readable | head'   # empty = no sudo needed

# The version on monoco versus the image emily will run. The entrypoint compares
# the two on first start: a NEWER image rsyncs its code in and runs `occ upgrade`
# (one major version at a time, so a large gap needs stepping stones), while an
# OLDER image refuses to start at all — "downgrading is not supported".
ssh monoco 'docker exec nextcloud php -r "require \"/var/www/html/version.php\"; echo implode(\".\", \$OC_Version), PHP_EOL;"'
ssh emily  'podman run --rm --entrypoint php docker.io/library/nextcloud:latest \
              -r "require \"/usr/src/nextcloud/version.php\"; echo implode(\".\", \$OC_Version), PHP_EOL;"'
```

### 1. Freeze monoco

```bash
# Stop the app first: a stopped container cannot write to data/ while it is being
# copied, and no maintenance-mode dance is needed. Leave it stopped — its database
# is on emily, and two live copies would diverge.
ssh monoco 'docker stop nextcloud'
```

### 2. Make the database correspond (only if it moved since the cluster move)

The cluster was dumped on **2026-10-15**. If monoco's Nextcloud has served writes
since then, the files you are about to copy are newer than the database on
`emily`, and restoring the files alone would leave the instance inconsistent.
Re-dump the one database and reload it — it is 320 MB, so this is cheap.

If the app is already running on `emily` from an earlier attempt, stop it first:
`DROP DATABASE` needs the connections gone, and a running Nextcloud holds several.

```bash
ssh emily 'systemctl --user stop nextcloud.service'   # only if it is already deployed

```bash
# Dump on monoco (Postgres is a separate container and is still running).
ssh monoco 'mkdir -p ~/pg-migrate && cd ~/pg-migrate
  docker exec -e PGPASSWORD=123456 postgres pg_dump -U postgres -Fc nextcloud \
    | zstd -T0 -6 -o nextcloud-$(date +%Y%m%d-%H%M).dump.zst
  ls -lh nextcloud-*.dump.zst'

# Transfer, then check the checksum on both sides before trusting it.
ssh monoco 'cat ~/pg-migrate/nextcloud-*.dump.zst' \
  | ssh emily 'cat > ~/pg-migrate/nextcloud.dump.zst'
a=$(ssh monoco 'sha256sum ~/pg-migrate/nextcloud-*.dump.zst | cut -d" " -f1')
b=$(ssh emily  'sha256sum ~/pg-migrate/nextcloud.dump.zst | cut -d" " -f1')
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
  || { echo "MISMATCH — do not restore: $a vs $b"; false; }

# Reload it on emily. This DROPS the existing database — it is the one the
# cluster move restored, and the frozen dump replaces it.
ssh emily 'cd ~/pg-migrate
  podman exec -e PGPASSWORD=123456 postgres \
    psql -U postgres -d postgres -v ON_ERROR_STOP=1 \
      -c "DROP DATABASE IF EXISTS nextcloud" \
      -c "CREATE DATABASE nextcloud OWNER postgres"
  zstd -dc nextcloud.dump.zst | podman exec -i -e PGPASSWORD=123456 postgres \
    pg_restore -U postgres -d nextcloud --no-owner --exit-on-error
  podman exec -e PGPASSWORD=123456 postgres \
    psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'
```

If monoco's container simply sat idle after the cluster move, skip this and go
straight to the copy — but the file counts at the end of step 3 are what tell you
whether the two halves agree.

### 3. Copy the files

```bash
# 1. Create the destination: the mount target of the image's /var/www/html.
ssh emily 'mkdir -p ~/podman-data/nextcloud'

# 2. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command.
#
#    The WHOLE tree is copied — code as well as state — because /var/www/html is
#    both. See "What is state, and what is not" above.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
#
#    `--no-same-owner` is explicit and important: the source is 33:33 (and partly
#    root), and under keep-id:uid=33 the *container's* 33 is tanshu on the host.
#    Extracting as tanshu is therefore already the correct ownership, and
#    restoring the source's numeric owners would be wrong.
set -o pipefail
ssh monoco 'sudo tar -C /var/lib/nextcloud -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/nextcloud -xf -'

#    Without sudo, when step 0 proved everything is readable:
# ssh monoco 'tar -C /var/lib/nextcloud -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/nextcloud -xf -'

# 3. Verify by content, over the whole tree. Read as root on the source side
#    because data/ is 0770.
ssh monoco 'cd /var/lib/nextcloud && sudo bash -c "find . -type f -print0 | sort -z | xargs -0 -r sha256sum" | sha256sum'
ssh emily  'cd ~/podman-data/nextcloud && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 4. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/nextcloud ! -user tanshu'    # empty = correct

# 5. A cheap sanity check that the two halves agree — compare with the count from
#    step 2, and with monoco's own tree.
ssh emily  'find ~/podman-data/nextcloud/data -type f | wc -l'
ssh monoco 'sudo find /var/lib/nextcloud/data -type f | wc -l'
```

Ownership needs no `chown` pass: `keep-id:uid=33` maps the container's `www-data`
onto the connecting user, so files extracted as `tanshu` are already what the
container expects to own — and what plain `tar`, `rsync` and `rm -rf` keep working
with.

If the link is slow or drops, do it in two hops instead: resumable, and it shows
progress.

```bash
rsync -aP --rsync-path='sudo rsync' monoco:/var/lib/nextcloud/ ~/nextcloud-copy/
rsync -aP ~/nextcloud-copy/ emily:~/podman-data/nextcloud/
```

### If you ran the playbook before copying

Running this play against an empty `~/podman-data/nextcloud` does not fail — it
creates the webroot, writes the environment file, starts the image, and the
entrypoint initialises a **fresh, uninstalled** instance there: the image's own
code rsynced in, `data/.ocdata` created, and no `config.php`, because there is no
`NEXTCLOUD_ADMIN_USER` to install with.

That is not a usable instance, and worse, copying monoco's tree on top of it
**merges two versions of the code** — the drift this page spends the rest of its
time warning about, produced silently by a play that reported success. The role
now warns when it finds a webroot with no `config/config.php`, but the recovery is
to start from a clean webroot:

```bash
ssh emily 'systemctl --user stop nextcloud.service
           rm -rf ~/podman-data/nextcloud'
```

`rm -rf` needs no `podman unshare`: the role creates the webroot as the connection
user, and under `keep-id:uid=33` everything the container writes lands on that same
user. Then take the copy in step 3 and come back to step 4.

### 4. Deploy

```bash
cd prepare-server && ansible-playbook -u tanshu ../nextcloud/playbook.yml
```

Run it **after** the copy: the first start is when the entrypoint compares the
copied `version.php` with the image and, if the image is newer, runs
`occ upgrade`. Starting before the copy would initialise an empty instance and
then be overwritten by it.

Rolling back means rolling back all three halves — `ssh monoco 'docker start nextcloud'`
is only meaningful while monoco's own Postgres 17 cluster is intact, and it would
then be writing to a database emily's copy has already moved past.

## Verification

```bash
systemctl --user status nextcloud
journalctl --user -u nextcloud -f
podman logs nextcloud

# What the playbook checks for you:
podman exec -u www-data nextcloud php occ status          # installed, version, maintenance
podman exec nextcloud getent hosts db                     # config.php's database host
podman exec nextcloud getent hosts redis                  # config.php's cache host
podman exec nextcloud getent hosts postgres valkey        # the container names

# The background job, and that it is actually scheduled:
systemctl --user list-timers nextcloud-cron.timer
systemctl --user start nextcloud-cron.service             # run one by hand
journalctl --user -u nextcloud-cron -n 20

# The data really is the copied tree:
ssh emily 'ls -ln ~/podman-data/nextcloud | head'
ssh emily 'test -f ~/podman-data/nextcloud/config/config.php && echo "config.php present"'

# The database really is the migrated one — run the same query on both hosts.
ssh emily  'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'
ssh monoco 'docker exec -e PGPASSWORD=123456 postgres psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'

# The site, end to end:
curl -sSI https://cloud.hopsngrains.com
```

Then open it **in a browser** and do two things a `curl` cannot: log in, and
**upload a file**. Uploading is the check that matters here — it exercises the
uid-33 mapping (a wrong `keep-id` shows up as a permissions error), the Redis
file locking in `config.php`, and PHP sessions, which the image configures from
`REDIS_HOST`.

After the upload, confirm on the host that the new file is owned by `tanshu`:

```bash
ssh emily 'find ~/podman-data/nextcloud/data -newermt "-10 minutes" -type f ! -user tanshu'
# empty = the write path lands on the right uid
```

Then reboot the host and confirm both the container and the timer return
unattended — without `loginctl enable-linger tanshu` neither will, and the failure
is silent.

Finally, if `config.php` predates the proxy (it should not — monoco already served
this site through Caddy, but check), Nextcloud needs the proxy trusted before it
will believe `X-Forwarded-For`:

```bash
podman exec -u www-data nextcloud php occ config:system:get trusted_proxies
podman exec -u www-data nextcloud php occ config:system:get overwrite.cli.url
```

An empty `trusted_proxies` means every client looks like the proxy; see
[caddy/readme.md](../caddy/readme.md), "Client IPs", which also explains why the
real address may not survive the rootless port forward at all.

### Debugging a unit that will not start

A unit that has hit systemd's start limit reports only a state, and `Restart=always`
means the interesting output is in the journal, not in `podman logs` (the container
may already be gone — Quadlet passes `--rm`). The play collects all of this itself
when a start fails; by hand:

```bash
ssh emily 'systemctl --user status nextcloud.service --no-pager -l | head -25
           systemctl --user cat nextcloud.service            # what systemd is running
           journalctl --user -u nextcloud.service --no-pager -n 60
           podman ps -a --filter name=nextcloud
           podman logs --tail 40 nextcloud'
```

Read it in this order:

1. **`journalctl`** — the entrypoint's own output. `cannot create ...: Permission
   denied` for a path in `/usr/local/etc/php/conf.d` is the session-file problem
   above; anything about `db`/`redis` is the network attach; `downgrading is not
   supported` is version drift.
2. **`podman ps -a`** — a container left behind under this name makes `podman run`
   refuse to start. Quadlet passes `--replace`, so this is rare, and it means an
   interrupted run.
3. **`systemctl --user cat`** — the generated unit, if a drop-in or a `Network=`
   is in question (the usual failure is a `.network` file that does not exist).

The first real run of this playbook failed at a bare `is-active` check with no
diagnostics at all, which is why the wait now lives inside the start block whose
rescue collects the above.

## Notes and gotchas

- **`REDIS_HOST` makes the entrypoint write into the image, which it is not
  allowed to do.** `/usr/local/etc/php/conf.d` belongs to the image's root, so the
  crash is a `Permission denied` on `redis-session.ini`, the unit ends up `failed`
  via systemd's start limit, and nothing about it looks like a permissions
  problem. The unit mounts a host-owned file over that path; see
  ["The PHP session file"](#the-php-session-file) above, including why a refused
  write is no longer fatal.
- **Maintenance mode travels with `config.php`.** The backup plan recommends
  `occ maintenance:mode --on` before a consistent dump
  ([docs/backup-plan.md §3.4](../docs/backup-plan.md)); if monoco was left that
  way, the copied configuration carries `maintenance => true` and the site returns
  `503` for everything except `status.php`. The play **waits for maintenance mode
  to clear** — it is on for the whole of a first-start `occ upgrade`, so it cannot
  simply assert — and if it never clears, fails with the one-line command to turn
  it off. That is deliberate: "the app is up but every page 503s" points anywhere
  but at maintenance mode.
- **Version drift is the one irreversible step.** The copied `version.php` is what
  the entrypoint compares against the image. A newer image upgrades the instance
  on first start (and can only step one major version at a time); an older image
  makes the container exit with "downgrading is not supported". Check both before
  copying (step 0) and take a database dump before the first start.
- **`latest` floats.** The Docker-era playbook used the unqualified
  `nextcloud:latest`; this one fully qualifies the same tag, because rootless
  Podman has no TTY to answer a short-name prompt when the unit starts. Neither
  pins a digest, so an image bump is a normal `podman pull` away — and it is
  exactly the event that triggers an in-place upgrade, because the code lives in
  the mount.
- **The code lives in the data directory.** That is the image's design, not this
  playbook's, and it is why the whole tree is copied and why an image bump edits
  `~/podman-data/nextcloud`. It is also why the backup plan lists `apps/`, `core/`
  and `lib/` as regenerable: they are, but only from an image that is the same
  version or newer.
- **`occ` must run as `www-data`.** It refuses to run as any other user, and under
  `keep-id:uid=33` that is the same user the host sees as `tanshu`. Every
  `podman exec` in this file therefore carries `-u www-data`; the plain form works
  for `getent`, which does not care.
- **The init lock file.** `~/podman-data/nextcloud/nextcloud-init-sync.lock` is
  pre-created by the role so it stays owned by `tanshu`. If a run ever reports it
  as a subuid, the pre-create was skipped on a first start — delete it and re-run
  the playbook.
- **Client IPs.** Nextcloud uses the client address for brute-force protection and
  for the `X-Forwarded-For` it logs. Whether the original address survives the
  rootless port forward depends on the forwarder. See
  [caddy/readme.md](../caddy/readme.md), "Client IPs", before relying on it.
- **Registering restarts all of Caddy** (a brief restart of every hosted site),
  **attaching restarts Postgres** (a few seconds without a database for every
  attached app) and **restarts Valkey** (seconds without sessions or locks, though
  the keys themselves survive the graceful stop). See the trade-offs in
  [caddy/readme.md](../caddy/readme.md), [postgres/readme.md](../postgres/readme.md)
  and [valkey/readme.md](../valkey/readme.md).
- **If the Caddyfile does not exist yet**, this playbook creates it containing only
  the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a missing
  or zero-length file. Deploy Caddy first on a fresh host.
- **DNS.** `cloud.hopsngrains.com` must resolve to `emily` before Caddy can obtain
  a certificate for it; the old certificates live in `~/podman-data/caddy/data`
  and are reused if the hostname is unchanged.
- **The backup script still assumes Docker.** `~/bin/nextcloud-backup.sh` uses
  `--link postgres:db`, which Podman does not have, and its `sudo tar` /
  `chown $(logname)` gymnastics are unnecessary now that the data is owned by you.
  It needs the `podman exec` form from
  [postgres/readme.md](../postgres/readme.md), "Backups"; the database dump should
  also be taken with `occ maintenance:mode --on` around it, as
  [docs/backup-plan.md §3.4](../docs/backup-plan.md) says.
- **`~/bin/empty-trash.sh` targets a container named `cloud`**, which never
  existed on this host under that name — it is `nextcloud` — and it runs as root,
  which cannot `podman exec` into these containers at all.
- **Do not restart the container on monoco.** It would write to monoco's old
  cluster, which is being kept intact purely as a rollback.

## Not done here

- Rotating the secrets that are committed in plaintext — `pg_password` is the
  fleet's shared `123456`, and it lives in `nextcloud.env` at mode `0600` rather
  than in the world-readable unit file.
- Switching the background-job mode to `cron` (`occ background:cron`), which the
  Docker-era deployment also never did.
- Replacing the broken `~/bin/nextcloud-backup.sh` with a systemd user timer, and
  fixing `empty-trash.sh`.
- Turning on ongoing monitoring — the `status.php` probe in the role is where a
  timer or an external check would hook in.