playbooks/nextcloud/readme.md

Nextcloud (rootless Podman + Quadlet)

Runs Nextcloud on emily as a rootless Podman container managed by a systemd user unit (Quadlet), behind the shared rootless Caddy, with its database in the shared rootless Postgres and its cache in the shared rootless Valkey. Replaces the docker_container deployment on monoco.

The container name, network, unit, Caddyfile upstream and database are all nextcloud, as they have been since the Docker-era playbook. The host is unchanged: cloud.hopsngrains.com.

Nextcloud is the one service here whose webroot is also its code directory. The Docker-era playbook bind-mounted /var/lib/nextcloud at /var/www/html, and this one keeps that shape: the image's VOLUME, config/config.php, data/, custom_apps/ and every PHP file live under one mount. That single fact drives both the migration and the ownership story below — see What is state, and what is not.

Ordering

The three shared services are already migrated and running on emily, and this playbook is the last step:

1. caddy/playbook.yml — the reverse proxy, which owns the Caddyfile this site appends to, supplies the caddy.service handlers, and is the one play that applies the rootless-Podman prerequisites.
2. postgres/playbook.yml — deployed, and its cluster brought across from monoco. The nextcloud database is already restored (320 MB, the largest of the 12 databases in postgres/migrate-from-monoco.md).
3. valkey/playbook.yml — new, and this is the first service that needs it. It has to exist before this playbook runs: the attach include only writes drop-ins next to valkey.container, so a Restart Valkey handler against an undeployed unit fails.
4. This playbook — run it after the file copy below, not before. The app is what writes to ~/podman-data/nextcloud, and its first start is when the entrypoint runs its version check against whatever is in the webroot.

config.php connects to the hosts db and redis. Until this playbook has attached nextcloud_net to Postgres and to Valkey, neither name resolves on this network, so running this playbook is what makes the app reachable — there is no separate step.

Layout

nextcloud/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── postgres/     # attaches this app's network to Postgres (+ its handler)
    ├── valkey/       # attaches this app's network to Valkey (+ its handler)
    ├── nextcloud/    # webroot, the 0600 env file, the Quadlet unit, the cron
    │                 # timer, start + verify
    └── caddy/        # appends this site's block to the shared Caddyfile

Quadlet units and state:

On the host	What
~/.config/containers/systemd/nextcloud.container	the generated unit (Ansible-templated)
~/.config/containers/systemd/nextcloud.network	the app network, written by caddy/roles/register
~/.config/containers/systemd/caddy.container.d/50-nextcloud.conf	attaches Caddy to that network
~/.config/containers/systemd/postgres.container.d/50-nextcloud.conf	attaches Postgres to it
~/.config/containers/systemd/postgres.container.d/10-network-aliases.conf	the shared db alias
~/.config/containers/systemd/valkey.container.d/50-nextcloud.conf	attaches Valkey to it
~/.config/containers/systemd/valkey.container.d/10-network-aliases.conf	the shared redis alias
~/.config/systemd/user/nextcloud-cron.{service,timer}	the background-job timer (not Quadlet)
~/podman-data/nextcloud/	the whole /var/www/html: code, config/config.php, data/, custom_apps/
~/podman-data/nextcloud/nextcloud.env	POSTGRES_* + REDIS_HOST, mode 0600, read by systemd
~/podman-data/nextcloud/php-conf.d/redis-session.ini	PHP's session handler, mounted into the image at /usr/local/etc/php/conf.d/
~/podman-data/nextcloud/nextcloud-init-sync.lock	the entrypoint's init lock, pre-created so it stays host-owned

What is state, and what is not

Per docs/backup-plan.md §2.1, the instance is 1.2 GB and only a fraction of it is irreplaceable:

Path	Keep?	Why
data/	required	user files, appdata_*, versions, thumbnails, the log
config/	required	config.php — without it the instance is not restorable, and neither is the encryption material
custom_apps/	required	apps installed from the app store
themes/	if themed	only exists when a theme was installed
version.php, core/, lib/, apps/, 3rdparty/, dist/	regenerable	shipped by the image, rsynced into the webroot on start

The migration copies all of it anyway, and that is deliberate. Because the mount holds the code, the entrypoint decides what to do by reading /var/www/html/version.php from the mount: copy the whole tree and the first start is close to a no-op; copy only the state and version.php is missing, so every start looks like a brand-new instance — no occ upgrade, no version tracking — and Nextcloud would then have to be talked through a manual upgrade.

How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls the shared include in caddy/roles/register, which writes the network unit and the drop-in that pulls it into Caddy:

- name: Register Nextcloud's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"

It runs in pre_tasks, before the app unit is started, because nextcloud.container references nextcloud.network: Quadlet fails unit generation — so systemctl --user start reports "unit not found" — when a referenced .network file does not exist yet. Both attaches below depend on it for the same reason.

network_name is nextcloud, so the file is nextcloud.network and the Podman network is nextcloud_net (NetworkName= in the network unit pins it, keeping the Docker-era name).

The Docker-era roles/network is deleted rather than ported. It existed to attach an already-created container to a network — docker_network's connected:/appends:, with the redis alias for the old valkey container — which podman_network has no equivalent for (docs/podman-compat-audit.md §3.3). Nothing has to be connected after the fact now: the app's own unit names its Network=, and Caddy, Postgres and Valkey learn about the network from drop-ins written in the same run.

The site block is appended to the shared Caddyfile by roles/caddy with blockinfile, exactly as the other services do. Only caddyfile_path changed — from /var/lib/caddy/conf/Caddyfile to ~/podman-data/caddy/conf/Caddyfile — and the block itself is the Docker-era one, HSTS header included:

cloud.hopsngrains.com {
    reverse_proxy nextcloud:80

    header {
        Strict-Transport-Security "max-age=15552000; includeSubDomains; preload"
    }
}

The marker in the role is unchanged from the Docker era, so an existing block on a reused Caddyfile is replaced in place rather than duplicated. There is no request-body limit to carry over: Caddy streams request bodies and imposes none by default, so the client_max_body_size 525M the dead nginx front end carried has no equivalent and needs none.

How it attaches to Postgres

Postgres knows nothing about its clients, and this playbook uses the shared include in postgres/roles/attach rather than writing the drop-ins itself. That writes two files, both mergeable into postgres.container by Quadlet:

File	Content
postgres.container.d/50-nextcloud.conf	Network=nextcloud.network
postgres.container.d/10-network-aliases.conf	NetworkAlias=db

config.php's dbhost is db, which is what the Docker-era container's POSTGRES_HOST=db wrote; the postgres container name resolves too, and the play checks both.

roles/postgres wraps that include for two reasons:

• it defines the Restart Postgres handler the include notifies; and
• it flushes handlers immediately afterwards. On a first attach the drop-in is new, so Postgres has to be recreated to join nextcloud_net — and that restart is a handler, which by default runs at the end of the play, after the nextcloud role has already started the app. The app would then come up before db resolves, fail its own readiness check, and fail the play on a machine where nothing is wrong. The flush also fires the Restart Caddy queued by the register pre_task, which is what attaches Caddy to the network. On later runs nothing is pending and it is a no-op.

Note the cost stated in postgres/readme.md: a new drop-in recreates Postgres, which is a few seconds of downtime for every attached app, not just this one.

How it attaches to Valkey

The same idea at the cache end, via valkey/roles/attach:

File	Content
valkey.container.d/50-nextcloud.conf	Network=nextcloud.network
valkey.container.d/10-network-aliases.conf	NetworkAlias=redis

redis is not a name this playbook chose. The image writes config.php's Redis host from REDIS_HOST, the Docker-era env: block set it to redis, and the same variable rewrites PHP's session-handler config on every start. Renaming it would mean editing config.php on a live instance for nothing.

It is also not optional. config.php carries memcache.locking pointed at Redis, so an unresolvable redis does not merely make things slower — uploads fail, and they fail at write time rather than at startup. That is why roles/valkey flushes handlers before the app starts (same reason as Postgres), and why REDIS_HOST is in the environment file at all.

The cost of the attach is a recreate of Valkey, and it is not the same story as Postgres's. No keys are lost — RDB persistence is on by default, the stop is graceful, and the new container loads the snapshot back off the bind mount — but for the seconds it is down, Nextcloud's PHP sessions and its memcache.locking are unreachable, so logins and uploads stall rather than fail cleanly. See valkey/readme.md, "The trade-off".

Background jobs

Nextcloud expects cron.php to run every five minutes, and the Docker-era playbook scheduled it in root's crontab:

*/5 * * * * docker exec -u www-data nextcloud php cron.php

That cannot survive the move: root cannot podman exec into another user's rootless containers, and there is no container user to own a crontab (docs/podman-migration.md §8(b)). The role installs a systemd user pair next to the Quadlet units instead:

Unit	What it does
~/.config/systemd/user/nextcloud-cron.service	Type=oneshot, podman exec -u www-data nextcloud php /var/www/html/cron.php
~/.config/systemd/user/nextcloud-cron.timer	OnCalendar=*:0/5, Persistent=true

Unlike the Quadlet units these are plain units, so they really are enabled — that is what brings the timer back after a reboot. No sudo, no crontab, and journalctl --user -u nextcloud-cron shows every run and every failure, which the old crontab never did.

The background-job mode is left alone: config.php came from monoco with whatever it had, and this playbook does not switch it to cron. Both the timer and AJAX polling run jobs; job locking is what keeps them from colliding. If you want AJAX disabled, that is occ background:cron, a deliberate change rather than a migration step.

Settings

Variable	Value / purpose
podman_user	Derived from the connection (ansible_facts['user_id']), not set by hand
podman_data_dir	~/podman-data
quadlet_dir	~/.config/containers/systemd
host	cloud.hopsngrains.com — the site block Caddy serves
container_name	nextcloud, and the Quadlet unit / container name
image	docker.io/library/nextcloud:latest — fully qualified on purpose
network_name	nextcloud → nextcloud.network / nextcloud_net
host_directory / webroot	~/podman-data/nextcloud → /var/www/html
app_uid / app_gid	33 — www-data, the uid the server writes as
container_port	80, the port inside the container that Caddy proxies to
env_file	~/podman-data/nextcloud/nextcloud.env, mode 0600
postgres_container / postgres_unit	The shared database unit this app attaches to
pg_user / pg_password / pg_database	Credentials, and the database name — nextcloud
db_host	db — the alias config.php connects to
valkey_container / valkey_unit	The shared cache unit this app attaches to
redis_host	redis — the alias config.php and PHP sessions connect to
cron_unit / cron_schedule / user_unit_dir	The background-job timer
caddy_name / caddyfile_path	The shared Caddy unit and its Caddyfile

The three podman_* / quadlet_dir vars are duplicated from caddy/vars/default.yml, as in every service playbook. That is a deliberate trade: a play cannot include the Caddy vars file wholesale, because it also sets host_directory (to the Caddy data directory) and every service defines that for itself. Keep them in sync. Caddy's fourth, podman_unprivileged_port_start, belongs to the podman tasks this play does not run — Nextcloud publishes nothing host-side.

Like Caddy, podman_user is derived from the connection and the play carries no become. Because the podman tasks are not run here, there is no guard against connecting as root; the prerequisites, including that check, live in caddy/playbook.yml.

UserNS=keep-id:uid=33,gid=33 — 33 is the uid the server writes as. Whatever the entrypoint's own starting uid under this mapping, the writes that matter land on podman_user: the code is rsynced with --chown www-data:www-data, and Apache's workers run as www-data (docs/podman-compat-audit.md §3.2). 33 is also what the data is owned by on monoco, which is what makes the copy below need no chown pass.

Two paths the entrypoint writes fall outside that arrangement, and both are handled explicitly rather than left to chance — each one is somewhere a naive deployment ends up with a file owned by a 165536+ subuid, or with no deployment at all:

• /var/www/html/nextcloud-init-sync.lock, which the entrypoint creates with a root-owned shell redirect. The role pre-creates it as podman_user: the redirect truncates the file rather than recreating it, so its owner never changes afterwards.
• /usr/local/etc/php/conf.d/redis-session.ini, in the image, which is not writable at all — it is covered in its own section below.

The PHP session file

The webroot is not the only thing the entrypoint writes. With REDIS_HOST set — which it is, because config.php puts memcache.locking in Redis and PHP sessions should survive a redeploy — the entrypoint generates that session configuration on every start:

/entrypoint.sh:121
    } > /usr/local/etc/php/conf.d/redis-session.ini

That directory belongs to the image's root, created by the php base image with no chown:

RUN set -eux; \
	mkdir -p "$PHP_INI_DIR/conf.d"; \
	...

so the uid this container runs as has no write bit on it, and the redirect fails:

cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied

The entrypoint exits 2 before Apache is ever reached, Restart=always retries it, systemd hits its start limit, and the unit ends up failed — with the whole thing looking like a network or database problem, because that message only appears in the journal.

The fix is a mount: nextcloud.container binds ~/podman-data/nextcloud/php-conf.d/redis-session.ini over that exact path. A bind-mounted file replaces the directory entry, so the directory's ownership stops mattering and the write lands on a file owned by podman_user. The role also templates the file with the five lines the entrypoint would have written, so a start where the write is somehow refused still gets working Redis sessions instead of a crash loop.

Apache's own runtime directories need none of this — the same base image chowns /var/run/apache2, /var/lock/apache2 and /var/log/apache2 to www-data and chmods them 1777:

	for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR" \
		"$APACHE_RUN_DIR/socks"; do
		rm -rvf "$dir"; mkdir -p "$dir"; \
		chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir"; \
		chmod 1777 "$dir"; \
	done

POSTGRES_PASSWORD is the fleet's shared 123456, which matches pg_password in postgres/vars/default.yml and what config.php already carries — a migrated app connects unchanged. It lives in nextcloud.env at mode 0600 rather than in the world-readable unit file. Both are committed in plaintext as before; flagged as a follow-up, not changed here.

Running

ansible-playbook -u tanshu nextcloud/playbook.yml

The play targets emily directly, so the inventory must contain it — the one in prepare-server/ does, and its ansible.cfg names it.

This play applies only this app's units and data. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them.

systemctl --user enable nextcloud is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself. The cron timer is a plain unit and is enabled explicitly, as described above.

Migrating the data from monoco

The database and the files have to correspond, so this is a freeze, not a best-effort copy. Run it from the machine that holds the monoco and emily SSH aliases, as in postgres/migrate-from-monoco.md.

Two things are already done: the nextcloud database is restored on emily, and the Caddyfile and certificates live there too. What is left is /var/lib/nextcloud.

0. Look first

# What is there, how big, and whether it is readable without sudo. Nextcloud's
# data/ is normally 0770 www-data:www-data, so this says "no" — the sudo form
# below is the expected one, not the exception.
ssh monoco 'du -sh /var/lib/nextcloud; ls -ln /var/lib/nextcloud'
ssh monoco 'find /var/lib/nextcloud -type f ! -readable | head'   # empty = no sudo needed

# The version on monoco versus the image emily will run. The entrypoint compares
# the two on first start: a NEWER image rsyncs its code in and runs `occ upgrade`
# (one major version at a time, so a large gap needs stepping stones), while an
# OLDER image refuses to start at all — "downgrading is not supported".
ssh monoco 'docker exec nextcloud php -r "require \"/var/www/html/version.php\"; echo implode(\".\", \$OC_Version), PHP_EOL;"'
ssh emily  'podman run --rm --entrypoint php docker.io/library/nextcloud:latest \
              -r "require \"/usr/src/nextcloud/version.php\"; echo implode(\".\", \$OC_Version), PHP_EOL;"'

1. Freeze monoco

# Stop the app first: a stopped container cannot write to data/ while it is being
# copied, and no maintenance-mode dance is needed. Leave it stopped — its database
# is on emily, and two live copies would diverge.
ssh monoco 'docker stop nextcloud'

2. Make the database correspond (only if it moved since the cluster move)

The cluster was dumped on 2026-10-15. If monoco's Nextcloud has served writes since then, the files you are about to copy are newer than the database on emily, and restoring the files alone would leave the instance inconsistent. Re-dump the one database and reload it — it is 320 MB, so this is cheap.

If the app is already running on emily from an earlier attempt, stop it first: DROP DATABASE needs the connections gone, and a running Nextcloud holds several.

ssh emily 'systemctl --user stop nextcloud.service'   # only if it is already deployed

```bash
# Dump on monoco (Postgres is a separate container and is still running).
ssh monoco 'mkdir -p ~/pg-migrate && cd ~/pg-migrate
  docker exec -e PGPASSWORD=123456 postgres pg_dump -U postgres -Fc nextcloud \
    | zstd -T0 -6 -o nextcloud-$(date +%Y%m%d-%H%M).dump.zst
  ls -lh nextcloud-*.dump.zst'

# Transfer, then check the checksum on both sides before trusting it.
ssh monoco 'cat ~/pg-migrate/nextcloud-*.dump.zst' \
  | ssh emily 'cat > ~/pg-migrate/nextcloud.dump.zst'
a=$(ssh monoco 'sha256sum ~/pg-migrate/nextcloud-*.dump.zst | cut -d" " -f1')
b=$(ssh emily  'sha256sum ~/pg-migrate/nextcloud.dump.zst | cut -d" " -f1')
[ -n "$a" ] && [ "$a" = "$b" ] && echo "transfer OK: $a" \
  || { echo "MISMATCH — do not restore: $a vs $b"; false; }

# Reload it on emily. This DROPS the existing database — it is the one the
# cluster move restored, and the frozen dump replaces it.
ssh emily 'cd ~/pg-migrate
  podman exec -e PGPASSWORD=123456 postgres \
    psql -U postgres -d postgres -v ON_ERROR_STOP=1 \
      -c "DROP DATABASE IF EXISTS nextcloud" \
      -c "CREATE DATABASE nextcloud OWNER postgres"
  zstd -dc nextcloud.dump.zst | podman exec -i -e PGPASSWORD=123456 postgres \
    pg_restore -U postgres -d nextcloud --no-owner --exit-on-error
  podman exec -e PGPASSWORD=123456 postgres \
    psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'

If monoco's container simply sat idle after the cluster move, skip this and go straight to the copy — but the file counts at the end of step 3 are what tell you whether the two halves agree.

3. Copy the files

# 1. Create the destination: the mount target of the image's /var/www/html.
ssh emily 'mkdir -p ~/podman-data/nextcloud'

# 2. Copy. tar streams it across the workstation; rsync cannot copy
#    remote-to-remote in one command.
#
#    The WHOLE tree is copied — code as well as state — because /var/www/html is
#    both. See "What is state, and what is not" above.
#
#    `set -o pipefail` matters here: without it the pipeline reports only the
#    *receiving* tar's status, so a read error or a dropped stream on the source
#    side exits 0 and leaves a silently short copy.
#
#    `--no-same-owner` is explicit and important: the source is 33:33 (and partly
#    root), and under keep-id:uid=33 the *container's* 33 is tanshu on the host.
#    Extracting as tanshu is therefore already the correct ownership, and
#    restoring the source's numeric owners would be wrong.
set -o pipefail
ssh monoco 'sudo tar -C /var/lib/nextcloud -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/nextcloud -xf -'

#    Without sudo, when step 0 proved everything is readable:
# ssh monoco 'tar -C /var/lib/nextcloud -cf - .' \
#   | ssh emily 'tar --no-same-owner -C ~/podman-data/nextcloud -xf -'

# 3. Verify by content, over the whole tree. Read as root on the source side
#    because data/ is 0770.
ssh monoco 'cd /var/lib/nextcloud && sudo bash -c "find . -type f -print0 | sort -z | xargs -0 -r sha256sum" | sha256sum'
ssh emily  'cd ~/podman-data/nextcloud && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum'
# identical digests = a byte-faithful copy

# 4. Nothing may end up owned by root or by a subuid.
ssh emily 'find ~/podman-data/nextcloud ! -user tanshu'    # empty = correct

# 5. A cheap sanity check that the two halves agree — compare with the count from
#    step 2, and with monoco's own tree.
ssh emily  'find ~/podman-data/nextcloud/data -type f | wc -l'
ssh monoco 'sudo find /var/lib/nextcloud/data -type f | wc -l'

Ownership needs no chown pass: keep-id:uid=33 maps the container's www-data onto the connecting user, so files extracted as tanshu are already what the container expects to own — and what plain tar, rsync and rm -rf keep working with.

If the link is slow or drops, do it in two hops instead: resumable, and it shows progress.

rsync -aP --rsync-path='sudo rsync' monoco:/var/lib/nextcloud/ ~/nextcloud-copy/
rsync -aP ~/nextcloud-copy/ emily:~/podman-data/nextcloud/

If you ran the playbook before copying

Running this play against an empty ~/podman-data/nextcloud does not fail — it creates the webroot, writes the environment file, starts the image, and the entrypoint initialises a fresh, uninstalled instance there: the image's own code rsynced in, data/.ocdata created, and no config.php, because there is no NEXTCLOUD_ADMIN_USER to install with.

That is not a usable instance, and worse, copying monoco's tree on top of it merges two versions of the code — the drift this page spends the rest of its time warning about, produced silently by a play that reported success. The role now warns when it finds a webroot with no config/config.php, but the recovery is to start from a clean webroot:

ssh emily 'systemctl --user stop nextcloud.service
           rm -rf ~/podman-data/nextcloud'

rm -rf needs no podman unshare: the role creates the webroot as the connection user, and under keep-id:uid=33 everything the container writes lands on that same user. Then take the copy in step 3 and come back to step 4.

4. Deploy

cd prepare-server && ansible-playbook -u tanshu ../nextcloud/playbook.yml

Run it after the copy: the first start is when the entrypoint compares the copied version.php with the image and, if the image is newer, runs occ upgrade. Starting before the copy would initialise an empty instance and then be overwritten by it.

Rolling back means rolling back all three halves — ssh monoco 'docker start nextcloud' is only meaningful while monoco's own Postgres 17 cluster is intact, and it would then be writing to a database emily's copy has already moved past.

Verification

systemctl --user status nextcloud
journalctl --user -u nextcloud -f
podman logs nextcloud

# What the playbook checks for you:
podman exec -u www-data nextcloud php occ status          # installed, version, maintenance
podman exec nextcloud getent hosts db                     # config.php's database host
podman exec nextcloud getent hosts redis                  # config.php's cache host
podman exec nextcloud getent hosts postgres valkey        # the container names

# The background job, and that it is actually scheduled:
systemctl --user list-timers nextcloud-cron.timer
systemctl --user start nextcloud-cron.service             # run one by hand
journalctl --user -u nextcloud-cron -n 20

# The data really is the copied tree:
ssh emily 'ls -ln ~/podman-data/nextcloud | head'
ssh emily 'test -f ~/podman-data/nextcloud/config/config.php && echo "config.php present"'

# The database really is the migrated one — run the same query on both hosts.
ssh emily  'podman exec -e PGPASSWORD=123456 postgres psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'
ssh monoco 'docker exec -e PGPASSWORD=123456 postgres psql -U postgres -d nextcloud -tAc "select count(*) from oc_filecache"'

# The site, end to end:
curl -sSI https://cloud.hopsngrains.com

Then open it in a browser and do two things a curl cannot: log in, and upload a file. Uploading is the check that matters here — it exercises the uid-33 mapping (a wrong keep-id shows up as a permissions error), the Redis file locking in config.php, and PHP sessions, which the image configures from REDIS_HOST.

After the upload, confirm on the host that the new file is owned by tanshu:

ssh emily 'find ~/podman-data/nextcloud/data -newermt "-10 minutes" -type f ! -user tanshu'
# empty = the write path lands on the right uid

Then reboot the host and confirm both the container and the timer return unattended — without loginctl enable-linger tanshu neither will, and the failure is silent.

Finally, if config.php predates the proxy (it should not — monoco already served this site through Caddy, but check), Nextcloud needs the proxy trusted before it will believe X-Forwarded-For:

podman exec -u www-data nextcloud php occ config:system:get trusted_proxies
podman exec -u www-data nextcloud php occ config:system:get overwrite.cli.url

An empty trusted_proxies means every client looks like the proxy; see caddy/readme.md, "Client IPs", which also explains why the real address may not survive the rootless port forward at all.

Debugging a unit that will not start

A unit that has hit systemd's start limit reports only a state, and Restart=always means the interesting output is in the journal, not in podman logs (the container may already be gone — Quadlet passes --rm). The play collects all of this itself when a start fails; by hand:

ssh emily 'systemctl --user status nextcloud.service --no-pager -l | head -25
           systemctl --user cat nextcloud.service            # what systemd is running
           journalctl --user -u nextcloud.service --no-pager -n 60
           podman ps -a --filter name=nextcloud
           podman logs --tail 40 nextcloud'

Read it in this order:

1. journalctl — the entrypoint's own output. cannot create ...: Permission denied for a path in /usr/local/etc/php/conf.d is the session-file problem above; anything about db/redis is the network attach; downgrading is not supported is version drift.
2. podman ps -a — a container left behind under this name makes podman run refuse to start. Quadlet passes --replace, so this is rare, and it means an interrupted run.
3. systemctl --user cat — the generated unit, if a drop-in or a Network= is in question (the usual failure is a .network file that does not exist).

The first real run of this playbook failed at a bare is-active check with no diagnostics at all, which is why the wait now lives inside the start block whose rescue collects the above.

Notes and gotchas

• REDIS_HOST makes the entrypoint write into the image, which it is not allowed to do. /usr/local/etc/php/conf.d belongs to the image's root, so the crash is a Permission denied on redis-session.ini, the unit ends up failed via systemd's start limit, and nothing about it looks like a permissions problem. The unit mounts a host-owned file over that path; see "The PHP session file" above, including why a refused write is no longer fatal.
• Maintenance mode travels with config.php. The backup plan recommends occ maintenance:mode --on before a consistent dump (docs/backup-plan.md §3.4); if monoco was left that way, the copied configuration carries maintenance => true and the site returns 503 for everything except status.php. The play waits for maintenance mode to clear — it is on for the whole of a first-start occ upgrade, so it cannot simply assert — and if it never clears, fails with the one-line command to turn it off. That is deliberate: "the app is up but every page 503s" points anywhere but at maintenance mode.
• Version drift is the one irreversible step. The copied version.php is what the entrypoint compares against the image. A newer image upgrades the instance on first start (and can only step one major version at a time); an older image makes the container exit with "downgrading is not supported". Check both before copying (step 0) and take a database dump before the first start.
• latest floats. The Docker-era playbook used the unqualified nextcloud:latest; this one fully qualifies the same tag, because rootless Podman has no TTY to answer a short-name prompt when the unit starts. Neither pins a digest, so an image bump is a normal podman pull away — and it is exactly the event that triggers an in-place upgrade, because the code lives in the mount.
• The code lives in the data directory. That is the image's design, not this playbook's, and it is why the whole tree is copied and why an image bump edits ~/podman-data/nextcloud. It is also why the backup plan lists apps/, core/ and lib/ as regenerable: they are, but only from an image that is the same version or newer.
• occ must run as www-data. It refuses to run as any other user, and under keep-id:uid=33 that is the same user the host sees as tanshu. Every podman exec in this file therefore carries -u www-data; the plain form works for getent, which does not care.
• The init lock file. ~/podman-data/nextcloud/nextcloud-init-sync.lock is pre-created by the role so it stays owned by tanshu. If a run ever reports it as a subuid, the pre-create was skipped on a first start — delete it and re-run the playbook.
• Client IPs. Nextcloud uses the client address for brute-force protection and for the X-Forwarded-For it logs. Whether the original address survives the rootless port forward depends on the forwarder. See caddy/readme.md, "Client IPs", before relying on it.
• Registering restarts all of Caddy (a brief restart of every hosted site), attaching restarts Postgres (a few seconds without a database for every attached app) and restarts Valkey (seconds without sessions or locks, though the keys themselves survive the graceful stop). See the trade-offs in caddy/readme.md, postgres/readme.md and valkey/readme.md.
• If the Caddyfile does not exist yet, this playbook creates it containing only the site block. That is a valid Caddyfile; the Caddy playbook's seeded global-options block is then not added, because its seed only fires on a missing or zero-length file. Deploy Caddy first on a fresh host.
• DNS. cloud.hopsngrains.com must resolve to emily before Caddy can obtain a certificate for it; the old certificates live in ~/podman-data/caddy/data and are reused if the hostname is unchanged.
• The backup script still assumes Docker. ~/bin/nextcloud-backup.sh uses --link postgres:db, which Podman does not have, and its sudo tar / chown $(logname) gymnastics are unnecessary now that the data is owned by you. It needs the podman exec form from postgres/readme.md, "Backups"; the database dump should also be taken with occ maintenance:mode --on around it, as docs/backup-plan.md §3.4 says.
• ~/bin/empty-trash.sh targets a container named cloud, which never existed on this host under that name — it is nextcloud — and it runs as root, which cannot podman exec into these containers at all.
• Do not restart the container on monoco. It would write to monoco's old cluster, which is being kept intact purely as a rollback.

Not done here

• Rotating the secrets that are committed in plaintext — pg_password is the fleet's shared 123456, and it lives in nextcloud.env at mode 0600 rather than in the world-readable unit file.
• Switching the background-job mode to cron (occ background:cron), which the Docker-era deployment also never did.
• Replacing the broken ~/bin/nextcloud-backup.sh with a systemd user timer, and fixing empty-trash.sh.
• Turning on ongoing monitoring — the status.php probe in the role is where a timer or an external check would hook in.