Mount PHP's session config, the one path the entrypoint cannot write

The real first run of this play did not fail on a transition. The journal shows
the entrypoint dying on every attempt:

    nextcloud[302063]: => Configuring PHP session handler...
    nextcloud[302063]: /entrypoint.sh: 121: cannot create
      /usr/local/etc/php/conf.d/redis-session.ini: Permission denied

With REDIS_HOST set, the entrypoint generates PHP's session-handler config on
every start, and the target directory belongs to the image's root: the php base
image does `mkdir -p "$PHP_INI_DIR/conf.d"` with no chown. The uid this container
runs as has no write bit on it, the entrypoint exits 2 before Apache is reached,
Restart=always retries that, and systemd's start limit turns it into a `failed`
unit. Nothing in that shape says "permissions": the symptom is a unit state and a
message that only exists in journalctl.

The container now mounts a host-owned file over that exact path. A bind-mounted
file replaces the directory entry, so the directory's ownership stops mattering
and the write lands on podman_user. The role templates the five lines the
entrypoint would have written, so a refused write is no longer fatal either --
PHP still gets Redis sessions instead of a crash loop.

This also corrects a reading of the previous run that this play got wrong and
documented wrongly: `activating` is not evidence of a healthy start. A
crash-looping container spends its time `activating` between restarts, so the
earlier "reports a healthy start as a failure" comment and commit message were
describing a start that was in fact failing. The wait is still right -- it stops
promptly at a terminal state and hands over to a rescue that has the logs -- but
its comment now says what it actually buys.

Apache's own runtime directories need none of this, and the php base image is why:
it chowns /var/run/apache2, /var/lock/apache2 and /var/log/apache2 to www-data and
chmods them 1777, so only the session file was ever going to bite. The readme
gains a section on it, the diagnostic commands the play now collects, and a note
that the uid mapping is not to be derived from a `docker exec id -u` of 0.

nextcloud/readme.md

@@ -70,6 +70,8 @@ Quadlet units and state:
 | `~/.config/systemd/user/nextcloud-cron.{service,timer}` | the background-job timer (not Quadlet) |
 | `~/podman-data/nextcloud/` | the whole `/var/www/html`: code, `config/config.php`, `data/`, `custom_apps/` |
 | `~/podman-data/nextcloud/nextcloud.env` | `POSTGRES_*` + `REDIS_HOST`, mode `0600`, read by systemd |
+| `~/podman-data/nextcloud/php-conf.d/redis-session.ini` | PHP's session handler, mounted into the image at `/usr/local/etc/php/conf.d/` |
+| `~/podman-data/nextcloud/nextcloud-init-sync.lock` | the entrypoint's init lock, pre-created so it stays host-owned |
 
 ## What is state, and what is not
 
@@ -271,20 +273,77 @@ Like Caddy, `podman_user` is derived from the connection and the play carries no
 connecting as `root`**; the prerequisites, including that check, live in
 `caddy/playbook.yml`.
 
-`UserNS=keep-id:uid=33,gid=33` — `33` is the uid the server writes as. The
+`UserNS=keep-id:uid=33,gid=33` — `33` is the uid the server writes as. Whatever
-entrypoint starts as container-root, but it rsyncs the code in with
+the entrypoint's *own* starting uid under this mapping, the writes that matter land
-`--chown www-data:www-data` and Apache then drops to `www-data`, so everything it
+on `podman_user`: the code is rsynced with `--chown www-data:www-data`, and Apache's
-writes lands on `podman_user` and no subuid-owned files appear to break backups
+workers run as `www-data`
-([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `33` is
+([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `33` is also
-also what the data is owned by on monoco, which is what makes the copy below need
+what the data is owned by on monoco, which is what makes the copy below need no
-no `chown` pass.
+`chown` pass.
 
-One file is created by container-root and never chowned: the entrypoint takes an
+Two paths the entrypoint writes fall outside that arrangement, and both are handled
-`flock` on `/var/www/html/nextcloud-init-sync.lock` using a root-owned shell
+explicitly rather than left to chance — each one is somewhere a naive deployment
-redirect. The role **pre-creates it** as `podman_user`, because the redirect
+ends up with a file owned by a `165536+` subuid, or with no deployment at all:
-truncates the file rather than recreating it — so its owner never changes. Left
+
-to the container it would be the one file in the webroot owned by a `165536+`
+- **`/var/www/html/nextcloud-init-sync.lock`**, which the entrypoint creates with a
-subuid and not removable by the host user.
+  root-owned shell redirect. The role **pre-creates it** as `podman_user`: the
+  redirect truncates the file rather than recreating it, so its owner never changes
+  afterwards.
+- **`/usr/local/etc/php/conf.d/redis-session.ini`**, in the image, which is not
+  writable at all — it is covered in its own section below.
+
+### The PHP session file
+
+The webroot is not the only thing the entrypoint writes. With `REDIS_HOST` set —
+which it is, because `config.php` puts `memcache.locking` in Redis and PHP
+sessions should survive a redeploy — the entrypoint generates that session
+configuration on **every** start:
+
+```sh
+/entrypoint.sh:121
+    } > /usr/local/etc/php/conf.d/redis-session.ini
+```
+
+That directory belongs to the image's root, created by the php base image with no
+`chown`:
+
+```dockerfile
+RUN set -eux; \
+	mkdir -p "$PHP_INI_DIR/conf.d"; \
+	...
+```
+
+so the uid this container runs as has no write bit on it, and the redirect fails:
+
+```
+cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied
+```
+
+The entrypoint exits `2` before Apache is ever reached, `Restart=always` retries
+it, systemd hits its start limit, and the unit ends up **`failed`** — with the
+whole thing looking like a network or database problem, because that message only
+appears in the journal.
+
+The fix is a mount: `nextcloud.container` binds
+`~/podman-data/nextcloud/php-conf.d/redis-session.ini` over that exact path. A
+bind-mounted file replaces the directory entry, so the directory's ownership stops
+mattering and the write lands on a file owned by `podman_user`. The role also
+templates the file with the five lines the entrypoint would have written, so a
+start where the write is somehow refused still gets working Redis sessions instead
+of a crash loop.
+
+Apache's *own* runtime directories need none of this — the same base image chowns
+`/var/run/apache2`, `/var/lock/apache2` and `/var/log/apache2` to `www-data` and
+chmods them `1777`:
+
+```dockerfile
+	for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR" \
+		"$APACHE_RUN_DIR/socks"; do
+		rm -rvf "$dir"; mkdir -p "$dir"; \
+		chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir"; \
+		chmod 1777 "$dir"; \
+	done
+```
 
 `POSTGRES_PASSWORD` is the fleet's shared `123456`, which matches
 `pg_password` in [postgres/vars/default.yml](../postgres/vars/default.yml) and
@@ -548,8 +607,46 @@ An empty `trusted_proxies` means every client looks like the proxy; see
 [caddy/readme.md](../caddy/readme.md), "Client IPs", which also explains why the
 real address may not survive the rootless port forward at all.
 
+### Debugging a unit that will not start
+
+A unit that has hit systemd's start limit reports only a state, and `Restart=always`
+means the interesting output is in the journal, not in `podman logs` (the container
+may already be gone — Quadlet passes `--rm`). The play collects all of this itself
+when a start fails; by hand:
+
+```bash
+ssh emily 'systemctl --user status nextcloud.service --no-pager -l | head -25
+           systemctl --user cat nextcloud.service            # what systemd is running
+           journalctl --user -u nextcloud.service --no-pager -n 60
+           podman ps -a --filter name=nextcloud
+           podman logs --tail 40 nextcloud'
+```
+
+Read it in this order:
+
+1. **`journalctl`** — the entrypoint's own output. `cannot create ...: Permission
+   denied` for a path in `/usr/local/etc/php/conf.d` is the session-file problem
+   above; anything about `db`/`redis` is the network attach; `downgrading is not
+   supported` is version drift.
+2. **`podman ps -a`** — a container left behind under this name makes `podman run`
+   refuse to start. Quadlet passes `--replace`, so this is rare, and it means an
+   interrupted run.
+3. **`systemctl --user cat`** — the generated unit, if a drop-in or a `Network=`
+   is in question (the usual failure is a `.network` file that does not exist).
+
+The first real run of this playbook failed at a bare `is-active` check with no
+diagnostics at all, which is why the wait now lives inside the start block whose
+rescue collects the above.
+
 ## Notes and gotchas
 
+- **`REDIS_HOST` makes the entrypoint write into the image, which it is not
+  allowed to do.** `/usr/local/etc/php/conf.d` belongs to the image's root, so the
+  crash is a `Permission denied` on `redis-session.ini`, the unit ends up `failed`
+  via systemd's start limit, and nothing about it looks like a permissions
+  problem. The unit mounts a host-owned file over that path; see
+  ["The PHP session file"](#the-php-session-file) above, including why a refused
+  write is no longer fatal.
 - **Maintenance mode travels with `config.php`.** The backup plan recommends
   `occ maintenance:mode --on` before a consistent dump
   ([docs/backup-plan.md §3.4](../docs/backup-plan.md)); if monoco was left that

nextcloud/roles/nextcloud/tasks/main.yaml

@@ -77,6 +77,43 @@
     mode: '0600'
   notify: Restart Nextcloud
 
+# The second path the entrypoint writes, and the one that made this play fail for
+# real: with REDIS_HOST set, the entrypoint generates PHP's session-handler config
+# at {{ php_session_ini_mount }} on every start, and that path's directory in the
+# php base image is
+#
+#     RUN mkdir -p "$PHP_INI_DIR/conf.d"
+#
+# — created by root, never chowned. The uid this container runs as therefore has
+# no write bit on the directory, the entrypoint dies with
+#
+#     /entrypoint.sh: 121: cannot create
+#     /usr/local/etc/php/conf.d/redis-session.ini: Permission denied
+#
+# and `Restart=always` turns that into a crash loop that systemd eventually marks
+# `failed`. The Quadlet unit mounts this file over that exact path, which is what
+# gives the write somewhere it is allowed: the mount replaces the directory entry,
+# so the directory's ownership stops mattering.
+#
+# The file is templated rather than left empty so that a start in which the
+# entrypoint's write is refused still has working Redis sessions.
+- name: Ensure the PHP session config directory exists
+  ansible.builtin.file:
+    path: "{{ php_conf_dir }}"
+    state: directory
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0755'
+
+- name: Write the PHP session handler config
+  ansible.builtin.template:
+    src: redis-session.ini.j2
+    dest: "{{ php_session_ini }}"
+    owner: "{{ podman_user }}"
+    group: "{{ podman_user }}"
+    mode: '0644'
+  notify: Restart Nextcloud
+
 - name: Install the Nextcloud Quadlet unit
   ansible.builtin.template:
     src: nextcloud.container.j2
@@ -101,22 +138,22 @@
     # `systemctl --user enable` is neither needed nor possible on a generated
     # unit.
     #
-    # This waits rather than checking once. `systemctl start` returns while the
+    # This waits rather than checking once, because `systemctl start` returns
-    # unit is still `activating` — the start job is done before the container has
+    # while the unit is still `activating` and `is-active` exits 3 for that state.
-    # settled — and `is-active` exits 3 for `activating`, so a single check
-    # reports a healthy start as a failure. That is what this play's first real
-    # run did, stopping with `stdout: "activating"`.
     #
-    # It retries only through that transition. A unit that is `failed` or
+    # `activating` is NOT evidence that a start is healthy, which this play got
-    # `inactive` is not going to become `active` by waiting, so it stops there
+    # wrong once and should not again: a container that crash-loops spends its
-    # and lets the rescue below get the reason — which is how the second run
+    # time `activating` between `Restart=always` attempts, and the first run here
-    # found out the container was crash-looping rather than merely slow. The case
+    # was read as a healthy start mid-transition when the entrypoint was in fact
-    # that genuinely needs patience, the app not answering yet, is the readiness
+    # dying on every attempt (it could not write
-    # gate further down, which has its own retries.
+    # /usr/local/etc/php/conf.d/redis-session.ini — see the task that mounts that
+    # file). What this wait actually buys is a prompt, bounded stop: it retries
+    # only while the state can still change on its own, and a terminal `failed`
+    # or `inactive` falls straight through to the rescue below, which has the logs.
     #
-    # Keeping this inside the block matters: an is-active check outside it fails
+    # Keeping it inside the block matters: an is-active check outside it fails with
-    # with a bare state and no diagnostics, which is exactly the dead end the
+    # a bare state and no diagnostics, which is the dead end the second run hit
-    # second run hit.
+    # before this was moved in.
     - name: Wait for the Nextcloud unit to become active
       ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
       register: nextcloud_active

nextcloud/roles/nextcloud/templates/nextcloud.container.j2

@@ -45,6 +45,19 @@ Network={{ network_name }}.network
 # migration copies the tree rather than only the state. See readme.md.
 Volume={{ host_directory }}:{{ webroot }}
 
+# PHP's session-handler config. The entrypoint writes this file itself on every
+# start, from REDIS_HOST, but it cannot create it: `/usr/local/etc/php/conf.d` is
+# created by root in the php base image with no chown, so the uid this container
+# runs as has no write bit on it. Mounting a host-owned file over the path is what
+# makes that write land somewhere allowed — and because the mounted file replaces
+# the directory entry, the directory's ownership never comes into it.
+#
+# This is the one place the entrypoint writes outside the webroot. Apache's own
+# runtime directories (/var/run/apache2, /var/lock/apache2, /var/log/apache2) do
+# not need the same treatment: the php base image chowns them to www-data and
+# chmods them 1777.
+Volume={{ php_session_ini }}:{{ php_session_ini_mount }}
+
 [Service]
 Restart=always
 # The entrypoint rsyncs the image's code into the webroot and, if the copied

nextcloud/roles/nextcloud/templates/redis-session.ini.j2

@@ -0,0 +1,18 @@
+; PHP session handler for Redis.
+; Managed by Ansible (nextcloud/roles/nextcloud). Do not edit by hand.
+;
+; The nextcloud image writes this file itself on every start — it is how
+; REDIS_HOST reaches PHP — and this file exists because that write has nowhere to
+; land on its own: the image's /usr/local/etc/php/conf.d belongs to root and is
+; not writable by the uid this container runs as, which makes the entrypoint exit
+; before Apache ever starts. See readme.md, "The PHP session file".
+;
+; The contents are what the entrypoint generates for a Redis host with no
+; password, so a start in which the entrypoint's own write is refused still gets
+; working Redis sessions — and the entrypoint overwrites this file with its own
+; version whenever it can.
+session.save_handler = redis
+session.save_path = "tcp://{{ redis_host }}:6379"
+redis.session.locking_enabled = 1
+redis.session.lock_retries = -1
+redis.session.lock_wait_time = 10000

nextcloud/vars/default.yml

@@ -42,16 +42,29 @@ network_name: "{{ container_name }}"
 host_directory: "{{ podman_data_dir }}/{{ container_name }}"
 webroot: /var/www/html
 
-# The uid the *server* writes as: the image's entrypoint starts as container-root,
+# The uid the *server* writes as: 33 is www-data, which owns the webroot, because
-# rsyncs the code in with --chown www-data:www-data, and Apache then drops to
+# the entrypoint rsyncs the code in with --chown www-data:www-data and Apache runs
-# www-data. `docker exec nextcloud id -u` reports 0, which is misleading
+# its workers as that account. `docker exec nextcloud id -u` reported 0 on the old
-# (docs/podman-compat-audit.md §3.2).
+# host, because the entrypoint drops privileges after starting — do not derive the
+# mapping from that number (docs/podman-compat-audit.md §3.2).
 app_uid: 33
 app_gid: 33
 
 # The port Apache listens on inside the container, and the one Caddy proxies to.
 container_port: 80
 
+# PHP's session-handler config, and the one path in the image this deployment has
+# to write to. The entrypoint generates the file from REDIS_HOST on every start,
+# but its target directory (`/usr/local/etc/php/conf.d`) is created by root in the
+# php base image with no chown, so the uid this container runs as cannot create a
+# file inside it. Mounting a host-owned file over that exact path is what gives
+# the write somewhere it is allowed to land, and the template below keeps the
+# image's own content in it so a refused write is not fatal either. See readme.md,
+# "The PHP session file".
+php_conf_dir: "{{ host_directory }}/php-conf.d"
+php_session_ini: "{{ php_conf_dir }}/redis-session.ini"
+php_session_ini_mount: /usr/local/etc/php/conf.d/redis-session.ini
+
 # POSTGRES_* and REDIS_HOST are read by the entrypoint on every start — REDIS_HOST
 # in particular rewrites PHP's session handler config — and POSTGRES_PASSWORD is in
 # there, so it goes in this 0600 file rather than in the world-readable unit.