diff --git a/nextcloud/readme.md b/nextcloud/readme.md index e13e1dd..f3ecea2 100644 --- a/nextcloud/readme.md +++ b/nextcloud/readme.md @@ -70,6 +70,8 @@ Quadlet units and state: | `~/.config/systemd/user/nextcloud-cron.{service,timer}` | the background-job timer (not Quadlet) | | `~/podman-data/nextcloud/` | the whole `/var/www/html`: code, `config/config.php`, `data/`, `custom_apps/` | | `~/podman-data/nextcloud/nextcloud.env` | `POSTGRES_*` + `REDIS_HOST`, mode `0600`, read by systemd | +| `~/podman-data/nextcloud/php-conf.d/redis-session.ini` | PHP's session handler, mounted into the image at `/usr/local/etc/php/conf.d/` | +| `~/podman-data/nextcloud/nextcloud-init-sync.lock` | the entrypoint's init lock, pre-created so it stays host-owned | ## What is state, and what is not @@ -271,20 +273,77 @@ Like Caddy, `podman_user` is derived from the connection and the play carries no connecting as `root`**; the prerequisites, including that check, live in `caddy/playbook.yml`. -`UserNS=keep-id:uid=33,gid=33` — `33` is the uid the server writes as. The -entrypoint starts as container-root, but it rsyncs the code in with -`--chown www-data:www-data` and Apache then drops to `www-data`, so everything it -writes lands on `podman_user` and no subuid-owned files appear to break backups -([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `33` is -also what the data is owned by on monoco, which is what makes the copy below need -no `chown` pass. +`UserNS=keep-id:uid=33,gid=33` — `33` is the uid the server writes as. Whatever +the entrypoint's *own* starting uid under this mapping, the writes that matter land +on `podman_user`: the code is rsynced with `--chown www-data:www-data`, and Apache's +workers run as `www-data` +([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). `33` is also +what the data is owned by on monoco, which is what makes the copy below need no +`chown` pass. -One file is created by container-root and never chowned: the entrypoint takes an -`flock` on `/var/www/html/nextcloud-init-sync.lock` using a root-owned shell -redirect. The role **pre-creates it** as `podman_user`, because the redirect -truncates the file rather than recreating it — so its owner never changes. Left -to the container it would be the one file in the webroot owned by a `165536+` -subuid and not removable by the host user. +Two paths the entrypoint writes fall outside that arrangement, and both are handled +explicitly rather than left to chance — each one is somewhere a naive deployment +ends up with a file owned by a `165536+` subuid, or with no deployment at all: + +- **`/var/www/html/nextcloud-init-sync.lock`**, which the entrypoint creates with a + root-owned shell redirect. The role **pre-creates it** as `podman_user`: the + redirect truncates the file rather than recreating it, so its owner never changes + afterwards. +- **`/usr/local/etc/php/conf.d/redis-session.ini`**, in the image, which is not + writable at all — it is covered in its own section below. + +### The PHP session file + +The webroot is not the only thing the entrypoint writes. With `REDIS_HOST` set — +which it is, because `config.php` puts `memcache.locking` in Redis and PHP +sessions should survive a redeploy — the entrypoint generates that session +configuration on **every** start: + +```sh +/entrypoint.sh:121 + } > /usr/local/etc/php/conf.d/redis-session.ini +``` + +That directory belongs to the image's root, created by the php base image with no +`chown`: + +```dockerfile +RUN set -eux; \ + mkdir -p "$PHP_INI_DIR/conf.d"; \ + ... +``` + +so the uid this container runs as has no write bit on it, and the redirect fails: + +``` +cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied +``` + +The entrypoint exits `2` before Apache is ever reached, `Restart=always` retries +it, systemd hits its start limit, and the unit ends up **`failed`** — with the +whole thing looking like a network or database problem, because that message only +appears in the journal. + +The fix is a mount: `nextcloud.container` binds +`~/podman-data/nextcloud/php-conf.d/redis-session.ini` over that exact path. A +bind-mounted file replaces the directory entry, so the directory's ownership stops +mattering and the write lands on a file owned by `podman_user`. The role also +templates the file with the five lines the entrypoint would have written, so a +start where the write is somehow refused still gets working Redis sessions instead +of a crash loop. + +Apache's *own* runtime directories need none of this — the same base image chowns +`/var/run/apache2`, `/var/lock/apache2` and `/var/log/apache2` to `www-data` and +chmods them `1777`: + +```dockerfile + for dir in "$APACHE_LOCK_DIR" "$APACHE_RUN_DIR" "$APACHE_LOG_DIR" \ + "$APACHE_RUN_DIR/socks"; do + rm -rvf "$dir"; mkdir -p "$dir"; \ + chown "$APACHE_RUN_USER:$APACHE_RUN_GROUP" "$dir"; \ + chmod 1777 "$dir"; \ + done +``` `POSTGRES_PASSWORD` is the fleet's shared `123456`, which matches `pg_password` in [postgres/vars/default.yml](../postgres/vars/default.yml) and @@ -548,8 +607,46 @@ An empty `trusted_proxies` means every client looks like the proxy; see [caddy/readme.md](../caddy/readme.md), "Client IPs", which also explains why the real address may not survive the rootless port forward at all. +### Debugging a unit that will not start + +A unit that has hit systemd's start limit reports only a state, and `Restart=always` +means the interesting output is in the journal, not in `podman logs` (the container +may already be gone — Quadlet passes `--rm`). The play collects all of this itself +when a start fails; by hand: + +```bash +ssh emily 'systemctl --user status nextcloud.service --no-pager -l | head -25 + systemctl --user cat nextcloud.service # what systemd is running + journalctl --user -u nextcloud.service --no-pager -n 60 + podman ps -a --filter name=nextcloud + podman logs --tail 40 nextcloud' +``` + +Read it in this order: + +1. **`journalctl`** — the entrypoint's own output. `cannot create ...: Permission + denied` for a path in `/usr/local/etc/php/conf.d` is the session-file problem + above; anything about `db`/`redis` is the network attach; `downgrading is not + supported` is version drift. +2. **`podman ps -a`** — a container left behind under this name makes `podman run` + refuse to start. Quadlet passes `--replace`, so this is rare, and it means an + interrupted run. +3. **`systemctl --user cat`** — the generated unit, if a drop-in or a `Network=` + is in question (the usual failure is a `.network` file that does not exist). + +The first real run of this playbook failed at a bare `is-active` check with no +diagnostics at all, which is why the wait now lives inside the start block whose +rescue collects the above. + ## Notes and gotchas +- **`REDIS_HOST` makes the entrypoint write into the image, which it is not + allowed to do.** `/usr/local/etc/php/conf.d` belongs to the image's root, so the + crash is a `Permission denied` on `redis-session.ini`, the unit ends up `failed` + via systemd's start limit, and nothing about it looks like a permissions + problem. The unit mounts a host-owned file over that path; see + ["The PHP session file"](#the-php-session-file) above, including why a refused + write is no longer fatal. - **Maintenance mode travels with `config.php`.** The backup plan recommends `occ maintenance:mode --on` before a consistent dump ([docs/backup-plan.md §3.4](../docs/backup-plan.md)); if monoco was left that diff --git a/nextcloud/roles/nextcloud/tasks/main.yaml b/nextcloud/roles/nextcloud/tasks/main.yaml index 5513872..aea1d72 100644 --- a/nextcloud/roles/nextcloud/tasks/main.yaml +++ b/nextcloud/roles/nextcloud/tasks/main.yaml @@ -77,6 +77,43 @@ mode: '0600' notify: Restart Nextcloud +# The second path the entrypoint writes, and the one that made this play fail for +# real: with REDIS_HOST set, the entrypoint generates PHP's session-handler config +# at {{ php_session_ini_mount }} on every start, and that path's directory in the +# php base image is +# +# RUN mkdir -p "$PHP_INI_DIR/conf.d" +# +# — created by root, never chowned. The uid this container runs as therefore has +# no write bit on the directory, the entrypoint dies with +# +# /entrypoint.sh: 121: cannot create +# /usr/local/etc/php/conf.d/redis-session.ini: Permission denied +# +# and `Restart=always` turns that into a crash loop that systemd eventually marks +# `failed`. The Quadlet unit mounts this file over that exact path, which is what +# gives the write somewhere it is allowed: the mount replaces the directory entry, +# so the directory's ownership stops mattering. +# +# The file is templated rather than left empty so that a start in which the +# entrypoint's write is refused still has working Redis sessions. +- name: Ensure the PHP session config directory exists + ansible.builtin.file: + path: "{{ php_conf_dir }}" + state: directory + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0755' + +- name: Write the PHP session handler config + ansible.builtin.template: + src: redis-session.ini.j2 + dest: "{{ php_session_ini }}" + owner: "{{ podman_user }}" + group: "{{ podman_user }}" + mode: '0644' + notify: Restart Nextcloud + - name: Install the Nextcloud Quadlet unit ansible.builtin.template: src: nextcloud.container.j2 @@ -101,22 +138,22 @@ # `systemctl --user enable` is neither needed nor possible on a generated # unit. # - # This waits rather than checking once. `systemctl start` returns while the - # unit is still `activating` — the start job is done before the container has - # settled — and `is-active` exits 3 for `activating`, so a single check - # reports a healthy start as a failure. That is what this play's first real - # run did, stopping with `stdout: "activating"`. + # This waits rather than checking once, because `systemctl start` returns + # while the unit is still `activating` and `is-active` exits 3 for that state. # - # It retries only through that transition. A unit that is `failed` or - # `inactive` is not going to become `active` by waiting, so it stops there - # and lets the rescue below get the reason — which is how the second run - # found out the container was crash-looping rather than merely slow. The case - # that genuinely needs patience, the app not answering yet, is the readiness - # gate further down, which has its own retries. + # `activating` is NOT evidence that a start is healthy, which this play got + # wrong once and should not again: a container that crash-loops spends its + # time `activating` between `Restart=always` attempts, and the first run here + # was read as a healthy start mid-transition when the entrypoint was in fact + # dying on every attempt (it could not write + # /usr/local/etc/php/conf.d/redis-session.ini — see the task that mounts that + # file). What this wait actually buys is a prompt, bounded stop: it retries + # only while the state can still change on its own, and a terminal `failed` + # or `inactive` falls straight through to the rescue below, which has the logs. # - # Keeping this inside the block matters: an is-active check outside it fails - # with a bare state and no diagnostics, which is exactly the dead end the - # second run hit. + # Keeping it inside the block matters: an is-active check outside it fails with + # a bare state and no diagnostics, which is the dead end the second run hit + # before this was moved in. - name: Wait for the Nextcloud unit to become active ansible.builtin.command: "systemctl --user is-active {{ unit_name }}" register: nextcloud_active diff --git a/nextcloud/roles/nextcloud/templates/nextcloud.container.j2 b/nextcloud/roles/nextcloud/templates/nextcloud.container.j2 index 075ce09..c011cda 100644 --- a/nextcloud/roles/nextcloud/templates/nextcloud.container.j2 +++ b/nextcloud/roles/nextcloud/templates/nextcloud.container.j2 @@ -45,6 +45,19 @@ Network={{ network_name }}.network # migration copies the tree rather than only the state. See readme.md. Volume={{ host_directory }}:{{ webroot }} +# PHP's session-handler config. The entrypoint writes this file itself on every +# start, from REDIS_HOST, but it cannot create it: `/usr/local/etc/php/conf.d` is +# created by root in the php base image with no chown, so the uid this container +# runs as has no write bit on it. Mounting a host-owned file over the path is what +# makes that write land somewhere allowed — and because the mounted file replaces +# the directory entry, the directory's ownership never comes into it. +# +# This is the one place the entrypoint writes outside the webroot. Apache's own +# runtime directories (/var/run/apache2, /var/lock/apache2, /var/log/apache2) do +# not need the same treatment: the php base image chowns them to www-data and +# chmods them 1777. +Volume={{ php_session_ini }}:{{ php_session_ini_mount }} + [Service] Restart=always # The entrypoint rsyncs the image's code into the webroot and, if the copied diff --git a/nextcloud/roles/nextcloud/templates/redis-session.ini.j2 b/nextcloud/roles/nextcloud/templates/redis-session.ini.j2 new file mode 100644 index 0000000..ad1f1ae --- /dev/null +++ b/nextcloud/roles/nextcloud/templates/redis-session.ini.j2 @@ -0,0 +1,18 @@ +; PHP session handler for Redis. +; Managed by Ansible (nextcloud/roles/nextcloud). Do not edit by hand. +; +; The nextcloud image writes this file itself on every start — it is how +; REDIS_HOST reaches PHP — and this file exists because that write has nowhere to +; land on its own: the image's /usr/local/etc/php/conf.d belongs to root and is +; not writable by the uid this container runs as, which makes the entrypoint exit +; before Apache ever starts. See readme.md, "The PHP session file". +; +; The contents are what the entrypoint generates for a Redis host with no +; password, so a start in which the entrypoint's own write is refused still gets +; working Redis sessions — and the entrypoint overwrites this file with its own +; version whenever it can. +session.save_handler = redis +session.save_path = "tcp://{{ redis_host }}:6379" +redis.session.locking_enabled = 1 +redis.session.lock_retries = -1 +redis.session.lock_wait_time = 10000 diff --git a/nextcloud/vars/default.yml b/nextcloud/vars/default.yml index 33e3350..6db2313 100644 --- a/nextcloud/vars/default.yml +++ b/nextcloud/vars/default.yml @@ -42,16 +42,29 @@ network_name: "{{ container_name }}" host_directory: "{{ podman_data_dir }}/{{ container_name }}" webroot: /var/www/html -# The uid the *server* writes as: the image's entrypoint starts as container-root, -# rsyncs the code in with --chown www-data:www-data, and Apache then drops to -# www-data. `docker exec nextcloud id -u` reports 0, which is misleading -# (docs/podman-compat-audit.md §3.2). +# The uid the *server* writes as: 33 is www-data, which owns the webroot, because +# the entrypoint rsyncs the code in with --chown www-data:www-data and Apache runs +# its workers as that account. `docker exec nextcloud id -u` reported 0 on the old +# host, because the entrypoint drops privileges after starting — do not derive the +# mapping from that number (docs/podman-compat-audit.md §3.2). app_uid: 33 app_gid: 33 # The port Apache listens on inside the container, and the one Caddy proxies to. container_port: 80 +# PHP's session-handler config, and the one path in the image this deployment has +# to write to. The entrypoint generates the file from REDIS_HOST on every start, +# but its target directory (`/usr/local/etc/php/conf.d`) is created by root in the +# php base image with no chown, so the uid this container runs as cannot create a +# file inside it. Mounting a host-owned file over that exact path is what gives +# the write somewhere it is allowed to land, and the template below keeps the +# image's own content in it so a refused write is not fatal either. See readme.md, +# "The PHP session file". +php_conf_dir: "{{ host_directory }}/php-conf.d" +php_session_ini: "{{ php_conf_dir }}/redis-session.ini" +php_session_ini_mount: /usr/local/etc/php/conf.d/redis-session.ini + # POSTGRES_* and REDIS_HOST are read by the entrypoint on every start — REDIS_HOST # in particular rewrites PHP's session handler config — and POSTGRES_PASSWORD is in # there, so it goes in this 0600 file rather than in the world-readable unit.