Get the reason for a failed Nextcloud unit, not just the state
The second run of this play stopped at "Wait for the Nextcloud unit to become
active" with:
{"attempts": 2, "cmd": ["systemctl", "--user", "is-active",
"nextcloud.service"], "rc": 3, "stdout": "failed"}
The wait itself did the right thing: `failed` is not `activating`, so it stopped
immediately instead of retrying for five minutes. But it sat outside the start
block, so the rescue that pulls status, journalctl and podman logs never ran, and
the failure carried no reason at all.
Move the wait inside "Start Nextcloud" so one rescue covers both, and widen the
diagnostics for the failure that is most likely after an interrupted run:
- `systemctl --user cat` -- the generated unit, which settles what systemd thinks
it is waiting for when a unit sits in `activating`.
- `podman ps -a --filter name=...` -- a container left behind under this name
makes `podman run` refuse to start, `Restart=always` retries that into systemd's
start limit, and the unit ends up `failed` while `podman logs` shows only the
previous container and explains nothing.
The final message now names that case and the commands that clear it, and points
an `activating`-for-the-whole-budget unit at the generated unit and journalctl
instead.
This commit is contained in:
1 parent
d9ab3bc904
commit
90b1149413
1 file changed
+55
-31
@@ -88,7 +88,7 @@
|
||||
- Reload systemd user units
|
||||
- Restart Nextcloud
|
||||
|
||||
- name: Start Nextcloud
|
||||
- name: Start Nextcloud and wait for the unit to become active
|
||||
block:
|
||||
- name: Start Nextcloud
|
||||
ansible.builtin.systemd_service:
|
||||
@@ -96,16 +96,57 @@
|
||||
scope: user
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
# Quadlet honours [Install] WantedBy= when it generates the unit, so
|
||||
# `systemctl --user enable` is neither needed nor possible on a generated
|
||||
# unit.
|
||||
#
|
||||
# This waits rather than checking once. `systemctl start` returns while the
|
||||
# unit is still `activating` — the start job is done before the container has
|
||||
# settled — and `is-active` exits 3 for `activating`, so a single check
|
||||
# reports a healthy start as a failure. That is what this play's first real
|
||||
# run did, stopping with `stdout: "activating"`.
|
||||
#
|
||||
# It retries only through that transition. A unit that is `failed` or
|
||||
# `inactive` is not going to become `active` by waiting, so it stops there
|
||||
# and lets the rescue below get the reason — which is how the second run
|
||||
# found out the container was crash-looping rather than merely slow. The case
|
||||
# that genuinely needs patience, the app not answering yet, is the readiness
|
||||
# gate further down, which has its own retries.
|
||||
#
|
||||
# Keeping this inside the block matters: an is-active check outside it fails
|
||||
# with a bare state and no diagnostics, which is exactly the dead end the
|
||||
# second run hit.
|
||||
- name: Wait for the Nextcloud unit to become active
|
||||
ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
|
||||
register: nextcloud_active
|
||||
until: >-
|
||||
nextcloud_active.stdout == 'active'
|
||||
or nextcloud_active.stdout not in ['activating', 'reloading']
|
||||
retries: 60
|
||||
delay: 5
|
||||
changed_when: false
|
||||
failed_when: nextcloud_active.stdout != 'active'
|
||||
rescue:
|
||||
# systemd reports "control process exited with error code" and nothing else,
|
||||
# which is useless for the usual failures (an unattached network, an
|
||||
# unreachable database, an unwritable webroot). Pull the real reason out.
|
||||
# unreachable database, an unwritable webroot, a container name left behind
|
||||
# by an interrupted run). Pull the real reason out.
|
||||
#
|
||||
# `cat` is in the list because the generated unit settles what systemd thinks
|
||||
# it is waiting for, and `podman ps -a` is in it because the failure most
|
||||
# likely to bite after an aborted run is a container that already exists
|
||||
# under this name: `podman run` then refuses to start, `Restart=always`
|
||||
# retries it into systemd's start limit, and the unit ends up `failed` with
|
||||
# nothing in `podman logs` to explain it.
|
||||
- name: Collect Nextcloud diagnostics
|
||||
ansible.builtin.command: "{{ item }}"
|
||||
loop:
|
||||
- "systemctl --user status {{ unit_name }} --no-pager"
|
||||
- "journalctl --user -u {{ unit_name }} --no-pager -n 40"
|
||||
- "podman logs --tail 40 {{ container_name }}"
|
||||
- "systemctl --user status {{ unit_name }} --no-pager -l"
|
||||
- "systemctl --user cat {{ unit_name }}"
|
||||
- "journalctl --user -u {{ unit_name }} --no-pager -n 60"
|
||||
- "podman ps -a --filter name={{ container_name }}"
|
||||
- "podman logs --tail 60 {{ container_name }}"
|
||||
register: nextcloud_diag
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
@@ -116,32 +157,15 @@
|
||||
|
||||
- name: Fail
|
||||
ansible.builtin.fail:
|
||||
msg: "Nextcloud did not start — see the diagnostics above."
|
||||
|
||||
# Quadlet honours [Install] WantedBy= when it generates the unit, so
|
||||
# `systemctl --user enable` is neither needed nor possible on a generated unit.
|
||||
#
|
||||
# This waits rather than checking once. `systemctl start` returns while the unit
|
||||
# is still `activating`: the start job is done before conmon has signalled
|
||||
# readiness, and `is-active` exits 3 for `activating` — so a single check reports
|
||||
# a healthy start as a failure. That is exactly what happened on this play's
|
||||
# first real run, which stopped here with `stdout: "activating"`.
|
||||
#
|
||||
# It retries only through that transition. A unit that is `failed` or `inactive`
|
||||
# is not going to become `active` by waiting, so it fails immediately with the
|
||||
# state in the message instead of five minutes later. The case that genuinely
|
||||
# needs patience — the app answering yet — is the readiness gate below, which has
|
||||
# the richer rescue.
|
||||
- name: Wait for the Nextcloud unit to become active
|
||||
ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
|
||||
register: nextcloud_active
|
||||
until: >-
|
||||
nextcloud_active.stdout == 'active'
|
||||
or nextcloud_active.stdout not in ['activating', 'reloading']
|
||||
retries: 60
|
||||
delay: 5
|
||||
changed_when: false
|
||||
failed_when: nextcloud_active.stdout != 'active'
|
||||
msg: >-
|
||||
Nextcloud did not come up — see the diagnostics above. If `podman ps
|
||||
-a` shows a container that already exists under this name, it was left
|
||||
behind by an interrupted run; clear it and start again with:
|
||||
systemctl --user reset-failed {{ unit_name }} ;
|
||||
systemctl --user stop {{ unit_name }} ; podman rm -f {{ container_name }}
|
||||
A unit that is `activating` for the whole retry budget is a different
|
||||
problem: look at the generated unit in the diagnostics, and at
|
||||
journalctl for the container's own output.
|
||||
|
||||
# `is-active` says the container is running; it does not say the app can serve.
|
||||
# The entrypoint rsyncs the image's code into the webroot and, when the copied
|
||||
|
||||
Reference in new issue
Block a user