tanshu
c926f45faa
The deploy failed in the network role: the Caddy restart exited 125 with "unable to find network with name or ID mozimo_net", and with Restart=always and a 100ms delay the start limit was spent in under a second — taking every other site this host serves down with it. The network was gone, but mozimo-network.service was not: it is a oneshot with RemainAfterExit=yes, so systemd kept reporting it active after the network had been removed out of band. The generated Requires=mozimo-network.service looked satisfied, so the flushed "Restart Caddy" handler started Caddy against a network that no longer existed, and nothing recreated it. - ansible/roles/network/tasks/main.yaml: daemon-reload after templating the unit so it can be started on a first deploy; ask podman (`podman network exists`) rather than systemd whether the network is really there; restart mozimo-network.service when it is not; confirm it exists before the handler flush, so a network that cannot be repaired fails the play rather than the host's reverse proxy. - The Caddy restart condition includes the recreated case: a network that comes back keeps its name but not its identity, so Caddy's network namespace is stale even when neither template changed. - ansible/README.md: the oneshot/stale-state trap and the manual recovery. Verified on emily after recovering Caddy by hand: the play completes with 22 ok / 6 changed / 0 failed, and mozimo and vince run behind Caddy (TLS for mozimo.in still waits on the DNS cutover).
74 lines
3.0 KiB
YAML
74 lines
3.0 KiB
YAML
---
|
|
# mozimo's own podman network, and its registration with the shared Caddy.
|
|
#
|
|
# This is how services register themselves on this host: the network is a
|
|
# Quadlet unit, and Caddy joins it through a drop-in file rather than through
|
|
# a global list of upstream networks. Deleting the drop-in removes Caddy from
|
|
# the network on the next daemon-reload — the reverse of deployment.
|
|
- name: Install the mozimo network Quadlet
|
|
ansible.builtin.template:
|
|
src: mozimo.network.j2
|
|
dest: "{{ quadlet_dir }}/{{ network_quadlet }}.network"
|
|
mode: "0644"
|
|
register: mozimo_network_unit
|
|
notify: Restart Caddy
|
|
|
|
# The generator only turns the .network file into a service on a reload, and
|
|
# the recreate below has to be able to start that service on a first deploy.
|
|
- name: Generate the user service from the network Quadlet
|
|
ansible.builtin.systemd_service:
|
|
daemon_reload: true
|
|
scope: user
|
|
|
|
# systemd reports a finished oneshot as "active (exited)" for as long as its
|
|
# unit file exists, so the unit's state says nothing about the network
|
|
# underneath it: a manual `podman network rm`, a prune, or a wiped runtime
|
|
# directory all leave mozimo-network.service looking healthy. Ask podman.
|
|
#
|
|
# This matters because Caddy *requires* this network. Starting Caddy against a
|
|
# network that is not there kills it, and its restart loop then takes every
|
|
# other site on this host down with it — so a missing network has to be
|
|
# repaired here, before the handler below is flushed.
|
|
- name: Check whether the mozimo network still exists
|
|
ansible.builtin.command: "podman network exists {{ network_name }}"
|
|
register: mozimo_network_check
|
|
changed_when: false
|
|
failed_when: false
|
|
check_mode: false
|
|
|
|
- name: Recreate the mozimo network if it went missing
|
|
ansible.builtin.systemd_service:
|
|
name: "{{ network_quadlet }}-network.service"
|
|
state: restarted
|
|
scope: user
|
|
when: mozimo_network_check.rc != 0 and not ansible_check_mode
|
|
|
|
# Fail here, with the network named, rather than one task later in the Caddy
|
|
# restart, where the same failure shows up as the host's other sites going down.
|
|
- name: Confirm the mozimo network is there before Caddy restarts onto it
|
|
ansible.builtin.command: "podman network exists {{ network_name }}"
|
|
changed_when: false
|
|
check_mode: false
|
|
when: not ansible_check_mode
|
|
|
|
- name: Register the network with Caddy
|
|
ansible.builtin.template:
|
|
src: caddy-network.conf.j2
|
|
dest: "{{ caddy_dropin_dir }}/{{ caddy_dropin_file }}"
|
|
mode: "0644"
|
|
register: mozimo_caddy_dropin
|
|
notify: Restart Caddy
|
|
|
|
# Caddy has to be on the network before the site block starts proxying to it,
|
|
# otherwise the first requests after a reload are 502s. Restarting here also
|
|
# flushes before the mozimo unit comes up.
|
|
#
|
|
# A recreated network keeps its name but not its identity, so Caddy has to be
|
|
# restarted onto it even when neither template changed.
|
|
- name: Restart Caddy now if its network membership changed
|
|
ansible.builtin.meta: flush_handlers
|
|
when: >-
|
|
mozimo_network_unit.changed
|
|
or mozimo_caddy_dropin.changed
|
|
or mozimo_network_check.rc != 0
|