Run Valkey as a rootless Podman Quadlet
Replaces the docker_container playbook, which published 127.0.0.1:6379 and kept its /data in an anonymous Docker volume -- the one the audit flagged because no backup script can see it (docs/podman-compat-audit.md 3.1). Valkey becomes the base Quadlet unit, the same shape as Postgres: it knows nothing about its clients, and a service attaches itself by dropping a file into valkey.container.d/ via valkey/roles/attach. Nothing on the host needs 6379, so there is no PublishPort at all rather than a narrower one. The name stays `valkey` for the directory, container, unit and Podman network, and only the alias clients connect on is `redis`: Nextcloud's config.php (written by the image from REDIS_HOST=redis), outline and overleaf reach it as `redis`, while paperless-ngx and yamtrack use the container name. Renaming would touch every REDIS_HOST for no gain. The uid/gid mapping is the detail worth spelling out, because the audit only records the uid. The Alpine image's `valkey` account is 999:1000, not 999:999 -- Alpine already has a gid 999, and the Dockerfile says so before creating the group with the next id. 999 is the uid the anonymous volume was owned by, so the uid matches what was already on disk, but mapping only the uid would leave every file valkey-server writes, dump.rdb included, group-owned by a subgid. Both ids map onto the connection user, so ~/podman-data/valkey stays tanshu:tanshu. Persistence stays on deliberately, and the mount with it. The contents are disposable -- nothing is copied off monoco, and a cold start is survivable -- but this is not purely a cache: Nextcloud keeps its PHP sessions and memcache.locking in it, and Yamtrack uses it as its Celery broker. With RDB on, a rebuild is a graceful stop that writes a final snapshot on SIGTERM that the next container loads, so a redeploy does not log every Nextcloud user out or discard queued tasks. Tmpfs=/data with `save ""` would remove the mount and the whole keep-id mapping with it, and is the right shape for a pure cache, but it would make every redeploy exactly that. The rootless-Podman prerequisites are not applied here: they are the shared task file at caddy/roles/podman, and a host that has run caddy/playbook.yml has them. Two ordering details are load-bearing: - The attach include asserts that <name>.network exists before writing a drop-in that names it. Quadlet rejects the whole valkey unit, with an error that never mentions the drop-in, when a referenced .network file is missing. - A service that attaches flushes handlers before starting the app. The restart that joins Valkey to the new network is a handler, so by default it runs at the end of the play, after the app has already come up and failed to resolve `redis`. nextcloud/roles/valkey is the working example. The play waits on `valkey-cli ping` inside the container rather than trusting systemctl is-active. Restart=always reports active even while the container crash-loops, which is exactly what a wrong keep-id looks like.
This commit is contained in:
tanshu committed
1 parent
9bf2cbd7a9
commit
3587bc9001
7 files changed
+635
-52
No files matched your search
@@ -0,0 +1,94 @@
|
||||
---
|
||||
# Attach a service's network to Valkey.
|
||||
#
|
||||
# Valkey is the *base* unit and knows nothing about its clients, the same way
|
||||
# Caddy knows nothing about its upstreams. A service that needs a cache adds
|
||||
# itself by dropping files next to valkey.container, and Quadlet merges them in.
|
||||
#
|
||||
# Include this from a service playbook:
|
||||
#
|
||||
# - name: Attach this service to Valkey
|
||||
# ansible.builtin.include_tasks: "{{ playbook_dir }}/../valkey/roles/attach/tasks/main.yaml"
|
||||
# vars:
|
||||
# network_name: "{{ network_name }}"
|
||||
#
|
||||
# The including play must:
|
||||
#
|
||||
# 1. have created {{ network_name }}.network already — caddy/roles/register
|
||||
# does that, and every service here that needs a cache is also behind
|
||||
# Caddy, so its register include comes first;
|
||||
# 2. run the `podman` role (or include its task file and define the same four
|
||||
# vars), for podman_user and quadlet_dir;
|
||||
# 3. define a "Restart Valkey" handler — see
|
||||
# roles/valkey/handlers/main.yaml for the exact snippet; and
|
||||
# 4. flush handlers before starting the app, on a first attach. This include
|
||||
# notifies that restart, so by default it runs at the end of the play —
|
||||
# after the app has already started and failed to reach `redis`. A
|
||||
# `meta: flush_handlers` immediately after this include is enough; see
|
||||
# nextcloud/roles/valkey for the working example.
|
||||
#
|
||||
# It runs as the connection user, which must be the rootless podman_user.
|
||||
#
|
||||
# A service may instead write these two files from its own role; the bytes have
|
||||
# to match this file exactly, or the two will overwrite each other on every run.
|
||||
|
||||
- name: "Check that the {{ network_name }} network unit exists"
|
||||
ansible.builtin.stat:
|
||||
path: "{{ quadlet_dir }}/{{ network_name }}.network"
|
||||
register: valkey_attach_network
|
||||
|
||||
# A drop-in naming a .network file that does not exist makes Quadlet fail to
|
||||
# generate the whole valkey unit, with a message that does not mention the
|
||||
# drop-in at all. Fail here instead, where the cause is obvious.
|
||||
- name: "Fail if {{ network_name }} has no network unit"
|
||||
ansible.builtin.assert:
|
||||
that: valkey_attach_network.stat.exists
|
||||
fail_msg: >-
|
||||
{{ quadlet_dir }}/{{ network_name }}.network does not exist, so attaching
|
||||
it to Valkey would break the valkey unit. Create the network first
|
||||
(caddy/roles/register does it), then include this file.
|
||||
|
||||
- name: Ensure the Valkey Quadlet drop-in directory exists
|
||||
ansible.builtin.file:
|
||||
path: "{{ quadlet_dir }}/{{ valkey_container | default('valkey') }}.container.d"
|
||||
state: directory
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0755'
|
||||
|
||||
- name: "Attach {{ network_name }} to Valkey"
|
||||
ansible.builtin.copy:
|
||||
dest: >-
|
||||
{{ quadlet_dir }}/{{ valkey_container | default('valkey') }}.container.d/50-{{ network_name }}.conf
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0644'
|
||||
content: |
|
||||
# Added by the {{ network_name }} playbook. Do not edit by hand.
|
||||
#
|
||||
# Deleting this file removes the network from {{ valkey_container | default('valkey') }} on
|
||||
# the next daemon-reload; nothing else changes.
|
||||
[Container]
|
||||
Network={{ network_name }}.network
|
||||
notify: Restart Valkey
|
||||
|
||||
# NetworkAlias= applies to every network the container joins, which is exactly
|
||||
# right for a hub: this repo's cache clients reach Valkey as `redis` (nextcloud,
|
||||
# which writes REDIS_HOST=redis into its config, plus outline and overleaf),
|
||||
# while the rest use the container name. It is a fixed value in one shared file,
|
||||
# not a per-service option, so the same bytes are written no matter how many
|
||||
# services attach — two services writing their own aliases would make Quadlet
|
||||
# pass --network-alias twice.
|
||||
- name: "Give {{ valkey_container | default('valkey') }} the shared 'redis' alias"
|
||||
ansible.builtin.copy:
|
||||
dest: >-
|
||||
{{ quadlet_dir }}/{{ valkey_container | default('valkey') }}.container.d/10-network-aliases.conf
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0644'
|
||||
content: |
|
||||
# Shared container aliases for {{ valkey_container | default('valkey') }}.
|
||||
# Managed by Ansible (the service playbooks that use it). Do not edit.
|
||||
[Container]
|
||||
NetworkAlias=redis
|
||||
notify: Restart Valkey
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
# All handlers run as the connection user ({{ podman_user }}), so systemctl
|
||||
# --user works without setting XDG_RUNTIME_DIR.
|
||||
|
||||
- name: Reload systemd user units
|
||||
ansible.builtin.command: systemctl --user daemon-reload
|
||||
changed_when: true
|
||||
listen: Reload systemd user units
|
||||
|
||||
- name: Restart Valkey
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ unit_name }}"
|
||||
scope: user
|
||||
state: restarted
|
||||
daemon_reload: true
|
||||
listen: Restart Valkey
|
||||
|
||||
# Reference implementation for service playbooks. A service that attaches its
|
||||
# network with valkey/roles/attach must notify "Restart Valkey", and defines its
|
||||
# own copy of this handler, e.g.:
|
||||
#
|
||||
# - name: Restart Valkey
|
||||
# ansible.builtin.systemd_service:
|
||||
# name: valkey.service
|
||||
# scope: user
|
||||
# state: restarted
|
||||
# daemon_reload: true
|
||||
# listen: Restart Valkey
|
||||
#
|
||||
# A new drop-in changes the generated unit, so Valkey has to be recreated to join
|
||||
# the network — Quadlet has no live equivalent of `podman network connect`.
|
||||
#
|
||||
# The cost is different from Postgres's: Valkey is a cache, so every attached app
|
||||
# sees a cold cache rather than a period without its database. Correctness is
|
||||
# unaffected; the first requests after the restart are simply slower.
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
# Start Valkey
|
||||
#
|
||||
# Everything here runs as {{ podman_user }} — the connection user. No become, no
|
||||
# XDG_RUNTIME_DIR: SSH gave us a real PAM session, so systemd's user manager is
|
||||
# reachable via the standard environment.
|
||||
|
||||
# Created here rather than left to the container's entrypoint. That entrypoint
|
||||
# runs as container-root, which under keep-id is a subuid on the host, so a
|
||||
# directory it created would be owned by e.g. 165536 and the host user could not
|
||||
# `rm -rf` it without `podman unshare`. Creating it up front keeps everything in
|
||||
# the data root owned by {{ podman_user }}.
|
||||
#
|
||||
# Mode 0755 rather than Postgres's 0700: Valkey's entrypoint runs `find` in this
|
||||
# directory as root before dropping to uid {{ app_uid }}, and nothing sensitive
|
||||
# lives here — it is a cache.
|
||||
- name: Ensure the Valkey data directory exists
|
||||
ansible.builtin.file:
|
||||
path: "{{ host_directory }}"
|
||||
state: directory
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0755'
|
||||
|
||||
- name: Ensure the Quadlet drop-in directory exists
|
||||
ansible.builtin.file:
|
||||
path: "{{ valkey_dropin_dir }}"
|
||||
state: directory
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0755'
|
||||
|
||||
- name: Install the base Valkey Quadlet unit
|
||||
ansible.builtin.template:
|
||||
src: valkey.container.j2
|
||||
dest: "{{ quadlet_dir }}/{{ container_name }}.container"
|
||||
owner: "{{ podman_user }}"
|
||||
group: "{{ podman_user }}"
|
||||
mode: '0644'
|
||||
notify:
|
||||
- Reload systemd user units
|
||||
- Restart Valkey
|
||||
|
||||
# The wait is part of the block on purpose. `Restart=always` means systemd
|
||||
# reports the unit active even while the container crash-loops — a wrong keep-id
|
||||
# shows up as an entrypoint permission error over and over — so is-active alone
|
||||
# would call a broken cache healthy, and every app that uses it would fail later
|
||||
# with a DNS or connection error that points nowhere near the cause.
|
||||
- name: Start Valkey and wait for it to answer
|
||||
block:
|
||||
- name: Start Valkey
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ unit_name }}"
|
||||
scope: user
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
# valkey-cli is the server's own client and is guaranteed present, the same
|
||||
# way pg_isready is in the Postgres image.
|
||||
- name: Wait for Valkey to answer PING
|
||||
ansible.builtin.command: "podman exec {{ container_name }} valkey-cli ping"
|
||||
register: valkey_ping
|
||||
until: valkey_ping.stdout == 'PONG'
|
||||
retries: 12
|
||||
delay: 5
|
||||
changed_when: false
|
||||
rescue:
|
||||
# systemd reports "control process exited with error code" and nothing else,
|
||||
# which is useless for the usual failures (a bad uid mapping, an unwritable
|
||||
# data directory, a missing image). Pull the real reason out.
|
||||
- name: Collect Valkey diagnostics
|
||||
ansible.builtin.command: "{{ item }}"
|
||||
loop:
|
||||
- "systemctl --user status {{ unit_name }} --no-pager"
|
||||
- "journalctl --user -u {{ unit_name }} --no-pager -n 40"
|
||||
register: valkey_diag
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Show Valkey diagnostics
|
||||
ansible.builtin.debug:
|
||||
msg: "{{ valkey_diag.results | map(attribute='stdout') | list }}"
|
||||
|
||||
- name: Fail
|
||||
ansible.builtin.fail:
|
||||
msg: "Valkey did not come up — see the diagnostics above."
|
||||
|
||||
# Quadlet honours [Install] WantedBy= when it generates the unit, so
|
||||
# `systemctl --user enable` is neither needed nor possible on a generated unit.
|
||||
- name: Verify the Valkey unit is active
|
||||
ansible.builtin.command: "systemctl --user is-active {{ unit_name }}"
|
||||
register: valkey_active
|
||||
changed_when: false
|
||||
failed_when: valkey_active.stdout != 'active'
|
||||
|
||||
- name: List the services attached to Valkey
|
||||
ansible.builtin.find:
|
||||
paths: "{{ valkey_dropin_dir }}"
|
||||
patterns: "*.conf"
|
||||
register: valkey_dropins
|
||||
|
||||
- name: Report the networks Valkey is attached to
|
||||
ansible.builtin.command: >-
|
||||
podman inspect {{ container_name }}
|
||||
--format '{% raw %}{{ range $k, $v := .NetworkSettings.Networks }}{{ $k }} {{ end }}{% endraw %}'
|
||||
register: valkey_networks_now
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Show attached services
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Valkey networks:
|
||||
{{ valkey_networks_now.stdout | default('(container not running)') }}.
|
||||
Attached by:
|
||||
{{ valkey_dropins.files | map(attribute='path') | map('basename') | join(', ')
|
||||
| default('(nothing yet)', true) }}.
|
||||
|
||||
# Say this out loud rather than let it be discovered by a failing app: with no
|
||||
# drop-in, Valkey has no Network= at all, so it sits on the default Podman
|
||||
# network and no app network can resolve it.
|
||||
- name: Note that no service has attached yet
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
No service has attached to Valkey. It is on the default Podman network
|
||||
only, and no app can resolve it until a service playbook includes
|
||||
valkey/roles/attach (see readme.md).
|
||||
when: valkey_dropins.matched == 0
|
||||
@@ -0,0 +1,43 @@
|
||||
# Valkey — rootless Podman Quadlet unit.
|
||||
# Managed by Ansible (valkey/roles/valkey). Do not edit by hand.
|
||||
#
|
||||
# Valkey deliberately does NOT list the app networks it serves. Each service
|
||||
# attaches itself by dropping a file into:
|
||||
#
|
||||
# {{ valkey_dropin_dir }}/
|
||||
#
|
||||
# which Quadlet merges into this unit. Removing a service therefore never means
|
||||
# editing this file. See readme.md, "How services attach themselves".
|
||||
|
||||
[Unit]
|
||||
Description=Valkey in-memory data store
|
||||
Documentation=https://valkey.io/docs/
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName={{ container_name }}
|
||||
Image={{ image }}
|
||||
|
||||
# Map the container's valkey account (uid {{ app_uid }}, gid {{ app_gid }}) onto
|
||||
# {{ podman_user }} on the host, so {{ host_directory }} stays owned by
|
||||
# {{ podman_user }} — owner *and* group — and never by a subuid or a subgid.
|
||||
# The two ids differ because Alpine already has a gid 999; see vars/default.yml.
|
||||
# The image's entrypoint runs as container-root, chowns the owner of anything in
|
||||
# its WORKDIR that is not already valkey's, then drops to
|
||||
# {{ app_uid }}:{{ app_gid }} — so everything it creates lands on
|
||||
# {{ podman_user }} too.
|
||||
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
|
||||
|
||||
# No PublishPort, on purpose. The Docker-era playbook published
|
||||
# 127.0.0.1:6379; here Valkey is reachable only from the app networks that
|
||||
# attach themselves to it, so there is no host port to firewall or to collide
|
||||
# with a second cache.
|
||||
Volume={{ host_directory }}:{{ data_mount }}
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
TimeoutStartSec=120
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user