playbooks/valkey

Valkey (rootless Podman + Quadlet)

Runs Valkey (the Redis fork this repo already used) as a rootless Podman container managed by a systemd user unit (Quadlet). Replaces the docker_container playbook, which published 127.0.0.1:6379 and kept its /data in an anonymous Docker volume — docker volume ls state that every backup script in ~/bin/ missed entirely (docs/podman-compat-audit.md §3.1).

Valkey is the base unit and knows nothing about its clients. It does not list any app network and does not need editing when a service is added or removed. Each service attaches itself. That is the same design as Postgres, and the mirror image of Caddy.

The name is split on purpose: the directory, container, unit and Podman network are valkey, exactly as the Docker-era playbook had them, and only the alias clients connect on is redis. Renaming the container would touch every app's REDIS_HOST for no gain.

Layout

valkey/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── valkey/     # the base valkey.container unit, data dir, handlers
    └── attach/     # includable task file: a service attaches itself to Valkey

Everything is installed under {{ quadlet_dir }} (~/.config/containers/systemd/). State lives under {{ host_directory }} (~/podman-data/valkey/).

How services attach themselves

Valkey joins the app network of every service that needs a cache. A service joins by writing Quadlet drop-ins next to valkey.container:

~/.config/containers/systemd/
├── valkey.container                    # base unit, no app networks
├── valkey.container.d/                 # ← drop-ins, one per client service
│   ├── 10-network-aliases.conf         # [Container] NetworkAlias=redis (shared)
│   ├── 50-nextcloud.conf               # [Container] Network=nextcloud.network
│   └── 50-affine.conf
├── nextcloud.network
└── affine.network

A service playbook does both halves — its own network, then the attach:

- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

- name: Attach this service to Valkey
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../valkey/roles/attach/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

The play must also run the podman role (for podman_user and quadlet_dir) and define a Restart Valkey handler — see roles/valkey/handlers/main.yaml for the exact snippet. nextcloud is the first service to do this and is the working example.

Order matters in one direction only: the attach include needs <name>.network to already exist, which is why Caddy's register comes first. It fails with a readable message if it does not, because a drop-in naming a missing .network file makes Quadlet reject the whole valkey unit with an error that never mentions the drop-in.

Names: valkey and redis

Both resolve on every attached network. valkey comes from ContainerName=valkey; redis is added by 10-network-aliases.conf, because the clients are not consistent: Nextcloud's config.php (written by the image from REDIS_HOST=redis), outline and overleaf connect to redis, while paperless-ngx and yamtrack connect to valkey — the container name — because that is the host their REDIS_* URL already had under Docker.

NetworkAlias= applies to every network the container joins, which is exactly what a hub wants, and it lives in one shared drop-in with a fixed value for the same reasons as Postgres's db alias — see postgres/readme.md, "Names: postgres and db".

One way to attach, and one rule

roles/attach is the shared include, and is what every service uses. A service may instead write the same two files from its own role, but then it must write identical bytes on purpose: the drop-ins live in one directory, so anything less means whichever play runs last rewrites the other's file on every run. If you add a second form, make it match.

Whichever form you use, flush handlers before the app starts on a first attach. The restart that joins Valkey to the new network is a handler, so by default it runs at the end of the play — after the app has already come up and failed to resolve redis. nextcloud/roles/valkey is the working example.

The trade-off

A new drop-in changes the generated unit, so Valkey is recreated to join the network. That is cheaper here than Postgres's restart: the recreate is a graceful stop, valkey-server writes a final dump.rdb on SIGTERM, and the new container loads it back from the bind mount. Attached apps keep their keys across the restart — what they lose is the seconds the container is down. Only a hard kill or a power loss costs the changes since the last save point.

Persistence is on deliberately, and host_directory stays even though everything in it is disposable. Two clients do not use Valkey as a pure cache: Nextcloud keeps its PHP sessions and memcache.locking there, so an eviction logs every active user out and drops their file locks, and Yamtrack uses it as its Celery broker, so an eviction discards queued tasks. The stateless alternative — Tmpfs=/data with save "" — is the right shape for a cache and would take UserNS=keep-id and the whole uid/gid mapping with it, but it would make every redeploy exactly that logout and dropped queue. Not copying the old anonymous volume (below) is a separate question: a cold start is survivable, it is just not free.

Who runs as whom

UserNS=keep-id:uid=999,gid=1000, which are the two ids the server writes as: the image's entrypoint starts as root, chowns the owner of anything in its WORKDIR that is not already valkey's, and drops to the account with setpriv. The uid is 999 — the same uid the old anonymous volume was owned by (docs/podman-compat-audit.md §3.2), so the mapping matches what was already on disk — but the gid is 1000, not 999: Alpine already has a gid 999, which is why the image's Dockerfile creates the valkey group with the next id. Mapping the uid alone would leave every file valkey-server writes group-owned by a subgid.

keep-id maps both ids onto podman_user, which means the data root — and everything in it — is owned by tanshu:tanshu rather than by a subuid or a subgid. roles/valkey creates ~/podman-data/valkey up front for the same reason Postgres's role creates its version directory: a directory created by container-root under keep-id is stranded as e.g. 165536, readable but not removable by the host user.

Like Caddy, podman_user is derived from the connection (ansible_facts['user_id']) and the play carries no become. This playbook does not run the podman role, so it has no guard against connecting as root; the prerequisites are applied separately (see Running).

Settings

Variable	Purpose
podman_user, podman_data_dir, quadlet_dir	Rootless-Podman contract, duplicated from caddy/vars/default.yml
container_name	Container / unit name, and the prefix of the drop-in directory
image	docker.io/valkey/valkey:alpine, fully qualified on purpose
app_uid / app_gid	The ids the server writes as — 999 / 1000 in the Alpine image
host_directory	Where state lives (~/podman-data/valkey)
data_mount	The image's WORKDIR and VOLUME — /data
container_port	6379, the port inside the container clients connect to
valkey_dropin_dir	Where services attach their networks

-alpine floats within the image's tag for security patches; there is no pinned digest, the same as the other service playbooks.

No published port

There is deliberately no PublishPort. The Docker-era playbook published 127.0.0.1:6379:6379, which meant anything that could reach the loopback interface on the host could read and flush every app's cache. Clients reach it over the app networks now, so there is no host port at all. Verify with ss -tlnp | grep 6379 — it should print nothing.

Running

The play targets emily directly (hosts: emily):

ansible-playbook -u tanshu valkey/playbook.yml

The inventory that defines emily lives in prepare-server/, whose ansible.cfg names it — run from there, or pass an inventory explicitly.

This play applies only the Valkey unit and its data directory. The rootless-Podman prerequisites — installing podman, loginctl enable-linger, the low-port sysctl and the Quadlet directory — are the shared task file at caddy/roles/podman/tasks/main.yaml, and are not run here. A host that has already run caddy/playbook.yml has them.

systemctl --user enable valkey is neither needed nor possible: the unit is generated, and Quadlet applies [Install] WantedBy=default.target itself.

Deploy this before any service that attaches to it. The attach include only writes drop-ins next to valkey.container; if the unit has never been deployed, the first Restart Valkey handler fails against a unit that does not exist. The prerequisite order is caddy → postgres → valkey → the service.

Migrating from the Docker version

There is nothing to migrate. The Docker-era Valkey kept /data in an anonymous volume — the one the audit flagged because no backup script could see it — and everything in it is disposable: it holds the RDB snapshot and, if it was ever enabled, the AOF. Starting empty costs one cold cache, with the same user-visible edges as any eviction (The trade-off): Nextcloud sessions gone and queued Celery tasks dropped. Nothing that has to be carried across.

That is the opposite of hiwikijs, the other anonymous volume, whose content is irreplaceable (docs/podman-compat-audit.md §3.1).

# Optional: look at what the old container had, before Docker goes away.
ssh monoco 'docker inspect valkey --format "{{range .Mounts}}{{.Name}} {{.Destination}}{{end}}"'
ssh monoco 'docker volume ls'

# Nothing is copied to emily. The volume is abandoned deliberately, and can be
# removed once the old host is being decommissioned:
# ssh monoco 'docker volume rm <name from the line above>'

Do not start the old Valkey container again once apps are on emily: it would serve a second, divergent cache.

Operational notes

systemctl --user status valkey
systemctl --user restart valkey
journalctl --user -u valkey -f
podman logs valkey
podman exec -it valkey valkey-cli
podman exec valkey valkey-cli ping
podman exec valkey valkey-cli info keyspace
podman inspect valkey --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'

The play waits on valkey-cli ping inside the container rather than trusting systemd. Restart=always means the unit reports active even while the container crash-loops, so a wrong keep-id (an entrypoint permission error, over and over) would otherwise look like success.

Debugging a unit that will not generate

/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
systemd-analyze --user --generators=true verify valkey.service

The most common cause is a Network= naming a .network file that does not exist — which is why attach checks first.

Verification

• systemctl --user status valkey is active, and stays active after a reboot
• podman exec valkey valkey-cli ping prints PONG
• ss -tlnp | grep 6379 on the host prints nothing
• ls -ln ~/podman-data/valkey shows tanshu as owner and group, not a 165536+ subuid/subgid
• From each client app's container: getent hosts valkey and getent hosts redis
• Each attaching app can read and write through the cache — in Nextcloud that means an upload, which exercises memcache.locking and the PHP session handler the image configures from REDIS_HOST
• A SAVE produces dump.rdb under ~/podman-data/valkey, owned by tanshu:tanshu — the check that catches a wrong app_gid