playbooks/valkey/readme.md

# Valkey (rootless Podman + Quadlet)

Runs [Valkey](https://valkey.io/) (the Redis fork this repo already used) as a
**rootless Podman** container managed by a **systemd user unit** (Quadlet).
Replaces the `docker_container` playbook, which published `127.0.0.1:6379` and
kept its `/data` in an **anonymous Docker volume** — `docker volume ls` state
that every backup script in `~/bin/` missed entirely
([docs/podman-compat-audit.md §3.1](../docs/podman-compat-audit.md)).

**Valkey is the base unit and knows nothing about its clients.** It does not list
any app network and does not need editing when a service is added or removed.
Each service attaches *itself*. That is the same design as
[Postgres](../postgres/readme.md), and the mirror image of
[Caddy](../caddy/readme.md).

The name is split on purpose: the directory, container, unit and Podman network
are `valkey`, exactly as the Docker-era playbook had them, and only the *alias*
clients connect on is `redis`. Renaming the container would touch every app's
`REDIS_HOST` for no gain.

## Layout

```
valkey/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── valkey/     # the base valkey.container unit, data dir, handlers
    └── attach/     # includable task file: a service attaches itself to Valkey
```

Everything is installed under `{{ quadlet_dir }}`
(`~/.config/containers/systemd/`). State lives under `{{ host_directory }}`
(`~/podman-data/valkey/`).

## How services attach themselves

Valkey joins the app network of every service that needs a cache. A service joins
by writing Quadlet *drop-ins* next to `valkey.container`:

```
~/.config/containers/systemd/
├── valkey.container                    # base unit, no app networks
├── valkey.container.d/                 # ← drop-ins, one per client service
│   ├── 10-network-aliases.conf         # [Container] NetworkAlias=redis (shared)
│   ├── 50-nextcloud.conf               # [Container] Network=nextcloud.network
│   └── 50-affine.conf
├── nextcloud.network
└── affine.network
```

A service playbook does both halves — its own network, then the attach:

```yaml
- name: Register this service with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"

- name: Attach this service to Valkey
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../valkey/roles/attach/tasks/main.yaml"
  vars:
    network_name: "{{ container_name }}"
```

The play must also run the `podman` role (for `podman_user` and `quadlet_dir`)
and define a `Restart Valkey` handler — see
[roles/valkey/handlers/main.yaml](roles/valkey/handlers/main.yaml) for the exact
snippet. [nextcloud](../nextcloud/readme.md) is the first service to do this and
is the working example.

Order matters in one direction only: the `attach` include needs
`<name>.network` to already exist, which is why Caddy's `register` comes first.
It fails with a readable message if it does not, because a drop-in naming a
missing `.network` file makes Quadlet reject the *whole* valkey unit with an
error that never mentions the drop-in.

### Names: `valkey` and `redis`

Both resolve on every attached network. `valkey` comes from
`ContainerName=valkey`; `redis` is added by `10-network-aliases.conf`, because
the clients are not consistent: Nextcloud's `config.php` (written by the image
from `REDIS_HOST=redis`), outline and overleaf connect to `redis`, while
paperless-ngx and yamtrack connect to `valkey` — the container name — because
that is the host their `REDIS_*` URL already had under Docker.

`NetworkAlias=` applies to *every* network the container joins, which is exactly
what a hub wants, and it lives in one shared drop-in with a fixed value for the
same reasons as Postgres's `db` alias — see
[postgres/readme.md](../postgres/readme.md), "Names: `postgres` and `db`".

### One way to attach, and one rule

[roles/attach](roles/attach/tasks/main.yaml) is the shared include, and is what
every service uses. A service may instead write the same two files from its own
role, but then it **must write identical bytes** on purpose: the drop-ins live in
one directory, so anything less means whichever play runs last rewrites the
other's file on every run. If you add a second form, make it match.

Whichever form you use, **flush handlers before the app starts on a first
attach.** The restart that joins Valkey to the new network is a handler, so by
default it runs at the end of the play — after the app has already come up and
failed to resolve `redis`. `nextcloud/roles/valkey` is the working example.

### The trade-off

A new drop-in changes the generated unit, so Valkey is **recreated** to join the
network. That is cheaper here than Postgres's restart: the recreate is a graceful
stop, valkey-server writes a final `dump.rdb` on SIGTERM, and the new container
loads it back from the bind mount. Attached apps keep their keys across the
restart — what they lose is the seconds the container is down. Only a hard kill
or a power loss costs the changes since the last save point.

Persistence is on **deliberately**, and `host_directory` stays even though
everything in it is disposable. Two clients do not use Valkey as a pure cache:
Nextcloud keeps its PHP sessions and `memcache.locking` there, so an eviction
logs every active user out and drops their file locks, and Yamtrack uses it as
its Celery broker, so an eviction discards queued tasks. The stateless
alternative — `Tmpfs=/data` with `save ""` — is the right shape for a cache
and would take `UserNS=keep-id` and the whole uid/gid mapping with it, but it
would make every redeploy exactly that logout and dropped queue. Not copying the
old anonymous volume (below) is a separate question: a cold start is survivable,
it is just not free.

## Who runs as whom

`UserNS=keep-id:uid=999,gid=1000`, which are the two ids the *server* writes as:
the image's entrypoint starts as root, `chown`s the owner of anything in its
WORKDIR that is not already `valkey`'s, and drops to the account with `setpriv`.
The uid is `999` — the same uid the old anonymous volume was owned by
([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)), so the
mapping matches what was already on disk — but the gid is `1000`, not `999`:
Alpine already has a gid 999, which is why the image's Dockerfile creates the
`valkey` group with the next id. Mapping the uid alone would leave every file
`valkey-server` writes group-owned by a subgid.

`keep-id` maps both ids onto `podman_user`, which means the data root — and
everything in it — is owned by `tanshu:tanshu` rather than by a subuid or a
subgid. `roles/valkey` creates `~/podman-data/valkey` up front for the same
reason Postgres's role creates its version directory: a directory created by
container-root under `keep-id` is stranded as e.g. `165536`, readable but not
removable by the host user.

Like Caddy, `podman_user` is derived from the connection
(`ansible_facts['user_id']`) and the play carries no `become`. This playbook does
not run the podman role, so it has no guard against connecting as `root`; the
prerequisites are applied separately (see Running).

## Settings

| Variable | Purpose |
|---|---|
| `podman_user`, `podman_data_dir`, `quadlet_dir` | Rootless-Podman contract, duplicated from [caddy/vars/default.yml](../caddy/vars/default.yml) |
| `container_name` | Container / unit name, and the prefix of the drop-in directory |
| `image` | `docker.io/valkey/valkey:alpine`, fully qualified on purpose |
| `app_uid` / `app_gid` | The ids the server writes as — `999` / `1000` in the Alpine image |
| `host_directory` | Where state lives (`~/podman-data/valkey`) |
| `data_mount` | The image's WORKDIR and VOLUME — `/data` |
| `container_port` | `6379`, the port inside the container clients connect to |
| `valkey_dropin_dir` | Where services attach their networks |

`-alpine` floats within the image's tag for security patches; there is no pinned
digest, the same as the other service playbooks.

### No published port

There is deliberately no `PublishPort`. The Docker-era playbook published
`127.0.0.1:6379:6379`, which meant anything that could reach the loopback
interface on the host could read and flush every app's cache. Clients reach it
over the app networks now, so there is no host port at all. Verify with
`ss -tlnp | grep 6379` — it should print nothing.

## Running

The play targets `emily` directly (`hosts: emily`):

```command
ansible-playbook -u tanshu valkey/playbook.yml
```

The inventory that defines `emily` lives in `prepare-server/`, whose `ansible.cfg`
names it — run from there, or pass an inventory explicitly.

This play applies **only** the Valkey unit and its data directory. The
rootless-Podman prerequisites — installing podman, `loginctl enable-linger`, the
low-port sysctl and the Quadlet directory — are the shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml), and
are not run here. A host that has already run `caddy/playbook.yml` has them.

`systemctl --user enable valkey` is neither needed nor possible: the unit is
*generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

**Deploy this before any service that attaches to it.** The attach include only
writes drop-ins next to `valkey.container`; if the unit has never been deployed,
the first `Restart Valkey` handler fails against a unit that does not exist. The
prerequisite order is `caddy` → `postgres` → `valkey` → the service.

## Migrating from the Docker version

There is **nothing to migrate.** The Docker-era Valkey kept `/data` in an
anonymous volume — the one the audit flagged because no backup script could see
it — and everything in it is disposable: it holds the RDB snapshot and, if it was
ever enabled, the AOF. Starting empty costs one cold cache, with the same
user-visible edges as any eviction ([The trade-off](#the-trade-off)): Nextcloud
sessions gone and queued Celery tasks dropped. Nothing that has to be carried
across.

That is the opposite of `hiwikijs`, the other anonymous volume, whose content is
irreplaceable ([docs/podman-compat-audit.md §3.1](../docs/podman-compat-audit.md)).

```bash
# Optional: look at what the old container had, before Docker goes away.
ssh monoco 'docker inspect valkey --format "{{range .Mounts}}{{.Name}} {{.Destination}}{{end}}"'
ssh monoco 'docker volume ls'

# Nothing is copied to emily. The volume is abandoned deliberately, and can be
# removed once the old host is being decommissioned:
# ssh monoco 'docker volume rm <name from the line above>'
```

Do **not** start the old Valkey container again once apps are on `emily`: it would
serve a second, divergent cache.

## Operational notes

```
systemctl --user status valkey
systemctl --user restart valkey
journalctl --user -u valkey -f
podman logs valkey
podman exec -it valkey valkey-cli
podman exec valkey valkey-cli ping
podman exec valkey valkey-cli info keyspace
podman inspect valkey --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
```

The play waits on `valkey-cli ping` inside the container rather than trusting
systemd. `Restart=always` means the unit reports `active` even while the
container crash-loops, so a wrong `keep-id` (an entrypoint permission error, over
and over) would otherwise look like success.

### Debugging a unit that will not generate

```
/usr/lib/systemd/user-generators/podman-user-generator --user --dryrun
systemd-analyze --user --generators=true verify valkey.service
```

The most common cause is a `Network=` naming a `.network` file that does not
exist — which is why `attach` checks first.

## Verification

- [ ] `systemctl --user status valkey` is active, and stays active after a reboot
- [ ] `podman exec valkey valkey-cli ping` prints `PONG`
- [ ] `ss -tlnp | grep 6379` on the host prints nothing
- [ ] `ls -ln ~/podman-data/valkey` shows `tanshu` as owner **and** group, not a
      `165536+` subuid/subgid
- [ ] From each client app's container: `getent hosts valkey` and `getent hosts redis`
- [ ] Each attaching app can read **and** write through the cache — in Nextcloud
      that means an upload, which exercises `memcache.locking` and the PHP session
      handler the image configures from `REDIS_HOST`
- [ ] A `SAVE` produces `dump.rdb` under `~/podman-data/valkey`, owned by
      `tanshu:tanshu` — the check that catches a wrong `app_gid`