playbooks/audiobookshelf/readme.md

# Audiobookshelf (rootless Podman + Quadlet)

Runs [Audiobookshelf](https://www.audiobookshelf.org/) on `emily` as a
**rootless Podman** container managed by a **systemd user unit** (Quadlet),
behind the shared [rootless Caddy](../caddy/readme.md). Replaces the plain
`docker run` container on `monoco`:

```command
docker run -d \
  --volume /var/lib/audiobookshelf/config:/config \
  --volume /var/lib/audiobookshelf/metadata:/metadata \
  --volume /var/lib/audiobookshelf/audiobooks:/audiobooks \
  --volume /var/lib/audiobookshelf/podcasts:/podcasts \
  --name=audiobookshelf \
  --env=TZ=Asia/Kolkata \
  --restart=unless-stopped \
  ghcr.io/advplyr/audiobookshelf:latest
```

Unlike [bitwarden/](../bitwarden/readme.md) and [tandoor/](../tandoor/readme.md),
**there is no database to attach.** Audiobookshelf's state is one SQLite file
plus files under `~/podman-data/audiobookshelf`, so this play has no `postgres`
role and nothing to restore from the shared cluster.

Two facts about the published image shape everything below, and both were
verified against the real image rather than read off the docs:

| Finding | Consequence |
|---|---|
| The image declares **no `VOLUME`** | The four `Volume=` lines in the unit are load-bearing, not decorative. Without them the app runs perfectly — and loses everything on the next recreate |
| The image ships **no `HEALTHCHECK`** | This play defines one (`HealthCmd=`) so it can gate on the app actually answering |

## Ordering

Run [caddy/playbook.yml](../caddy/playbook.yml) first. It is the reverse proxy,
owns the Caddyfile this site appends to, supplies the `caddy.service` handlers,
and is the one play that applies the rootless-Podman prerequisites (installing
podman, `loginctl enable-linger`, the low-port sysctl, the Quadlet directory).

There is no second prerequisite to wait for — no cluster to bring across, since
this app has no database.

This playbook does **not** move the library. Copy the data first (see
["Migrating the data"](#migrating-the-data-from-monoco)) so the first start finds
the real `config/`, rather than creating a fresh empty database next to it.

## Layout

```
audiobookshelf/
├── playbook.yml
├── vars/default.yml
└── roles/
    ├── audiobookshelf/   # the four data dirs, the Quadlet unit, start + verify
    └── caddy/            # appends this site's block to the shared Caddyfile
```

Quadlet units and state:

| On the host | What |
|---|---|
| `~/.config/containers/systemd/audiobookshelf.container` | the generated unit (Ansible-templated) |
| `~/.config/containers/systemd/audiobookshelf.network` | the app network, written by `caddy/roles/register` |
| `~/.config/containers/systemd/caddy.container.d/50-audiobookshelf.conf` | attaches Caddy to that network |
| `~/podman-data/audiobookshelf/config/` | `absdatabase.sqlite`, `migrations/` |
| `~/podman-data/audiobookshelf/metadata/` | `backups/`, `cache/`, `logs/`, `streams/`, cover art, item metadata |
| `~/podman-data/audiobookshelf/audiobooks/` | the book library (15 GB) |
| `~/podman-data/audiobookshelf/podcasts/` | downloaded podcast episodes |

## What is state, and what is not

Per [docs/backup-plan.md §2.1](../docs/backup-plan.md):

| Path | Keep? | Why |
|---|---|---|
| `config/` | **required** | `absdatabase.sqlite` — users, libraries, listening progress, bookmarks. 2.6 MB, irreplaceable |
| `metadata/` | **required** | cover art, item metadata, and `backups/`. 19 MB, irreplaceable |
| `audiobooks/` | media | 15 GB across 696 files. Re-downloadable, and [proposed for exclusion](../docs/backup-plan.md) from the backup ([§8](../docs/backup-plan.md)) |
| `podcasts/` | media | episodes the app downloads here. Unlike `audiobooks/`, this directory is **written to at runtime** |

So the real state is **22 MB, not 15 GB**. Treating the library as state is what
made a "full backup" of this service look impossible
([docs/backup-plan.md §2](../docs/backup-plan.md)).

`metadata/cache/`, `metadata/logs/` and `metadata/streams/` are regenerable —
transcode scratch space, the log, and the image cache. They live under
`metadata/` because that is where the app puts them, and they are small; the
migration copies them anyway rather than carving out exceptions.

**A live `absdatabase.sqlite` must not be `tar`-ed.** It is SQLite in WAL mode,
so the database file and its `-wal` sidecar are copied at different instants and
the result can be torn — a backup that will not open
([docs/backup-plan.md §3.1](../docs/backup-plan.md)). Use the app's own
**Settings → Backups** (which writes into `BACKUP_PATH`, defaulting to
`metadata/backups`), or stop the container and copy the directory.

## How it registers with Caddy

Caddy is the base unit and knows nothing about its upstreams. This playbook calls
the shared include in
[caddy/roles/register](../caddy/roles/register/tasks/main.yaml), which writes the
network unit **and** the drop-in that pulls it into Caddy:

```yaml
- name: Register Audiobookshelf's network with Caddy
  ansible.builtin.include_tasks: "{{ playbook_dir }}/../caddy/roles/register/tasks/main.yaml"
```

It runs in `pre_tasks`, **before** the app unit is started, because
`audiobookshelf.container` references `audiobookshelf.network`: Quadlet fails
*unit generation* — so `systemctl --user start` reports "unit not found" — when a
referenced `.network` file does not exist yet.

`network_name` is `audiobookshelf`, so the *file* is `audiobookshelf.network` and
the *Podman network* is `audiobookshelf_net` (`NetworkName=` in the network unit
pins it, matching the name the migration plan already assumed).

[roles/caddy](roles/caddy/tasks/main.yaml) appends the site block to the shared
Caddyfile with `blockinfile`, exactly as the other services do. Caddy handles the
WebSocket upgrade the app's live-socket features need, and imposes no
request-body limit, so there is nothing like the old nginx `client_max_body_size`
to configure.

**Nothing is published on the host.** The container listens on 80 and Caddy
reaches it as `audiobookshelf:80` over `audiobookshelf_net`, so this play needs
none of the port-floor work Caddy itself needs.

## Settings

| Variable | Value / purpose |
|---|---|
| `podman_user` | **Derived** from the connection (`ansible_facts['user_id']`), not set by hand |
| `podman_data_dir` | `~/podman-data` |
| `quadlet_dir` | `~/.config/containers/systemd` |
| `host` | `audiobooks.tanshu.com` — the site block Caddy serves |
| `container_name` | `audiobookshelf`, and the Quadlet unit / container name |
| `image` | `ghcr.io/advplyr/audiobookshelf:latest` — already fully qualified |
| `network_name` | `audiobookshelf` → `audiobookshelf.network` / `audiobookshelf_net` |
| `config_dir` / `metadata_dir` | `~/podman-data/audiobookshelf/{config,metadata}` → `/config`, `/metadata` |
| `audiobooks_dir` / `podcasts_dir` | `~/podman-data/audiobookshelf/{audiobooks,podcasts}` → `/audiobooks`, `/podcasts` |
| `container_port` | `80` — the image's `EXPOSE 80` / `ENV PORT=80` |
| `timezone` | `Asia/Kolkata`, the only setting the Docker-era container passed |
| `health_cmd` | The `HealthCmd=` the unit defines, since the image has none |
| `caddy_name` / `caddyfile_path` | The shared Caddy unit and its Caddyfile |

The three `podman_*` / `quadlet_dir` vars are **duplicated** from
[caddy/vars/default.yml](../caddy/vars/default.yml), as in every service
playbook. That is a deliberate trade: a play cannot include the Caddy vars file
wholesale, because it also sets `host_directory` (to the *Caddy* data directory)
and every service defines that for itself. Keep them in sync. Caddy's fourth,
`podman_unprivileged_port_start`, belongs to the podman tasks this play does not
run.

Like Caddy, `podman_user` is derived from the connection and the play carries no
`become`. Because the podman tasks are not run here, there is **no guard against
connecting as `root`**; the prerequisites, including that check, live in
`caddy/playbook.yml`.

`UserNS=keep-id:uid=0,gid=0` — the image has no `USER` instruction, so it runs as
container-root throughout and writes `/config` and `/metadata` as root
([docs/podman-compat-audit.md §3.2](../docs/podman-compat-audit.md)). Mapping
container root onto `podman_user` keeps `~/podman-data/audiobookshelf` owned by
`tanshu` and leaves no subuid-owned files to trip over backups. Upstream
explicitly does not support `PUID`/`PGID`, so this is the mechanism for the
ownership rather than an env var.

**There is no environment file here**, unlike `bitwarden/` and `tandoor/`. This
deployment holds no secrets: Audiobookshelf generates its own JWT signing key
inside `config/`, and `TZ` is not sensitive — so it goes on the unit as a plain
`Environment=` line, and the unit stays world-readable without exposing anything.

## Running

```command
ansible-playbook -u tanshu audiobookshelf/playbook.yml
```

The play targets `emily` directly, so the inventory must contain it — the one in
`prepare-server/` does, and its `ansible.cfg` names it.

This play applies **only** this app's units and data. Its `pre_tasks` run Caddy's
`register` include and nothing else. The rootless-Podman prerequisites are the
shared task file at
[caddy/roles/podman/tasks/main.yaml](../caddy/roles/podman/tasks/main.yaml) and
are **not** run here. A host that has already run `caddy/playbook.yml` has them.

`systemctl --user enable audiobookshelf` is neither needed nor possible: the unit
is *generated*, and Quadlet applies `[Install] WantedBy=default.target` itself.

## Migrating the data from monoco

Everything lives in `/var/lib/audiobookshelf` today, owned by **uid 0** (the
image runs as root). It moves to `~/podman-data/audiobookshelf`, and the whole
tree is ~15 GB — budget real time for the bulk copy
([docs/podman-migration.md §4.3](../docs/podman-migration.md)).

**Run all of this from the control node that holds both SSH aliases** — the
machine where `ssh monoco` and `ssh emily` both work. The two hosts cannot reach
each other here: `emily` is a fresh instance that knows nothing about `monoco`,
and the `monoco` alias exists only in the control node's SSH config. `rsync`
also cannot copy remote-to-remote, so the 15 GB is staged on the control node in
between — check it has the room before starting.

### 1. Look before you copy

```bash
# Sizes, numeric owners (`ls -ln`), and whether the tree is readable without
# sudo. The old container ran as root, so the files are uid 0.
ssh monoco 'du -sh /var/lib/audiobookshelf/*'
ssh monoco 'ls -ln /var/lib/audiobookshelf'
ssh monoco 'find /var/lib/audiobookshelf -type f ! -readable'   # empty = no sudo needed

# Space on the destination (it needs the full 15 GB) and on the control node
# (it holds a second copy until step 6 deletes it).
ssh emily 'df -h / | tail -1'
df -h /home | tail -1
```

If the `find` above prints anything, add `--rsync-path="sudo rsync"` to the
`monoco:` side of every `rsync` below so the transfer reads as root.

### 2. Create the destinations

```bash
ssh emily 'mkdir -p ~/podman-data/audiobookshelf/{config,metadata,audiobooks,podcasts}'
mkdir -p ~/audiobookshelf-migrate/{config,metadata,audiobooks,podcasts}
```

### 3. The 15 GB library, while the old container is still serving

Audiobookshelf only *reads* `/audiobooks`, so the slow part needs no downtime.
If a hop drops, run it again — `rsync -P` resumes at the file it was on.

```bash
# monoco -> control node
rsync -aP --no-owner --no-group \
  monoco:/var/lib/audiobookshelf/audiobooks/ ~/audiobookshelf-migrate/audiobooks/

# control node -> emily
rsync -aP --no-owner --no-group \
  ~/audiobookshelf-migrate/audiobooks/ emily:~/podman-data/audiobookshelf/audiobooks/
```

A second run of either command that prints **no filenames** means that hop is
complete — cheaper and stronger than comparing counts, and it is why `rsync` is
used here rather than a single `tar` stream. Do **not** add `--delete`: if the
playbook has already run on emily, `--delete` would remove files the app created
there that are not on monoco.

### 4. The 22 MB of state, with the old container stopped

```bash
# Not optional for config/: absdatabase.sqlite is live SQLite in WAL mode, and a
# hot copy can be torn (docs/backup-plan.md §3.1). podcasts/ is written at
# runtime too, so it is not safe to copy while the app is up either.
ssh monoco 'docker stop audiobookshelf'

for d in config metadata podcasts; do
  rsync -aP --no-owner --no-group "monoco:/var/lib/audiobookshelf/$d/" ~/audiobookshelf-migrate/"$d"/
  rsync -aP --no-owner --no-group ~/audiobookshelf-migrate/"$d"/ "emily:~/podman-data/audiobookshelf/$d/"
done

# Final incremental pass over the library, in case a scan wrote metadata beside
# a book during step 3.
rsync -aP --no-owner --no-group monoco:/var/lib/audiobookshelf/audiobooks/ ~/audiobookshelf-migrate/audiobooks/
rsync -aP --no-owner --no-group ~/audiobookshelf-migrate/audiobooks/ emily:~/podman-data/audiobookshelf/audiobooks/
```

`--no-owner` / `--no-group` are explicit because `-a` implies the opposite, and
the source files are **root**-owned on monoco. The container runs as uid 0
in-container, which `keep-id` maps to `tanshu`, so files that arrive owned by
`tanshu` are already correct — and restoring root ownership would be wrong.

### 5. Verify, before the first start

Verify *before* deploying: once the playbook has run, emily's
`config/absdatabase.sqlite` has been migrated in place and no longer matches the
source.

```bash
# File count and total bytes per directory, both ends. audiobooks/ should be
# 696 files and 15.0 GB on monoco (docs/backup-plan.md §2).
for d in config metadata audiobooks podcasts; do
  echo "== $d =="
  ssh monoco "find /var/lib/audiobookshelf/$d -type f | wc -l; du -sb /var/lib/audiobookshelf/$d | cut -f1"
  ssh emily  "find ~/podman-data/audiobookshelf/$d -type f | wc -l; du -sb ~/podman-data/audiobookshelf/$d | cut -f1"
done

# Byte-level digests for the irreplaceable 22 MB. Hashing the 15 GB library costs
# more than copying it did, so it is checked by count and size above instead.
for d in config metadata; do
  a=$(ssh monoco "cd /var/lib/audiobookshelf/$d && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum")
  b=$(ssh emily  "cd ~/podman-data/audiobookshelf/$d && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum")
  [ -n "$a" ] && [ "$a" = "$b" ] && echo "$d OK ${a%% *}" || { echo "$d MISMATCH: $a vs $b"; false; }
done

# Nothing may be owned by root or by a subuid.
ssh emily 'find ~/podman-data/audiobookshelf ! -user tanshu'   # empty = correct
```

### 6. Deploy, then clean up

```bash
# Run after the copy, so the first start finds the real database rather than
# creating an empty one beside it.
cd prepare-server && ansible-playbook -u tanshu ../audiobookshelf/playbook.yml

# Only after step 5 passed and the play reported a healthy container. The
# staging copy is ~15 GB; the library is on emily twice over until this runs.
rm -rf ~/audiobookshelf-migrate
```

Stop and leave the old container stopped. Its `config/` is now a stale snapshot,
and two live copies would diverge — rolling back means restoring monoco's
`config/` and starting it there, not starting it in place.

### Alternatives

Stream straight across with `tar` if the control node has no room for 15 GB. It
cannot resume: a drop at 90% starts over.

```bash
set -o pipefail
ssh monoco 'tar -C /var/lib/audiobookshelf/audiobooks -cf - .' \
  | ssh emily 'tar --no-same-owner -C ~/podman-data/audiobookshelf/audiobooks -xf -'
```

`set -o pipefail` matters: without it the pipeline reports only the *receiving*
`tar`'s status, so a read error or a dropped stream on the source side exits 0
and leaves a silently short copy. The 22 MB of state is small enough that the
same pipe is fine for it.

Or stop the old container up front and copy all four directories in one pass.
Simpler, and provably consistent because nothing is writing; the cost is that the
service is down for the whole 15 GB instead of just the 22 MB.

```bash
ssh monoco 'docker stop audiobookshelf'
for d in config metadata audiobooks podcasts; do
  rsync -aP --no-owner --no-group "monoco:/var/lib/audiobookshelf/$d/" ~/audiobookshelf-migrate/"$d"/
  rsync -aP --no-owner --no-group ~/audiobookshelf-migrate/"$d"/ "emily:~/podman-data/audiobookshelf/$d/"
done
```

## Verification

```bash
systemctl --user status audiobookshelf
journalctl --user -u audiobookshelf -f
podman logs audiobookshelf

# What the playbook checks for you:
podman healthcheck run audiobookshelf       # the HealthCmd= the unit defines
podman exec audiobookshelf ls -d /config /metadata /audiobooks /podcasts
test -e ~/podman-data/audiobookshelf/config/absdatabase.sqlite

# The health the unit reports on its own 30s timer, not just once:
podman inspect audiobookshelf --format '{{.State.Health.Status}}'

# The mounts are real bind mounts and not the container layer:
podman inspect audiobookshelf --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'

# The site, end to end.
curl -sSI https://audiobooks.tanshu.com/healthcheck   # 200 OK, body "OK"
```

Then open `https://audiobooks.tanshu.com` **in a browser** and confirm the
library is there and an existing book resumes at the right timestamp. A `curl`
cannot check that the migrated database is *readable*; only the app can.

On a *fresh* install the first visit asks you to create the admin account and add
a library — the playbook creates the directories, not the library entries.

Also exercise a **write** through the app — ask it to match/scan a book, or start
a podcast download — and confirm the files it creates under
`~/podman-data/audiobookshelf` are owned by `tanshu`. That is the check that
catches a wrong `keep-id`; a read-only smoke test will not.

Then reboot the host and confirm the unit returns unattended — without
`loginctl enable-linger tanshu` it will not, and the failure is silent.

## Notes and gotchas

- **The image declares no `VOLUME`.** This is the one silent failure worth
  knowing: with a missing `Volume=` line the app starts, serves, and looks
  healthy while writing into the container's writable layer — and loses it all on
  the next recreate. The play guards against it by asserting that
  `config/absdatabase.sqlite` exists on the *host* after the health check passes.
- **The image ships no `HEALTHCHECK`, so this unit defines one.** Upstream's docs
  discourage health checks as "generally redundant to setting a restart policy"
  and as log noise; that is fair, and the reason for the deviation is that
  `is-active` cannot tell a serving server from a crash-looping one under
  `Restart=always`. There is deliberately **no `HealthOnFailure=`** — that would
  let a temporarily slow start kill and restart a healthy server. Add
  `HealthOnFailure=kill` yourself if you would rather have self-healing.
- **`ROUTER_BASE_PATH` defaults to `/audiobookshelf`.** It is undocumented in the
  env reference but real, and the server logs it on every start. Bare paths still
  work — middleware rewrites non-prefixed URLs — which is why
  `reverse_proxy audiobookshelf:80` at the site root is correct. It matters only
  if you later serve the app under a subpath, where the prefix would appear twice.
- **`wget`, not `curl`.** `curl` exists only in the image's builder stage and is
  absent at runtime; `wget` is present (BusyBox). That is what `health_cmd` uses.
- **`latest` floats.** `:latest` is the newest stable release; `:edge` tracks
  every commit to master, and `vX.Y.Z` tags exist. Upstream makes no pinning
  recommendation either way. The image measured for this playbook was **2.37.1**,
  digest `sha256:581d68b2a6fc7ebf58d81c878a9f387cbbc0d88ac9d37b298b9cee10168af85b`
  — pin that digest in `vars/default.yml` if you want reproducible deploys.
  Upgrading is `podman pull` plus a restart; it may run SQLite migrations on
  first start.
- **The 15 GB library is not backed up, and was never covered.** No backup script
  exists for this service at all ([docs/backup-plan.md §13.6](../docs/backup-plan.md)),
  and the media is a deliberate exclusion. Decide explicitly whether the
  audiobooks exist anywhere else ([§8](../docs/backup-plan.md)) — this playbook
  takes no position on it.
- **Registering restarts all of Caddy.** Writing the network unit and the drop-in
  changes Caddy's generated unit, and Quadlet has no live equivalent of
  `docker network connect`, so Caddy is recreated — a brief restart of *every*
  hosted site, not just this one. See
  [caddy/readme.md](../caddy/readme.md), "Trade-off worth knowing".
- **The scanner/watcher has no environment variable.** "Automatically watch
  libraries for changes" is a UI setting (`scannerDisableWatcher`, needs a
  restart), plus a per-library flag. It cannot be set from this playbook.
- **Client IPs.** Whether the original client address survives the rootless port
  forward decides whether Audiobookshelf's own auth rate limiting
  (`RATE_LIMIT_AUTH_*`) sees real addresses. It is unverified on emily — see
  [caddy/readme.md](../caddy/readme.md), "Client IPs".
- **If the Caddyfile does not exist yet**, this playbook creates it containing
  only the site block. That is a valid Caddyfile; the Caddy playbook's seeded
  global-options block is then not added, because its seed only fires on a
  missing or zero-length file. Deploy Caddy first on a fresh host.
- **DNS.** `audiobooks.tanshu.com` must resolve to `emily` before Caddy can
  obtain a certificate for it.
- **Root's crontab cannot `podman exec`** into another user's rootless
  containers, so any scheduled `Settings → Backups` orchestration or library
  scan belongs in a **systemd user timer** next to the Quadlet units.
- **Do not restart the container on monoco.** Its `config/` is a stale snapshot
  of a database emily has moved past.

## Not done here

- Pinning the image by digest or version tag, or wiring up
  `AutoUpdate=registry` with `podman auto-update`.
- Backing up the 15 GB library, or deciding whether it belongs in scope at all.
- Creating a backup script or systemd user timer for the 22 MB of real state —
  neither exists today, and the app's own **Settings → Backups** is the
  preferred route rather than anything Ansible-side.
- Creating the admin account and library entries, which is a first-run UI step.
- `HealthOnFailure=kill`, and any external monitoring or alerting on the
  `unhealthy` state.