playbooks/docs/backup-plan.md

# Full backup plan for `monoco`

Status: **plan, not yet executed**
Audited: live host, read-only. No changes made.
Host: EC2 `i-08edfd71ce301b8ba`, `ap-south-1c`, Ubuntu 24.04.5 aarch64

---

## 1. Objective

A complete, **verified** backup of monoco that:

1. Can restore the entire host (disaster recovery), and
2. Can restore any single service in isolation (the important case day-to-day), and
3. Is a hard prerequisite for the Docker → Podman migration, because that migration
   changes file ownership on every data directory.

"Verified" is doing real work in that sentence. §10 defines what verification means,
and it is not optional.

---

## 2. What is actually on the box

| | |
|---|---|
| Provider | **EC2 confirmed** (IMDSv2), `i-08edfd71ce301b8ba`, `ap-south-1c` |
| Storage | **Single 80 GB EBS root volume** (`nvme0n1p1`), ext4, **no LVM**, no separate data volume |
| Free space | **30 GB** of 77 GB |
| Containers | 26, all plain `docker run`, all state on host bind mounts |
| Total app data | **~19 GB** |
| **Irreplaceable data** | **~2.4 GB** |

That last line is the most important number in this document. The gap between 19 GB
and 2.4 GB is almost entirely one thing:

```
/var/lib/audiobookshelf/audiobooks   15.0 GB   696 files. Re-downloadable media.
/var/lib/audiobookshelf/config        2.6 MB   ← absdatabase.sqlite, irreplaceable
/var/lib/audiobookshelf/metadata       19 MB   ← irreplaceable
```

So the real state of Audiobookshelf is **22 MB, not 15 GB**. Treating it as 15 GB is
what makes a "full backup" feel impossible; it isn't. Decide explicitly whether the
media is in scope (§8) — but do not let it drive the design.

### 2.1 What is irreplaceable, by service

| Service | Total | Back up | Skip (regenerable / from image) |
|---|---|---|---|
| `postgres` | 995 MB | 13 databases, ~890 MB raw | — |
| `gitea` | 747 MB | `git/` 302 MB + `gitea/` 445 MB **incl. gitea.db** | — |
| `nextcloud` | 1.2 GB | `data/` 324 MB, `config/` (config.php!), `custom_apps/` 5.8 MB | `apps/` 522 MB, `core/` 159 MB, `dist/` 83 MB, `3rdparty/` 55 MB, `lib/` 19 MB |
| `mozimo` | 147 MB | `pb_data/` (incl. `auxiliary.db` 18.6 MB + WAL) | `types.d.ts` |
| `mozimo-staging` | 137 MB | `pb_data/` | |
| `greatbear` | 101 MB | `pb_data/` | |
| `tandoor` | 98 MB | `mediafiles/` | `staticfiles/`, `externalfiles/` |
| `menu-app.*` ×3 | 210 MB | `/data` each | |
| `trillium` | 106 MB | `document.db` 3.8 MB + `backup/` 12 MB | `log/` 91 MB |
| `sms-wa-bridge` | 397 MB | `rules.json` (tiny) | `wwebjs_auth/` 307 MB — re-linkable via QR |
| `hops` | 19 MB | `pb_data/` | |
| `audiobookshelf` | 15 GB | `config/` + `metadata/` = **22 MB** | `audiobooks/` 15 GB |
| `bitwarden` | 4.2 MB | **Postgres `bitwarden` DB** | `data/db.sqlite3` — **stale**, see §3.3 |
| `yamtrack` | 2.3 MB | `data/db.sqlite3` | |
| `affine` | 916 KB | `config/` + `storage/` | |
| `caddy` | 736 KB | **all of it** — ACME account + certs | |
| `hiwikijs` | ? | **anonymous volume** — see §13.1 | `/var/lib/hiwikijs` is an empty decoy |
| `valkey` | ? | — | cache, rebuildable |

**~2.4 GB** once media and the WhatsApp session are excluded.

---

## 3. The consistency problem — read this before writing any backup

This is the part that silently produces useless backups. Every one of these is being
written to *right now* while the container runs.

### 3.1 SQLite in WAL mode

Live WAL sidecars are present on disk, which proves the databases are hot:

```
/var/lib/mozimo/pb_data/auxiliary.db        18,632,704   + -wal 1,116,552  + -shm
/var/lib/mozimo/pb_data/data.db                204,800   + -wal    78,312  + -shm
/var/lib/gitea/gitea/gitea.db                8,147,968   (modified 06:07 today)
/var/lib/audiobookshelf/config/absdatabase.sqlite  2,576,384
/var/lib/trillium/document.db                3,800,000   + -shm
/var/lib/yamtrack/data/db.sqlite3               + -wal  + -shm
```

**A plain `tar` of a live SQLite database can produce a backup that will not open.**
The main file and the WAL are copied at different instants, so the result can be torn.
This affects `gitea-backup.sh` today.

Three acceptable methods, in order of preference:

1. **Use the application's own dump command** — always best, because the app
   understands its own invariants:
   - Gitea: `docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini`
     → one zip containing the DB, repos, and config.
   - PocketBase (`mozimo`, `mozimo-staging`, `greatbear`, `hops`): use the built-in
     backup, or stop the container for the copy.
2. **`sqlite3 <db> ".backup <out>"`** — safe on a live database, needs `sqlite3`
   (present on the host: `~/.sqlite_history` exists).
3. **Stop the container, copy the directory, start it.** Simple, provable, costs
   seconds of downtime per service. For a pre-migration backup this is the right
   default, because the containers are being stopped anyway.

### 3.2 Postgres

13 databases, ~890 MB raw. A hot `tar` of `/var/lib/postgresql/data` is **not** a
valid backup — you need a logical dump or a proper physical base backup.

```
affine 11 MB | bitwarden 10 MB | brewman_acc 174 MB | brewman_exp 16 MB
brewman_hinchco 41 MB | brewman_hops 122 MB | brewman_mhl 132 MB | hiwikijs 16 MB
medusa-criollo 21 MB | nextcloud 319 MB | postgres 7.5 MB | tandoor 16 MB
zitadel 9.3 MB
```

Note two orphans with no running container — decide whether they are in scope:
**`medusa-criollo`** (21 MB, no container references it) and **`zitadel`** (9.3 MB,
container deleted). Everything else maps to a live service.

Method: `pg_dump -Fc` per database (parallel-safe, individually restorable) **plus**
`pg_dumpall --roles-only` for roles/globals, which per-database dumps do not capture.

### 3.3 The bitwarden trap

`bitwarden` has `DATABASE_URL=postgresql://…@postgres:5432/bitwarden`, and a 10 MB
`bitwarden` database exists in Postgres. **Postgres is authoritative.**

But `/var/lib/bitwarden/data/db.sqlite3` still exists with `-wal`/`-shm` sidecars, and
`~/bin/bitwarden-backup.sh` dumps the Postgres DB while `bitwarden-upload.sh` uploads a
`~/Data/warden/production.sqlite3` that **does not exist** (and `~/bin/gdrive` is not
installed).

So the current bitwarden backup story is three-quarters broken, and there is a stale
SQLite file that would look like a plausible restore source and silently lose months of
data. **Back up the Postgres database. Explicitly mark or delete the stale SQLite.**

### 3.4 Nextcloud

Files can be copied live; the **database cannot**. Dump the DB and the files together
so they correspond. Nextcloud's own guidance is maintenance mode during the backup:
`occ maintenance:mode --on`, dump, copy, `--off`. `config/config.php` must be included —
without it the instance is not restorable.

---

## 4. Backup architecture

Standard 3-2-1, using what this host actually has:

| Layer | What | Why | Where |
|---|---|---|---|
| **1. EBS snapshot** | Whole 80 GB volume | Instant whole-system restore; the only thing that captures the OS, Docker images, and 15 GB of media in one shot | AWS, S3-backed, inherently offsite |
| **2. Logical backup (restic)** | The ~2.4 GB of real state | Provably restorable, per-service, deduplicated, **encrypted** (matters — see §5) | S3 bucket in `ap-south-1` |
| **3. Local staging** | Transient only | Working space during the run | `/var/backups/` — **not** a backup |

Why both 1 and 2: an EBS snapshot is *crash-consistent*, not application-consistent —
it is equivalent to pulling the power cord. Restoring it means Postgres and every
SQLite database go through crash recovery. It is excellent for "the instance died" and
poor for "I need last night's database cleanly". The logical layer is the one that
gives you a *verified* restore.

**Region note:** an EBS snapshot in `ap-south-1` does not survive the loss of that
region or that AWS account. Enable **cross-region snapshot copy** (or at minimum
replicate the restic repo to a second bucket) before calling this disaster recovery.

---

## 5. Phase 0 — freeze the definitions (no downtime, minutes)

Do this first: it is tiny, has zero impact, and it is the part that is easiest to
forget and most annoying to reconstruct.

- **Every container's full config** — `docker inspect` for all containers, networks,
  volumes → JSON. This is the source of truth for the migration.
- **Secrets and configuration.** The env vars hold live credentials: Shopify client
  secret, Sentry auth token, PocketBase superuser passwords, JWT private keys,
  `MIDDLEWARE_SECRET_KEY`, DB passwords, `AUTH_SECRET`, Vince admin password. Capture
  them all — and note they are why the backup repository **must be encrypted at rest**.
- **Caddy** — the Caddyfile *and* `/var/lib/caddy/data` (ACME account key + ~20 issued
  certificates). Losing the account key means re-issuing everything against Let's
  Encrypt rate limits.
- **Deployment scripts** — everything in `~/bin/`, both crontabs, `~/.zshrc`.
- **Package and image inventory** — `dpkg --get-selections`, `docker images --digests`,
  so the host can be rebuilt from scratch if it must be.
- **DNS records** for the ~20 hostnames pointing at `13.202.6.2`.

None of this needs the services stopped.

---

## 6. Phase 1 — EBS snapshot (no downtime)

```
aws ec2 create-snapshot \
  --volume-id <vol-id from nvme0n1> \
  --description "monoco pre-podman-migration $(date -Iseconds)"
```

Then:
- Tag it with the date, purpose, and the fact that it **predates the Podman migration**.
- Wait for `State: completed` (not just `pending`). A pending snapshot is not a backup.
- **Copy it to a second region** if it is meant to survive regional loss.

This is your fastest rollback if the migration goes badly wrong. `awscli` is not
installed — install it or take the snapshot from the console.

---

## 7. Phase 2 — the logical backup (the one that matters)

Container-native, so it works identically before and after the Podman migration —
write it this way and you do not have to rewrite it during the cutover.

### 7.1 Postgres (~890 MB → est. 200–300 MB compressed)

```
# roles and globals first — per-DB dumps do NOT include these
docker exec postgres pg_dumpall -U postgres --roles-only | gzip > roles.sql.gz

# one restorable file per database
for db in affine bitwarden brewman_acc brewman_exp brewman_hinchco \
          brewman_hops brewman_mhl hiwikijs nextcloud tandoor; do
  docker exec postgres pg_dump -U postgres -Fc "$db" > "$db.dump"
done
```

Deliberately excluded: `postgres` (system), and the `medusa-criollo` / `zitadel`
orphans — unless you decide otherwise (§3.2).

### 7.2 Gitea — use its own dump

```
docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini -f /data/dumps
```

One archive with `gitea.db`, the repositories, and config, produced consistently.
A `tar` of `/var/lib/gitea` will *probably* work and will occasionally produce a
corrupt `gitea.db` (§3.1).

### 7.3 SQLite services — stop, copy, start

Simplest provably-correct method, and these are fast to restart:

```
for c in mozimo mozimo-staging greatbear hops trillium yamtrack audiobookshelf; do
  docker stop "$c"
done
# tar the relevant paths (§2.1 — skip audiobooks/, log/, staticfiles/)
for c in ...; do docker start "$c"; done
```

Alternatives if downtime is unacceptable: `sqlite3 "$db" ".backup"` per database, or
PocketBase's built-in backup for the four `pb_data` services.

### 7.4 Nextcloud

```
docker exec -u www-data nextcloud php occ maintenance:mode --on
docker exec postgres pg_dump -U postgres -Fc nextcloud > nextcloud.dump
tar -czf nextcloud-files.tar.gz \
  -C /var/lib/nextcloud data config custom_apps
docker exec -u www-data nextcloud php occ maintenance:mode --off
```

### 7.5 Caddy, affine, sms-wa-bridge, menu apps, tandoor media

Plain file copies — none of these need a quiesce:
`/var/lib/caddy` (all 736 KB), `/var/lib/affine/{config,storage}`,
`/var/lib/sms-wa-bridge/rules.json`, each `/var/lib/menu.*`, 
`/var/lib/tandoor/mediafiles`.

### 7.6 The anonymous volumes

`hiwikijs` and `valkey` are **not** under `/var/lib` — they live in
`/var/lib/docker/volumes/<hash>/_data` and are invisible to every existing script.
Export them to real directories before the migration (the migration plan requires it
anyway):

```
docker run --rm -v 9c1e3fde…:/from -v /var/backups/vol:/to alpine \
  sh -c 'cp -a /from/. /to/hiwikijs/'
```

`hiwikijs` matters; `valkey` is a rebuildable cache.

### 7.7 Package it

Install **restic** (static aarch64 binary — nothing is installed today), initialise an
encrypted repo on S3, and back up the staging directory:

```
restic -r s3:s3.ap-south-1.amazonaws.com/monoco-backup init
restic -r s3:… backup /var/backups/monoco-$(date +%F) --tag pre-podman-migration
restic -r s3:… check
```

restic gives you encryption, deduplication (so daily runs are cheap), and snapshots you
can mount and restore individual files from. No credentials are configured on this host
today, so this needs an IAM role on the instance or an access key placed deliberately.

---

## 8. Phase 3 — bulk media (decision required)

The 15 GB of audiobooks and the 307 MB WhatsApp session.

- **If the audiobooks exist elsewhere** (purchased library, cloud, seedbox), exclude
  them. They are already covered by the EBS snapshot for the "instance died" case, and
  including them inflates every backup run for no recovery benefit.
- **If they exist only here**, they must be in scope — but do not put them in restic
  daily. Handle them as a separate, infrequent, one-way copy.

This decision changes the backup from 2.4 GB to 18 GB. Make it explicitly.

---

## 9. Where the backup physically goes

There is **30 GB free** on a single volume. A backup written there is not a backup: it
shares the failure domain with the thing it protects, and a 19 GB archive would fill
the disk and take production down.

| Destination | Status |
|---|---|
| S3 `ap-south-1` (restic) | **Recommended.** Same region = fast and cheap; add cross-region for DR |
| EBS snapshot | Recommended, whole-system |
| Google Drive via `~/bin/gdrive` | **Broken** — binary not installed, `~/Data/warden` does not exist |
| Another server | No SSH hosts configured on the box; `registry.tanshu.com` (3.109.171.183) is a candidate if it has space |
| Local disk | Staging only |

---

## 10. Verification — acceptance criteria

A backup that has never been restored is a hypothesis. Before the migration proceeds:

- [ ] `restic check --read-data-subset=5%` passes
- [ ] **Every Postgres dump restores** into a scratch database, and a row count on a
      known table matches production
- [ ] `roles.sql.gz` restores (roles/globals are the classic omission)
- [ ] The Gitea dump extracts and `gitea.db` opens with `sqlite3 … "PRAGMA integrity_check"`
- [ ] One PocketBase `data.db` restores and opens
- [ ] `trillium/document.db` and `yamtrack/db.sqlite3` open
- [ ] Nextcloud restores **with** `config/config.php`, in a scratch instance
- [ ] `/var/lib/caddy/data` contains the ACME account key and the certificate set
- [ ] The EBS snapshot reaches `completed`, and is **booted as a test instance** with
      services confirmed running
- [ ] A manifest of checksums exists for the archive set
- [ ] **Original uid:gid ownership of every `/var/lib` path is recorded** — required to
      undo the migration's `chown` (§14)

---

## 11. Ongoing backups — currently there are none

This is the biggest single finding: **nothing is scheduled.**

- The only cron entry on the host is `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
- `tanshu`, `ubuntu`, and `git` have no crontab.
- No systemd timer mentions backup (only `dpkg-db-backup.timer`).
- No restic, borg, rclone, duplicity, or aws CLI is installed.
- The five archives in `~/` are **manual one-offs** — Jun 2025, Feb 2026, Sep 2026,
  Oct 2026 — for bitwarden, brewman_hops (SQL only), and nextcloud, plus obsolete
  `mongodb.tar` and `overleaf.tar`.

So backups depend on someone remembering. Recommended replacement — a `restic` run on a
systemd timer, with all the container-native dump steps from §7 baked into a script,
retention of **7 daily / 4 weekly / 6 monthly**, and **failure alerting**. A backup that
fails silently every night for three months is indistinguishable from no backup, which
is exactly the state today.

---

## 12. Retention

| Layer | Retention |
|---|---|
| Logical (restic) | 7 daily, 4 weekly, 6 monthly |
| Weekly full | 3 months |
| Monthly full | 12 months |
| EBS snapshots | Keep the pre-migration one indefinitely; daily snapshots 7–14 days |

---

## 13. Bugs in the existing backup scripts

Worth fixing regardless of the migration, because these are actively misleading:

**13.1 `hiwikijs-backup.sh` backs up nothing.** It runs
`tar -C /var/lib hiwikijs`, but `/var/lib/hiwikijs` is an **empty directory**
(`drwxr-xr-x www-data`, dated Dec 2022). The real Wiki.js content is in the anonymous
volume `9c1e3fde…`. The script exits successfully and produces a valid archive of
nothing. Anyone trusting it has no Wiki.js backup at all.

**13.2 `gitea-backup.sh` hot-copies a live SQLite database** (§3.1). It will usually
work, which is the worst property a backup can have. Replace with `gitea dump`.

**13.3 `bitwarden-backup.sh` + `bitwarden-upload.sh` are inconsistent with each other
and with reality** (§3.3) — one dumps Postgres, the other uploads a SQLite file that
does not exist, and neither runs on a schedule.

**13.4 `iot-backup.sh` and `kakawa-backup.sh` target services that no longer exist**
(the `iot_net`/`kakawa_net` Docker networks are still attached to Caddy, but the
containers are gone).

**13.5 All DB scripts use `--link postgres:db`**, which does not exist in Podman and
will break at cutover. Under rootless Podman the `--user $(id -u)` and `sudo chown`
gymnastics are also unnecessary once the data is owned by you.

**13.6 No coverage at all** for: `mozimo`, `mozimo-staging`, `greatbear`, `hops`,
the three `menu-app.*` containers, `affine`, `vince`, `audiobookshelf`, `valkey`, or
`/var/lib/caddy` (the certificates).

---

## 14. Pre-migration specifics

Beyond a normal full backup, the migration needs these captured, because it changes
ownership and layout:

1. **Original `uid:gid` and mode of every bind-mount source** — the migration's
   `chown -R tanshu:tanshu` is not trivially reversible without this table.
2. **The two anonymous volumes exported** (§7.6) — otherwise `hiwikijs` state is lost
   permanently when Docker is removed.
3. **`docker inspect` JSON for all 26 containers** — the only complete record of the
   runtime configuration.
4. **A restore rehearsal** onto a scratch host or the booted EBS snapshot, following the
   migration plan's rollback path.

---

## 15. Risks

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Backups silently produce corrupt SQLite | **High** | Data loss discovered at restore time | §3.1 methods; integrity-check every DB in verification |
| `hiwikijs` backed up as an empty directory | **Certain today** | Permanent wiki loss | Export the anonymous volume (§7.6) |
| Postgres roles/globals omitted | Medium | Restore fails on permissions | `pg_dumpall --roles-only` (§7.1) |
| Stale bitwarden SQLite restored instead of Postgres | Medium | Months of vault data lost | §3.3 — mark or delete it |
| Let's Encrypt account key lost | Low | ~20 certs blocked by rate limits | Include all of `/var/lib/caddy` |
| Backup written to the same volume | Medium | Both lost together | S3/Snapshot only (§9) |
| 30 GB free filled by staging | Medium | Production outage during backup | Stage only the ~2.4 GB of real state; media excluded |
| Snapshot left in `pending` | Low | Restore from an incomplete snapshot | Wait for `completed`; boot-test it |
| Same-region snapshot as sole DR | Medium | Region/account loss unrecoverable | Cross-region copy |
| No alerting on backup failure | **High** | Months of silent failure | Monitoring on the timer (§11) |

---

## 16. Open questions

1. **Is the 15 GB of audiobook media in scope?** (§8) This decides 2.4 GB vs 18 GB.
2. **Where do backups go?** S3 in `ap-south-1` is the natural fit — is there an IAM role
   on the instance, or should a dedicated bucket + credentials be set up?
3. **Are `medusa-criollo` and `zitadel` databases dead?** If so, exclude them and drop
   them.
4. **The 1.3 MB `bitwarden-backup_202602261629.tar.gz`** — is the SQLite in it the
   pre-Postgres state? If so it is a historical artifact worth keeping deliberately,
   and the live `db.sqlite3` should be deleted so nobody restores the wrong thing.
5. **Is `registry.tanshu.com` a viable second backup target** (independent host, but
   same region and presumably same provider account)?
6. **Should the ongoing backup be restic, or plain `pg_dump` + `tar` on a timer?**
   restic is better but adds a component; the container-native dumps are needed either
   way.