playbooks/docs/backup-plan.md

Full backup plan for monoco

Status: plan, not yet executed Audited: live host, read-only. No changes made. Host: EC2 i-08edfd71ce301b8ba, ap-south-1c, Ubuntu 24.04.5 aarch64

---

1. Objective

A complete, verified backup of monoco that:

1. Can restore the entire host (disaster recovery), and
2. Can restore any single service in isolation (the important case day-to-day), and
3. Is a hard prerequisite for the Docker → Podman migration, because that migration changes file ownership on every data directory.

"Verified" is doing real work in that sentence. §10 defines what verification means, and it is not optional.

---

2. What is actually on the box

Provider	EC2 confirmed (IMDSv2), i-08edfd71ce301b8ba, ap-south-1c
Storage	Single 80 GB EBS root volume (nvme0n1p1), ext4, no LVM, no separate data volume
Free space	30 GB of 77 GB
Containers	26, all plain docker run, all state on host bind mounts
Total app data	~19 GB
Irreplaceable data	~2.4 GB

That last line is the most important number in this document. The gap between 19 GB and 2.4 GB is almost entirely one thing:

/var/lib/audiobookshelf/audiobooks   15.0 GB   696 files. Re-downloadable media.
/var/lib/audiobookshelf/config        2.6 MB   ← absdatabase.sqlite, irreplaceable
/var/lib/audiobookshelf/metadata       19 MB   ← irreplaceable

So the real state of Audiobookshelf is 22 MB, not 15 GB. Treating it as 15 GB is what makes a "full backup" feel impossible; it isn't. Decide explicitly whether the media is in scope (§8) — but do not let it drive the design.

2.1 What is irreplaceable, by service

Service	Total	Back up	Skip (regenerable / from image)
postgres	995 MB	13 databases, ~890 MB raw	—
gitea	747 MB	git/ 302 MB + gitea/ 445 MB incl. gitea.db	—
nextcloud	1.2 GB	data/ 324 MB, config/ (config.php!), custom_apps/ 5.8 MB	apps/ 522 MB, core/ 159 MB, dist/ 83 MB, 3rdparty/ 55 MB, lib/ 19 MB
mozimo	147 MB	pb_data/ (incl. auxiliary.db 18.6 MB + WAL)	types.d.ts
mozimo-staging	137 MB	pb_data/
greatbear	101 MB	pb_data/
tandoor	98 MB	mediafiles/	staticfiles/, externalfiles/
menu-app.* ×3	210 MB	/data each
trillium	106 MB	document.db 3.8 MB + backup/ 12 MB	log/ 91 MB
sms-wa-bridge	397 MB	rules.json (tiny)	wwebjs_auth/ 307 MB — re-linkable via QR
hops	19 MB	pb_data/
audiobookshelf	15 GB	config/ + metadata/ = 22 MB	audiobooks/ 15 GB
bitwarden	4.2 MB	Postgres bitwarden DB	data/db.sqlite3 — stale, see §3.3
yamtrack	2.3 MB	data/db.sqlite3
affine	916 KB	config/ + storage/
caddy	736 KB	all of it — ACME account + certs
hiwikijs	?	anonymous volume — see §13.1	/var/lib/hiwikijs is an empty decoy
valkey	?	—	cache, rebuildable

~2.4 GB once media and the WhatsApp session are excluded.

---

3. The consistency problem — read this before writing any backup

This is the part that silently produces useless backups. Every one of these is being written to right now while the container runs.

3.1 SQLite in WAL mode

Live WAL sidecars are present on disk, which proves the databases are hot:

/var/lib/mozimo/pb_data/auxiliary.db        18,632,704   + -wal 1,116,552  + -shm
/var/lib/mozimo/pb_data/data.db                204,800   + -wal    78,312  + -shm
/var/lib/gitea/gitea/gitea.db                8,147,968   (modified 06:07 today)
/var/lib/audiobookshelf/config/absdatabase.sqlite  2,576,384
/var/lib/trillium/document.db                3,800,000   + -shm
/var/lib/yamtrack/data/db.sqlite3               + -wal  + -shm

A plain tar of a live SQLite database can produce a backup that will not open. The main file and the WAL are copied at different instants, so the result can be torn. This affects gitea-backup.sh today.

Three acceptable methods, in order of preference:

1. Use the application's own dump command — always best, because the app understands its own invariants:
   • Gitea: docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini → one zip containing the DB, repos, and config.
   • PocketBase (mozimo, mozimo-staging, greatbear, hops): use the built-in backup, or stop the container for the copy.
2. sqlite3 <db> ".backup <out>" — safe on a live database, needs sqlite3 (present on the host: ~/.sqlite_history exists).
3. Stop the container, copy the directory, start it. Simple, provable, costs seconds of downtime per service. For a pre-migration backup this is the right default, because the containers are being stopped anyway.

3.2 Postgres

13 databases, ~890 MB raw. A hot tar of /var/lib/postgresql/data is not a valid backup — you need a logical dump or a proper physical base backup.

affine 11 MB | bitwarden 10 MB | brewman_acc 174 MB | brewman_exp 16 MB
brewman_hinchco 41 MB | brewman_hops 122 MB | brewman_mhl 132 MB | hiwikijs 16 MB
medusa-criollo 21 MB | nextcloud 319 MB | postgres 7.5 MB | tandoor 16 MB
zitadel 9.3 MB

Note two orphans with no running container — decide whether they are in scope: medusa-criollo (21 MB, no container references it) and zitadel (9.3 MB, container deleted). Everything else maps to a live service.

Method: pg_dump -Fc per database (parallel-safe, individually restorable) plus pg_dumpall --roles-only for roles/globals, which per-database dumps do not capture.

3.3 The bitwarden trap

bitwarden has DATABASE_URL=postgresql://…@postgres:5432/bitwarden, and a 10 MB bitwarden database exists in Postgres. Postgres is authoritative.

But /var/lib/bitwarden/data/db.sqlite3 still exists with -wal/-shm sidecars, and ~/bin/bitwarden-backup.sh dumps the Postgres DB while bitwarden-upload.sh uploads a ~/Data/warden/production.sqlite3 that does not exist (and ~/bin/gdrive is not installed).

So the current bitwarden backup story is three-quarters broken, and there is a stale SQLite file that would look like a plausible restore source and silently lose months of data. Back up the Postgres database. Explicitly mark or delete the stale SQLite.

3.4 Nextcloud

Files can be copied live; the database cannot. Dump the DB and the files together so they correspond. Nextcloud's own guidance is maintenance mode during the backup: occ maintenance:mode --on, dump, copy, --off. config/config.php must be included — without it the instance is not restorable.

---

4. Backup architecture

Standard 3-2-1, using what this host actually has:

Layer	What	Why	Where
1. EBS snapshot	Whole 80 GB volume	Instant whole-system restore; the only thing that captures the OS, Docker images, and 15 GB of media in one shot	AWS, S3-backed, inherently offsite
2. Logical backup (restic)	The ~2.4 GB of real state	Provably restorable, per-service, deduplicated, encrypted (matters — see §5)	S3 bucket in ap-south-1
3. Local staging	Transient only	Working space during the run	/var/backups/ — not a backup

Why both 1 and 2: an EBS snapshot is crash-consistent, not application-consistent — it is equivalent to pulling the power cord. Restoring it means Postgres and every SQLite database go through crash recovery. It is excellent for "the instance died" and poor for "I need last night's database cleanly". The logical layer is the one that gives you a verified restore.

Region note: an EBS snapshot in ap-south-1 does not survive the loss of that region or that AWS account. Enable cross-region snapshot copy (or at minimum replicate the restic repo to a second bucket) before calling this disaster recovery.

---

5. Phase 0 — freeze the definitions (no downtime, minutes)

Do this first: it is tiny, has zero impact, and it is the part that is easiest to forget and most annoying to reconstruct.

• Every container's full config — docker inspect for all containers, networks, volumes → JSON. This is the source of truth for the migration.
• Secrets and configuration. The env vars hold live credentials: Shopify client secret, Sentry auth token, PocketBase superuser passwords, JWT private keys, MIDDLEWARE_SECRET_KEY, DB passwords, AUTH_SECRET, Vince admin password. Capture them all — and note they are why the backup repository must be encrypted at rest.
• Caddy — the Caddyfile and /var/lib/caddy/data (ACME account key + ~20 issued certificates). Losing the account key means re-issuing everything against Let's Encrypt rate limits.
• Deployment scripts — everything in ~/bin/, both crontabs, ~/.zshrc.
• Package and image inventory — dpkg --get-selections, docker images --digests, so the host can be rebuilt from scratch if it must be.
• DNS records for the ~20 hostnames pointing at 13.202.6.2.

None of this needs the services stopped.

---

6. Phase 1 — EBS snapshot (no downtime)

aws ec2 create-snapshot \
  --volume-id <vol-id from nvme0n1> \
  --description "monoco pre-podman-migration $(date -Iseconds)"

Then:

• Tag it with the date, purpose, and the fact that it predates the Podman migration.
• Wait for State: completed (not just pending). A pending snapshot is not a backup.
• Copy it to a second region if it is meant to survive regional loss.

This is your fastest rollback if the migration goes badly wrong. awscli is not installed — install it or take the snapshot from the console.

---

7. Phase 2 — the logical backup (the one that matters)

Container-native, so it works identically before and after the Podman migration — write it this way and you do not have to rewrite it during the cutover.

7.1 Postgres (~890 MB → est. 200–300 MB compressed)

# roles and globals first — per-DB dumps do NOT include these
docker exec postgres pg_dumpall -U postgres --roles-only | gzip > roles.sql.gz

# one restorable file per database
for db in affine bitwarden brewman_acc brewman_exp brewman_hinchco \
          brewman_hops brewman_mhl hiwikijs nextcloud tandoor; do
  docker exec postgres pg_dump -U postgres -Fc "$db" > "$db.dump"
done

Deliberately excluded: postgres (system), and the medusa-criollo / zitadel orphans — unless you decide otherwise (§3.2).

7.2 Gitea — use its own dump

docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini -f /data/dumps

One archive with gitea.db, the repositories, and config, produced consistently. A tar of /var/lib/gitea will probably work and will occasionally produce a corrupt gitea.db (§3.1).

7.3 SQLite services — stop, copy, start

Simplest provably-correct method, and these are fast to restart:

for c in mozimo mozimo-staging greatbear hops trillium yamtrack audiobookshelf; do
  docker stop "$c"
done
# tar the relevant paths (§2.1 — skip audiobooks/, log/, staticfiles/)
for c in ...; do docker start "$c"; done

Alternatives if downtime is unacceptable: sqlite3 "$db" ".backup" per database, or PocketBase's built-in backup for the four pb_data services.

7.4 Nextcloud

docker exec -u www-data nextcloud php occ maintenance:mode --on
docker exec postgres pg_dump -U postgres -Fc nextcloud > nextcloud.dump
tar -czf nextcloud-files.tar.gz \
  -C /var/lib/nextcloud data config custom_apps
docker exec -u www-data nextcloud php occ maintenance:mode --off

7.5 Caddy, affine, sms-wa-bridge, menu apps, tandoor media

Plain file copies — none of these need a quiesce: /var/lib/caddy (all 736 KB), /var/lib/affine/{config,storage}, /var/lib/sms-wa-bridge/rules.json, each /var/lib/menu.*, /var/lib/tandoor/mediafiles.

7.6 The anonymous volumes

hiwikijs and valkey are not under /var/lib — they live in /var/lib/docker/volumes/<hash>/_data and are invisible to every existing script. Export them to real directories before the migration (the migration plan requires it anyway):

docker run --rm -v 9c1e3fde…:/from -v /var/backups/vol:/to alpine \
  sh -c 'cp -a /from/. /to/hiwikijs/'

hiwikijs matters; valkey is a rebuildable cache.

7.7 Package it

Install restic (static aarch64 binary — nothing is installed today), initialise an encrypted repo on S3, and back up the staging directory:

restic -r s3:s3.ap-south-1.amazonaws.com/monoco-backup init
restic -r s3:… backup /var/backups/monoco-$(date +%F) --tag pre-podman-migration
restic -r s3:… check

restic gives you encryption, deduplication (so daily runs are cheap), and snapshots you can mount and restore individual files from. No credentials are configured on this host today, so this needs an IAM role on the instance or an access key placed deliberately.

---

8. Phase 3 — bulk media (decision required)

The 15 GB of audiobooks and the 307 MB WhatsApp session.

• If the audiobooks exist elsewhere (purchased library, cloud, seedbox), exclude them. They are already covered by the EBS snapshot for the "instance died" case, and including them inflates every backup run for no recovery benefit.
• If they exist only here, they must be in scope — but do not put them in restic daily. Handle them as a separate, infrequent, one-way copy.

This decision changes the backup from 2.4 GB to 18 GB. Make it explicitly.

---

9. Where the backup physically goes

There is 30 GB free on a single volume. A backup written there is not a backup: it shares the failure domain with the thing it protects, and a 19 GB archive would fill the disk and take production down.

Destination	Status
S3 ap-south-1 (restic)	Recommended. Same region = fast and cheap; add cross-region for DR
EBS snapshot	Recommended, whole-system
Google Drive via ~/bin/gdrive	Broken — binary not installed, ~/Data/warden does not exist
Another server	No SSH hosts configured on the box; registry.tanshu.com (3.109.171.183) is a candidate if it has space
Local disk	Staging only

---

10. Verification — acceptance criteria

A backup that has never been restored is a hypothesis. Before the migration proceeds:

• restic check --read-data-subset=5% passes
• Every Postgres dump restores into a scratch database, and a row count on a known table matches production
• roles.sql.gz restores (roles/globals are the classic omission)
• The Gitea dump extracts and gitea.db opens with sqlite3 … "PRAGMA integrity_check"
• One PocketBase data.db restores and opens
• trillium/document.db and yamtrack/db.sqlite3 open
• Nextcloud restores with config/config.php, in a scratch instance
• /var/lib/caddy/data contains the ACME account key and the certificate set
• The EBS snapshot reaches completed, and is booted as a test instance with services confirmed running
• A manifest of checksums exists for the archive set
• Original uid:gid ownership of every /var/lib path is recorded — required to undo the migration's chown (§14)

---

11. Ongoing backups — currently there are none

This is the biggest single finding: nothing is scheduled.

• The only cron entry on the host is */5 * * * * docker exec -u www-data nextcloud php cron.php.
• tanshu, ubuntu, and git have no crontab.
• No systemd timer mentions backup (only dpkg-db-backup.timer).
• No restic, borg, rclone, duplicity, or aws CLI is installed.
• The five archives in ~/ are manual one-offs — Jun 2025, Feb 2026, Sep 2026, Oct 2026 — for bitwarden, brewman_hops (SQL only), and nextcloud, plus obsolete mongodb.tar and overleaf.tar.

So backups depend on someone remembering. Recommended replacement — a restic run on a systemd timer, with all the container-native dump steps from §7 baked into a script, retention of 7 daily / 4 weekly / 6 monthly, and failure alerting. A backup that fails silently every night for three months is indistinguishable from no backup, which is exactly the state today.

---

12. Retention

Layer	Retention
Logical (restic)	7 daily, 4 weekly, 6 monthly
Weekly full	3 months
Monthly full	12 months
EBS snapshots	Keep the pre-migration one indefinitely; daily snapshots 7–14 days

---

13. Bugs in the existing backup scripts

Worth fixing regardless of the migration, because these are actively misleading:

13.1 hiwikijs-backup.sh backs up nothing. It runs tar -C /var/lib hiwikijs, but /var/lib/hiwikijs is an empty directory (drwxr-xr-x www-data, dated Dec 2022). The real Wiki.js content is in the anonymous volume 9c1e3fde…. The script exits successfully and produces a valid archive of nothing. Anyone trusting it has no Wiki.js backup at all.

13.2 gitea-backup.sh hot-copies a live SQLite database (§3.1). It will usually work, which is the worst property a backup can have. Replace with gitea dump.

13.3 bitwarden-backup.sh + bitwarden-upload.sh are inconsistent with each other and with reality (§3.3) — one dumps Postgres, the other uploads a SQLite file that does not exist, and neither runs on a schedule.

13.4 iot-backup.sh and kakawa-backup.sh target services that no longer exist (the iot_net/kakawa_net Docker networks are still attached to Caddy, but the containers are gone).

13.5 All DB scripts use --link postgres:db, which does not exist in Podman and will break at cutover. Under rootless Podman the --user $(id -u) and sudo chown gymnastics are also unnecessary once the data is owned by you.

13.6 No coverage at all for: mozimo, mozimo-staging, greatbear, hops, the three menu-app.* containers, affine, vince, audiobookshelf, valkey, or /var/lib/caddy (the certificates).

---

14. Pre-migration specifics

Beyond a normal full backup, the migration needs these captured, because it changes ownership and layout:

1. Original uid:gid and mode of every bind-mount source — the migration's chown -R tanshu:tanshu is not trivially reversible without this table.
2. The two anonymous volumes exported (§7.6) — otherwise hiwikijs state is lost permanently when Docker is removed.
3. docker inspect JSON for all 26 containers — the only complete record of the runtime configuration.
4. A restore rehearsal onto a scratch host or the booted EBS snapshot, following the migration plan's rollback path.

---

15. Risks

Risk	Likelihood	Impact	Mitigation
Backups silently produce corrupt SQLite	High	Data loss discovered at restore time	§3.1 methods; integrity-check every DB in verification
hiwikijs backed up as an empty directory	Certain today	Permanent wiki loss	Export the anonymous volume (§7.6)
Postgres roles/globals omitted	Medium	Restore fails on permissions	pg_dumpall --roles-only (§7.1)
Stale bitwarden SQLite restored instead of Postgres	Medium	Months of vault data lost	§3.3 — mark or delete it
Let's Encrypt account key lost	Low	~20 certs blocked by rate limits	Include all of /var/lib/caddy
Backup written to the same volume	Medium	Both lost together	S3/Snapshot only (§9)
30 GB free filled by staging	Medium	Production outage during backup	Stage only the ~2.4 GB of real state; media excluded
Snapshot left in pending	Low	Restore from an incomplete snapshot	Wait for completed; boot-test it
Same-region snapshot as sole DR	Medium	Region/account loss unrecoverable	Cross-region copy
No alerting on backup failure	High	Months of silent failure	Monitoring on the timer (§11)

---

16. Open questions

1. Is the 15 GB of audiobook media in scope? (§8) This decides 2.4 GB vs 18 GB.
2. Where do backups go? S3 in ap-south-1 is the natural fit — is there an IAM role on the instance, or should a dedicated bucket + credentials be set up?
3. Are medusa-criollo and zitadel databases dead? If so, exclude them and drop them.
4. The 1.3 MB bitwarden-backup_202602261629.tar.gz — is the SQLite in it the pre-Postgres state? If so it is a historical artifact worth keeping deliberately, and the live db.sqlite3 should be deleted so nobody restores the wrong thing.
5. Is registry.tanshu.com a viable second backup target (independent host, but same region and presumably same provider account)?
6. Should the ongoing backup be restic, or plain pg_dump + tar on a timer? restic is better but adds a component; the container-native dumps are needed either way.