Add the Podman migration planning docs
Three documents written before the container migration, all referenced from the service readmes, so they belong in the tree: - podman-compat-audit.md -- a read-only audit of the monoco fleet: every image's architecture, privileges and namespaces; the four things that actually break (anonymous volumes, per-container uids, tooling that assumes Docker, image naming); and the per-container uid matrix that the UserNS=keep-id lines in the Quadlet units come from. - podman-migration.md -- the plan: target architecture, the ownership/chown pass, the Quadlet unit shape, and the low-port and lingering prerequisites. - backup-plan.md -- per-service irreplaceable state, the consistency problems (live SQLite WAL, hot Postgres, the stale bitwarden SQLite), and the 3-2-1 architecture. caddy, postgres, tandoor and bitwarden each link into these from their readme, so leaving them untracked left those links dead.
This commit is contained in:
tanshu committed
1 parent
f431d94f06
commit
8136c2ddaf
3 files changed
+1466
No files matched your search
@@ -0,0 +1,469 @@
|
||||
# Full backup plan for `monoco`
|
||||
|
||||
Status: **plan, not yet executed**
|
||||
Audited: live host, read-only. No changes made.
|
||||
Host: EC2 `i-08edfd71ce301b8ba`, `ap-south-1c`, Ubuntu 24.04.5 aarch64
|
||||
|
||||
---
|
||||
|
||||
## 1. Objective
|
||||
|
||||
A complete, **verified** backup of monoco that:
|
||||
|
||||
1. Can restore the entire host (disaster recovery), and
|
||||
2. Can restore any single service in isolation (the important case day-to-day), and
|
||||
3. Is a hard prerequisite for the Docker → Podman migration, because that migration
|
||||
changes file ownership on every data directory.
|
||||
|
||||
"Verified" is doing real work in that sentence. §10 defines what verification means,
|
||||
and it is not optional.
|
||||
|
||||
---
|
||||
|
||||
## 2. What is actually on the box
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Provider | **EC2 confirmed** (IMDSv2), `i-08edfd71ce301b8ba`, `ap-south-1c` |
|
||||
| Storage | **Single 80 GB EBS root volume** (`nvme0n1p1`), ext4, **no LVM**, no separate data volume |
|
||||
| Free space | **30 GB** of 77 GB |
|
||||
| Containers | 26, all plain `docker run`, all state on host bind mounts |
|
||||
| Total app data | **~19 GB** |
|
||||
| **Irreplaceable data** | **~2.4 GB** |
|
||||
|
||||
That last line is the most important number in this document. The gap between 19 GB
|
||||
and 2.4 GB is almost entirely one thing:
|
||||
|
||||
```
|
||||
/var/lib/audiobookshelf/audiobooks 15.0 GB 696 files. Re-downloadable media.
|
||||
/var/lib/audiobookshelf/config 2.6 MB ← absdatabase.sqlite, irreplaceable
|
||||
/var/lib/audiobookshelf/metadata 19 MB ← irreplaceable
|
||||
```
|
||||
|
||||
So the real state of Audiobookshelf is **22 MB, not 15 GB**. Treating it as 15 GB is
|
||||
what makes a "full backup" feel impossible; it isn't. Decide explicitly whether the
|
||||
media is in scope (§8) — but do not let it drive the design.
|
||||
|
||||
### 2.1 What is irreplaceable, by service
|
||||
|
||||
| Service | Total | Back up | Skip (regenerable / from image) |
|
||||
|---|---|---|---|
|
||||
| `postgres` | 995 MB | 13 databases, ~890 MB raw | — |
|
||||
| `gitea` | 747 MB | `git/` 302 MB + `gitea/` 445 MB **incl. gitea.db** | — |
|
||||
| `nextcloud` | 1.2 GB | `data/` 324 MB, `config/` (config.php!), `custom_apps/` 5.8 MB | `apps/` 522 MB, `core/` 159 MB, `dist/` 83 MB, `3rdparty/` 55 MB, `lib/` 19 MB |
|
||||
| `mozimo` | 147 MB | `pb_data/` (incl. `auxiliary.db` 18.6 MB + WAL) | `types.d.ts` |
|
||||
| `mozimo-staging` | 137 MB | `pb_data/` | |
|
||||
| `greatbear` | 101 MB | `pb_data/` | |
|
||||
| `tandoor` | 98 MB | `mediafiles/` | `staticfiles/`, `externalfiles/` |
|
||||
| `menu-app.*` ×3 | 210 MB | `/data` each | |
|
||||
| `trillium` | 106 MB | `document.db` 3.8 MB + `backup/` 12 MB | `log/` 91 MB |
|
||||
| `sms-wa-bridge` | 397 MB | `rules.json` (tiny) | `wwebjs_auth/` 307 MB — re-linkable via QR |
|
||||
| `hops` | 19 MB | `pb_data/` | |
|
||||
| `audiobookshelf` | 15 GB | `config/` + `metadata/` = **22 MB** | `audiobooks/` 15 GB |
|
||||
| `bitwarden` | 4.2 MB | **Postgres `bitwarden` DB** | `data/db.sqlite3` — **stale**, see §3.3 |
|
||||
| `yamtrack` | 2.3 MB | `data/db.sqlite3` | |
|
||||
| `affine` | 916 KB | `config/` + `storage/` | |
|
||||
| `caddy` | 736 KB | **all of it** — ACME account + certs | |
|
||||
| `hiwikijs` | ? | **anonymous volume** — see §13.1 | `/var/lib/hiwikijs` is an empty decoy |
|
||||
| `valkey` | ? | — | cache, rebuildable |
|
||||
|
||||
**~2.4 GB** once media and the WhatsApp session are excluded.
|
||||
|
||||
---
|
||||
|
||||
## 3. The consistency problem — read this before writing any backup
|
||||
|
||||
This is the part that silently produces useless backups. Every one of these is being
|
||||
written to *right now* while the container runs.
|
||||
|
||||
### 3.1 SQLite in WAL mode
|
||||
|
||||
Live WAL sidecars are present on disk, which proves the databases are hot:
|
||||
|
||||
```
|
||||
/var/lib/mozimo/pb_data/auxiliary.db 18,632,704 + -wal 1,116,552 + -shm
|
||||
/var/lib/mozimo/pb_data/data.db 204,800 + -wal 78,312 + -shm
|
||||
/var/lib/gitea/gitea/gitea.db 8,147,968 (modified 06:07 today)
|
||||
/var/lib/audiobookshelf/config/absdatabase.sqlite 2,576,384
|
||||
/var/lib/trillium/document.db 3,800,000 + -shm
|
||||
/var/lib/yamtrack/data/db.sqlite3 + -wal + -shm
|
||||
```
|
||||
|
||||
**A plain `tar` of a live SQLite database can produce a backup that will not open.**
|
||||
The main file and the WAL are copied at different instants, so the result can be torn.
|
||||
This affects `gitea-backup.sh` today.
|
||||
|
||||
Three acceptable methods, in order of preference:
|
||||
|
||||
1. **Use the application's own dump command** — always best, because the app
|
||||
understands its own invariants:
|
||||
- Gitea: `docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini`
|
||||
→ one zip containing the DB, repos, and config.
|
||||
- PocketBase (`mozimo`, `mozimo-staging`, `greatbear`, `hops`): use the built-in
|
||||
backup, or stop the container for the copy.
|
||||
2. **`sqlite3 <db> ".backup <out>"`** — safe on a live database, needs `sqlite3`
|
||||
(present on the host: `~/.sqlite_history` exists).
|
||||
3. **Stop the container, copy the directory, start it.** Simple, provable, costs
|
||||
seconds of downtime per service. For a pre-migration backup this is the right
|
||||
default, because the containers are being stopped anyway.
|
||||
|
||||
### 3.2 Postgres
|
||||
|
||||
13 databases, ~890 MB raw. A hot `tar` of `/var/lib/postgresql/data` is **not** a
|
||||
valid backup — you need a logical dump or a proper physical base backup.
|
||||
|
||||
```
|
||||
affine 11 MB | bitwarden 10 MB | brewman_acc 174 MB | brewman_exp 16 MB
|
||||
brewman_hinchco 41 MB | brewman_hops 122 MB | brewman_mhl 132 MB | hiwikijs 16 MB
|
||||
medusa-criollo 21 MB | nextcloud 319 MB | postgres 7.5 MB | tandoor 16 MB
|
||||
zitadel 9.3 MB
|
||||
```
|
||||
|
||||
Note two orphans with no running container — decide whether they are in scope:
|
||||
**`medusa-criollo`** (21 MB, no container references it) and **`zitadel`** (9.3 MB,
|
||||
container deleted). Everything else maps to a live service.
|
||||
|
||||
Method: `pg_dump -Fc` per database (parallel-safe, individually restorable) **plus**
|
||||
`pg_dumpall --roles-only` for roles/globals, which per-database dumps do not capture.
|
||||
|
||||
### 3.3 The bitwarden trap
|
||||
|
||||
`bitwarden` has `DATABASE_URL=postgresql://…@postgres:5432/bitwarden`, and a 10 MB
|
||||
`bitwarden` database exists in Postgres. **Postgres is authoritative.**
|
||||
|
||||
But `/var/lib/bitwarden/data/db.sqlite3` still exists with `-wal`/`-shm` sidecars, and
|
||||
`~/bin/bitwarden-backup.sh` dumps the Postgres DB while `bitwarden-upload.sh` uploads a
|
||||
`~/Data/warden/production.sqlite3` that **does not exist** (and `~/bin/gdrive` is not
|
||||
installed).
|
||||
|
||||
So the current bitwarden backup story is three-quarters broken, and there is a stale
|
||||
SQLite file that would look like a plausible restore source and silently lose months of
|
||||
data. **Back up the Postgres database. Explicitly mark or delete the stale SQLite.**
|
||||
|
||||
### 3.4 Nextcloud
|
||||
|
||||
Files can be copied live; the **database cannot**. Dump the DB and the files together
|
||||
so they correspond. Nextcloud's own guidance is maintenance mode during the backup:
|
||||
`occ maintenance:mode --on`, dump, copy, `--off`. `config/config.php` must be included —
|
||||
without it the instance is not restorable.
|
||||
|
||||
---
|
||||
|
||||
## 4. Backup architecture
|
||||
|
||||
Standard 3-2-1, using what this host actually has:
|
||||
|
||||
| Layer | What | Why | Where |
|
||||
|---|---|---|---|
|
||||
| **1. EBS snapshot** | Whole 80 GB volume | Instant whole-system restore; the only thing that captures the OS, Docker images, and 15 GB of media in one shot | AWS, S3-backed, inherently offsite |
|
||||
| **2. Logical backup (restic)** | The ~2.4 GB of real state | Provably restorable, per-service, deduplicated, **encrypted** (matters — see §5) | S3 bucket in `ap-south-1` |
|
||||
| **3. Local staging** | Transient only | Working space during the run | `/var/backups/` — **not** a backup |
|
||||
|
||||
Why both 1 and 2: an EBS snapshot is *crash-consistent*, not application-consistent —
|
||||
it is equivalent to pulling the power cord. Restoring it means Postgres and every
|
||||
SQLite database go through crash recovery. It is excellent for "the instance died" and
|
||||
poor for "I need last night's database cleanly". The logical layer is the one that
|
||||
gives you a *verified* restore.
|
||||
|
||||
**Region note:** an EBS snapshot in `ap-south-1` does not survive the loss of that
|
||||
region or that AWS account. Enable **cross-region snapshot copy** (or at minimum
|
||||
replicate the restic repo to a second bucket) before calling this disaster recovery.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 0 — freeze the definitions (no downtime, minutes)
|
||||
|
||||
Do this first: it is tiny, has zero impact, and it is the part that is easiest to
|
||||
forget and most annoying to reconstruct.
|
||||
|
||||
- **Every container's full config** — `docker inspect` for all containers, networks,
|
||||
volumes → JSON. This is the source of truth for the migration.
|
||||
- **Secrets and configuration.** The env vars hold live credentials: Shopify client
|
||||
secret, Sentry auth token, PocketBase superuser passwords, JWT private keys,
|
||||
`MIDDLEWARE_SECRET_KEY`, DB passwords, `AUTH_SECRET`, Vince admin password. Capture
|
||||
them all — and note they are why the backup repository **must be encrypted at rest**.
|
||||
- **Caddy** — the Caddyfile *and* `/var/lib/caddy/data` (ACME account key + ~20 issued
|
||||
certificates). Losing the account key means re-issuing everything against Let's
|
||||
Encrypt rate limits.
|
||||
- **Deployment scripts** — everything in `~/bin/`, both crontabs, `~/.zshrc`.
|
||||
- **Package and image inventory** — `dpkg --get-selections`, `docker images --digests`,
|
||||
so the host can be rebuilt from scratch if it must be.
|
||||
- **DNS records** for the ~20 hostnames pointing at `13.202.6.2`.
|
||||
|
||||
None of this needs the services stopped.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phase 1 — EBS snapshot (no downtime)
|
||||
|
||||
```
|
||||
aws ec2 create-snapshot \
|
||||
--volume-id <vol-id from nvme0n1> \
|
||||
--description "monoco pre-podman-migration $(date -Iseconds)"
|
||||
```
|
||||
|
||||
Then:
|
||||
- Tag it with the date, purpose, and the fact that it **predates the Podman migration**.
|
||||
- Wait for `State: completed` (not just `pending`). A pending snapshot is not a backup.
|
||||
- **Copy it to a second region** if it is meant to survive regional loss.
|
||||
|
||||
This is your fastest rollback if the migration goes badly wrong. `awscli` is not
|
||||
installed — install it or take the snapshot from the console.
|
||||
|
||||
---
|
||||
|
||||
## 7. Phase 2 — the logical backup (the one that matters)
|
||||
|
||||
Container-native, so it works identically before and after the Podman migration —
|
||||
write it this way and you do not have to rewrite it during the cutover.
|
||||
|
||||
### 7.1 Postgres (~890 MB → est. 200–300 MB compressed)
|
||||
|
||||
```
|
||||
# roles and globals first — per-DB dumps do NOT include these
|
||||
docker exec postgres pg_dumpall -U postgres --roles-only | gzip > roles.sql.gz
|
||||
|
||||
# one restorable file per database
|
||||
for db in affine bitwarden brewman_acc brewman_exp brewman_hinchco \
|
||||
brewman_hops brewman_mhl hiwikijs nextcloud tandoor; do
|
||||
docker exec postgres pg_dump -U postgres -Fc "$db" > "$db.dump"
|
||||
done
|
||||
```
|
||||
|
||||
Deliberately excluded: `postgres` (system), and the `medusa-criollo` / `zitadel`
|
||||
orphans — unless you decide otherwise (§3.2).
|
||||
|
||||
### 7.2 Gitea — use its own dump
|
||||
|
||||
```
|
||||
docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini -f /data/dumps
|
||||
```
|
||||
|
||||
One archive with `gitea.db`, the repositories, and config, produced consistently.
|
||||
A `tar` of `/var/lib/gitea` will *probably* work and will occasionally produce a
|
||||
corrupt `gitea.db` (§3.1).
|
||||
|
||||
### 7.3 SQLite services — stop, copy, start
|
||||
|
||||
Simplest provably-correct method, and these are fast to restart:
|
||||
|
||||
```
|
||||
for c in mozimo mozimo-staging greatbear hops trillium yamtrack audiobookshelf; do
|
||||
docker stop "$c"
|
||||
done
|
||||
# tar the relevant paths (§2.1 — skip audiobooks/, log/, staticfiles/)
|
||||
for c in ...; do docker start "$c"; done
|
||||
```
|
||||
|
||||
Alternatives if downtime is unacceptable: `sqlite3 "$db" ".backup"` per database, or
|
||||
PocketBase's built-in backup for the four `pb_data` services.
|
||||
|
||||
### 7.4 Nextcloud
|
||||
|
||||
```
|
||||
docker exec -u www-data nextcloud php occ maintenance:mode --on
|
||||
docker exec postgres pg_dump -U postgres -Fc nextcloud > nextcloud.dump
|
||||
tar -czf nextcloud-files.tar.gz \
|
||||
-C /var/lib/nextcloud data config custom_apps
|
||||
docker exec -u www-data nextcloud php occ maintenance:mode --off
|
||||
```
|
||||
|
||||
### 7.5 Caddy, affine, sms-wa-bridge, menu apps, tandoor media
|
||||
|
||||
Plain file copies — none of these need a quiesce:
|
||||
`/var/lib/caddy` (all 736 KB), `/var/lib/affine/{config,storage}`,
|
||||
`/var/lib/sms-wa-bridge/rules.json`, each `/var/lib/menu.*`,
|
||||
`/var/lib/tandoor/mediafiles`.
|
||||
|
||||
### 7.6 The anonymous volumes
|
||||
|
||||
`hiwikijs` and `valkey` are **not** under `/var/lib` — they live in
|
||||
`/var/lib/docker/volumes/<hash>/_data` and are invisible to every existing script.
|
||||
Export them to real directories before the migration (the migration plan requires it
|
||||
anyway):
|
||||
|
||||
```
|
||||
docker run --rm -v 9c1e3fde…:/from -v /var/backups/vol:/to alpine \
|
||||
sh -c 'cp -a /from/. /to/hiwikijs/'
|
||||
```
|
||||
|
||||
`hiwikijs` matters; `valkey` is a rebuildable cache.
|
||||
|
||||
### 7.7 Package it
|
||||
|
||||
Install **restic** (static aarch64 binary — nothing is installed today), initialise an
|
||||
encrypted repo on S3, and back up the staging directory:
|
||||
|
||||
```
|
||||
restic -r s3:s3.ap-south-1.amazonaws.com/monoco-backup init
|
||||
restic -r s3:… backup /var/backups/monoco-$(date +%F) --tag pre-podman-migration
|
||||
restic -r s3:… check
|
||||
```
|
||||
|
||||
restic gives you encryption, deduplication (so daily runs are cheap), and snapshots you
|
||||
can mount and restore individual files from. No credentials are configured on this host
|
||||
today, so this needs an IAM role on the instance or an access key placed deliberately.
|
||||
|
||||
---
|
||||
|
||||
## 8. Phase 3 — bulk media (decision required)
|
||||
|
||||
The 15 GB of audiobooks and the 307 MB WhatsApp session.
|
||||
|
||||
- **If the audiobooks exist elsewhere** (purchased library, cloud, seedbox), exclude
|
||||
them. They are already covered by the EBS snapshot for the "instance died" case, and
|
||||
including them inflates every backup run for no recovery benefit.
|
||||
- **If they exist only here**, they must be in scope — but do not put them in restic
|
||||
daily. Handle them as a separate, infrequent, one-way copy.
|
||||
|
||||
This decision changes the backup from 2.4 GB to 18 GB. Make it explicitly.
|
||||
|
||||
---
|
||||
|
||||
## 9. Where the backup physically goes
|
||||
|
||||
There is **30 GB free** on a single volume. A backup written there is not a backup: it
|
||||
shares the failure domain with the thing it protects, and a 19 GB archive would fill
|
||||
the disk and take production down.
|
||||
|
||||
| Destination | Status |
|
||||
|---|---|
|
||||
| S3 `ap-south-1` (restic) | **Recommended.** Same region = fast and cheap; add cross-region for DR |
|
||||
| EBS snapshot | Recommended, whole-system |
|
||||
| Google Drive via `~/bin/gdrive` | **Broken** — binary not installed, `~/Data/warden` does not exist |
|
||||
| Another server | No SSH hosts configured on the box; `registry.tanshu.com` (3.109.171.183) is a candidate if it has space |
|
||||
| Local disk | Staging only |
|
||||
|
||||
---
|
||||
|
||||
## 10. Verification — acceptance criteria
|
||||
|
||||
A backup that has never been restored is a hypothesis. Before the migration proceeds:
|
||||
|
||||
- [ ] `restic check --read-data-subset=5%` passes
|
||||
- [ ] **Every Postgres dump restores** into a scratch database, and a row count on a
|
||||
known table matches production
|
||||
- [ ] `roles.sql.gz` restores (roles/globals are the classic omission)
|
||||
- [ ] The Gitea dump extracts and `gitea.db` opens with `sqlite3 … "PRAGMA integrity_check"`
|
||||
- [ ] One PocketBase `data.db` restores and opens
|
||||
- [ ] `trillium/document.db` and `yamtrack/db.sqlite3` open
|
||||
- [ ] Nextcloud restores **with** `config/config.php`, in a scratch instance
|
||||
- [ ] `/var/lib/caddy/data` contains the ACME account key and the certificate set
|
||||
- [ ] The EBS snapshot reaches `completed`, and is **booted as a test instance** with
|
||||
services confirmed running
|
||||
- [ ] A manifest of checksums exists for the archive set
|
||||
- [ ] **Original uid:gid ownership of every `/var/lib` path is recorded** — required to
|
||||
undo the migration's `chown` (§14)
|
||||
|
||||
---
|
||||
|
||||
## 11. Ongoing backups — currently there are none
|
||||
|
||||
This is the biggest single finding: **nothing is scheduled.**
|
||||
|
||||
- The only cron entry on the host is `*/5 * * * * docker exec -u www-data nextcloud php cron.php`.
|
||||
- `tanshu`, `ubuntu`, and `git` have no crontab.
|
||||
- No systemd timer mentions backup (only `dpkg-db-backup.timer`).
|
||||
- No restic, borg, rclone, duplicity, or aws CLI is installed.
|
||||
- The five archives in `~/` are **manual one-offs** — Jun 2025, Feb 2026, Sep 2026,
|
||||
Oct 2026 — for bitwarden, brewman_hops (SQL only), and nextcloud, plus obsolete
|
||||
`mongodb.tar` and `overleaf.tar`.
|
||||
|
||||
So backups depend on someone remembering. Recommended replacement — a `restic` run on a
|
||||
systemd timer, with all the container-native dump steps from §7 baked into a script,
|
||||
retention of **7 daily / 4 weekly / 6 monthly**, and **failure alerting**. A backup that
|
||||
fails silently every night for three months is indistinguishable from no backup, which
|
||||
is exactly the state today.
|
||||
|
||||
---
|
||||
|
||||
## 12. Retention
|
||||
|
||||
| Layer | Retention |
|
||||
|---|---|
|
||||
| Logical (restic) | 7 daily, 4 weekly, 6 monthly |
|
||||
| Weekly full | 3 months |
|
||||
| Monthly full | 12 months |
|
||||
| EBS snapshots | Keep the pre-migration one indefinitely; daily snapshots 7–14 days |
|
||||
|
||||
---
|
||||
|
||||
## 13. Bugs in the existing backup scripts
|
||||
|
||||
Worth fixing regardless of the migration, because these are actively misleading:
|
||||
|
||||
**13.1 `hiwikijs-backup.sh` backs up nothing.** It runs
|
||||
`tar -C /var/lib hiwikijs`, but `/var/lib/hiwikijs` is an **empty directory**
|
||||
(`drwxr-xr-x www-data`, dated Dec 2022). The real Wiki.js content is in the anonymous
|
||||
volume `9c1e3fde…`. The script exits successfully and produces a valid archive of
|
||||
nothing. Anyone trusting it has no Wiki.js backup at all.
|
||||
|
||||
**13.2 `gitea-backup.sh` hot-copies a live SQLite database** (§3.1). It will usually
|
||||
work, which is the worst property a backup can have. Replace with `gitea dump`.
|
||||
|
||||
**13.3 `bitwarden-backup.sh` + `bitwarden-upload.sh` are inconsistent with each other
|
||||
and with reality** (§3.3) — one dumps Postgres, the other uploads a SQLite file that
|
||||
does not exist, and neither runs on a schedule.
|
||||
|
||||
**13.4 `iot-backup.sh` and `kakawa-backup.sh` target services that no longer exist**
|
||||
(the `iot_net`/`kakawa_net` Docker networks are still attached to Caddy, but the
|
||||
containers are gone).
|
||||
|
||||
**13.5 All DB scripts use `--link postgres:db`**, which does not exist in Podman and
|
||||
will break at cutover. Under rootless Podman the `--user $(id -u)` and `sudo chown`
|
||||
gymnastics are also unnecessary once the data is owned by you.
|
||||
|
||||
**13.6 No coverage at all** for: `mozimo`, `mozimo-staging`, `greatbear`, `hops`,
|
||||
the three `menu-app.*` containers, `affine`, `vince`, `audiobookshelf`, `valkey`, or
|
||||
`/var/lib/caddy` (the certificates).
|
||||
|
||||
---
|
||||
|
||||
## 14. Pre-migration specifics
|
||||
|
||||
Beyond a normal full backup, the migration needs these captured, because it changes
|
||||
ownership and layout:
|
||||
|
||||
1. **Original `uid:gid` and mode of every bind-mount source** — the migration's
|
||||
`chown -R tanshu:tanshu` is not trivially reversible without this table.
|
||||
2. **The two anonymous volumes exported** (§7.6) — otherwise `hiwikijs` state is lost
|
||||
permanently when Docker is removed.
|
||||
3. **`docker inspect` JSON for all 26 containers** — the only complete record of the
|
||||
runtime configuration.
|
||||
4. **A restore rehearsal** onto a scratch host or the booted EBS snapshot, following the
|
||||
migration plan's rollback path.
|
||||
|
||||
---
|
||||
|
||||
## 15. Risks
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| Backups silently produce corrupt SQLite | **High** | Data loss discovered at restore time | §3.1 methods; integrity-check every DB in verification |
|
||||
| `hiwikijs` backed up as an empty directory | **Certain today** | Permanent wiki loss | Export the anonymous volume (§7.6) |
|
||||
| Postgres roles/globals omitted | Medium | Restore fails on permissions | `pg_dumpall --roles-only` (§7.1) |
|
||||
| Stale bitwarden SQLite restored instead of Postgres | Medium | Months of vault data lost | §3.3 — mark or delete it |
|
||||
| Let's Encrypt account key lost | Low | ~20 certs blocked by rate limits | Include all of `/var/lib/caddy` |
|
||||
| Backup written to the same volume | Medium | Both lost together | S3/Snapshot only (§9) |
|
||||
| 30 GB free filled by staging | Medium | Production outage during backup | Stage only the ~2.4 GB of real state; media excluded |
|
||||
| Snapshot left in `pending` | Low | Restore from an incomplete snapshot | Wait for `completed`; boot-test it |
|
||||
| Same-region snapshot as sole DR | Medium | Region/account loss unrecoverable | Cross-region copy |
|
||||
| No alerting on backup failure | **High** | Months of silent failure | Monitoring on the timer (§11) |
|
||||
|
||||
---
|
||||
|
||||
## 16. Open questions
|
||||
|
||||
1. **Is the 15 GB of audiobook media in scope?** (§8) This decides 2.4 GB vs 18 GB.
|
||||
2. **Where do backups go?** S3 in `ap-south-1` is the natural fit — is there an IAM role
|
||||
on the instance, or should a dedicated bucket + credentials be set up?
|
||||
3. **Are `medusa-criollo` and `zitadel` databases dead?** If so, exclude them and drop
|
||||
them.
|
||||
4. **The 1.3 MB `bitwarden-backup_202602261629.tar.gz`** — is the SQLite in it the
|
||||
pre-Postgres state? If so it is a historical artifact worth keeping deliberately,
|
||||
and the live `db.sqlite3` should be deleted so nobody restores the wrong thing.
|
||||
5. **Is `registry.tanshu.com` a viable second backup target** (independent host, but
|
||||
same region and presumably same provider account)?
|
||||
6. **Should the ongoing backup be restic, or plain `pg_dump` + `tar` on a timer?**
|
||||
restic is better but adds a component; the container-native dumps are needed either
|
||||
way.
|
||||
Reference in new issue
Block a user