Runbook: Backup & Restore (DR)

Synced read-only from /home/synvirt/synvirt-platform/docs-internal/runbooks/backup-restore.md. Edit at the source, not here.

Runbook: Backup & Restore (DR)

Status: WordPress (the public site) is backed up and its restore path has been rehearsed. Postgres is still draft for M8 — the pg_dump job below is not scheduled and its restore has never been tested. Do not treat this document as proof the platform is recoverable; only the WordPress half is.

What to back up

Asset	Method	Scheduled?	Destination
WordPress DB (MariaDB)	infra/scripts/backup-wordpress.sh (mariadb-dump --single-transaction)	yes — synvirt-backup.timer, daily 03:30	/var/backups/synvirt (on-server only)
WordPress uploads	same script, tar --zstd of apps/wordpress/uploads	yes — same run	/var/backups/synvirt (on-server only)
Postgres (per DB)	pg_dump per database	no — M8	off-server (offsite)
Host volumes	Veeam (filesystem level)	per existing ops	per existing ops
Keycloak realm	exported JSON in repo (infra/keycloak/)	n/a	git
Discourse uploads	included in its volume/backup (M7, not installed yet)	no	offsite
Theme, config, provisioning scripts	they are source — restoring them is a git checkout	n/a	git

Gap: WordPress backups are written to the same disk as the thing they back up. That protects against a bad deploy, a wrong wp command or a corrupted table — it does not protect against losing the box. Shipping /var/backups/synvirt offsite is the remaining work.

WordPress

task wp:backup                    # on demand
task wp:restore -- 2026-07-29     # prompts before overwriting the live DB
systemctl list-timers synvirt-backup
journalctl -u synvirt-backup

Retention is 14 days (SYNVIRT_BACKUP_RETENTION). The script refuses to accept a dump smaller than 10 KB, so a pipeline that “succeeds” while producing nothing fails loudly instead of silently replacing a good backup with an empty one.

Verifying a backup without touching the live site

A backup nobody has restored is a hypothesis. Restore into a scratch database:

RP=$(grep '^WP_DB_ROOT_PASSWORD=' .env | cut -d= -f2)
D=$(ls -t /var/backups/synvirt/wpdb_*.sql.zst | head -1)

docker exec -i synvirt-platform-wpdb-1 mariadb -uroot -p"$RP" \
  -e "DROP DATABASE IF EXISTS restore_test; CREATE DATABASE restore_test CHARACTER SET utf8mb4;"
zstd -dc "$D" | docker exec -i synvirt-platform-wpdb-1 mariadb -uroot -p"$RP" restore_test
docker exec synvirt-platform-wpdb-1 mariadb -uroot -p"$RP" restore_test -e \
  "SELECT COUNT(*) AS pages FROM svt_posts WHERE post_type='page' AND post_status='publish';"
docker exec synvirt-platform-wpdb-1 mariadb -uroot -p"$RP" -e "DROP DATABASE restore_test;"

Last rehearsed 2026-07-29: 16 pages, 12 attachments, 141 options, 23 taxonomies.

Postgres (portal, payload, keycloak, discourse)

for db in portal payload keycloak discourse; do
  docker compose --env-file .env -f infra/compose/compose.yaml exec -T postgres \
    pg_dump -U "$POSTGRES_USER" -Fc "$db" > "backup/${db}-$(date +%F).dump"
done
# ship backup/ offsite (rsync/rclone to external target)

Restore a single database

# recreate role/db if needed (infra/postgres/init runs only on first boot)
docker compose --env-file .env -f infra/compose/compose.yaml exec -T postgres \
  pg_restore -U "$POSTGRES_USER" -d portal --clean --if-exists < backup/portal-YYYY-MM-DD.dump

Secrets

infra/secrets/dev.enc.env is committed and encrypted; the age private key (~/.config/sops/age/keys.txt) is not in git. Losing that key means losing every secret in the repo — back it up separately from this machine, or the encrypted file is worthless in a real disaster.

Migratability note

Because each service owns its database, a single restore moves exactly one service’s data — no shared-schema untangling. The same dumps can seed a dedicated data host during the single→multi-host split.

Pre-launch checklist

• WordPress database + uploads backed up on a schedule
• WordPress restore rehearsed (into a scratch database)
• /var/backups/synvirt shipped offsite — currently same-disk only
• age private key backed up off this machine
• Scheduled pg_dump job (M8), verified landing offsite
• Postgres restore rehearsed into a scratch environment
• Recovery time + steps measured and documented here