v5.13.2 28 February 2026

Backup Script Rewrite — Production Gold

Why This Matters

The database is the only irreplaceable asset in the SpeyBooks stack. Code is in git. Config is versioned. Build output is regenerated on every deploy. But if the database is lost without a verified backup, the business is gone.

The previous backup script was a minimal shell wrapper — it dumped, uploaded to S3, and appended a single line to a log. No locking, no verification that the dump was restorable, no structured logging, and a permissions bug that prevented the log file from being written. This release replaces it with a script that treats database backup as the critical operation it is.

Backup Script v2.0.0

  • Dump verification After every pg_dump, the script runs pg_restore --list to read the table of contents without restoring. If the dump file is corrupt or truncated, this fails before the S3 upload. No more shipping broken dumps to storage.

  • Minimum size check A dump under 1KB is treated as a failure. Catches edge cases where pg_dump exits cleanly but produces an empty or near-empty file.

  • flock-based locking Prevents concurrent backup runs. The previous script had no protection against overlapping cron executions or manual runs during a scheduled backup.

  • Structured logging Timestamped, levelled log entries with UTC timestamps. Each phase logs its outcome. The summary includes file name, size, S3 path, and local retention count.

  • Typed exit codes Lock failure (1), dump failure (2), S3 failure (3), verification failure (4). Ready for external monitoring integration without parsing log files.

  • Permissions fix The log directory is now owned by the postgres user, matching the crontab context. The previous setup had root-owned log files that the postgres user could not write to.

What was removed

The restic-based approach was evaluated and rejected. SpeyBooks does not need file-level backup of /var/www or /etc — all code is in git, all config is versioned, and mirroring node_modules to S3 is waste. The backup script focuses exclusively on the database, which is the single irreplaceable asset.

Operational Impact

  • Backup cycle completes in under 2 seconds (dump + verify + S3 upload)
  • STH op-backups module passes immediately after a successful run
  • Corrupt dumps are caught before S3 upload, not after
  • Concurrent backup runs are safely prevented
  • Log output is parseable by any monitoring tool without regex gymnastics

Files Changed

Infrastructure:

  • /usr/local/bin/speybooks-backup.sh — v2.0.0: complete rewrite
  • /var/log/speybooks/ — directory ownership corrected to postgres:postgres
  • /var/backups/speybooks/postgres/ — staging directory ownership corrected
  • postgres crontab — removed log redirect (script handles its own logging)

A verified, restorable backup that runs in two seconds and fails loudly — the foundation everything else depends on.