Skip to content

HeyAira backup runbook

This runbook covers the change-driven PostgreSQL backup design for HeyAira. It is a repository reference, not proof that the production service is activated. The real bucket, credentials, private key custody, monitor and restore executor must be recorded separately without putting secrets in Git, HeyAira, chat or logs.

Operating model

The heartbeat is a cheap coordinator. It reads a PostgreSQL application-state marker (DML counters for user tables, excluding the backup control/evidence tables, bridge liveness tables, and short-lived OAuth/idempotency tables) and compares it with the last externally verified manifest. If the marker is already covered, it does nothing. If it changed, it coalesces one heyaira-backup work item. A leased worker then runs:

pg_dump --format=custom --no-owner
        -> age --recipient <public-recipient>
        -> rclone copyto encrypted-object and manifest
        -> rclone size --json verification

The worker never copies the live PostgreSQL volume. Plaintext exists only in its bounded temporary directory and is removed before upload. The successful manifest is the external evidence used by the next heartbeat. The 15-minute value from the original plan is a candidate maximum RPO/probe interval, not a blind full-dump timer; final cadence follows measured dump/upload duration and load.

The service file is selected explicitly with service=<name> in the pg_dump/pg_restore arguments. This prevents libpq from interpreting the service label as a database name when the production database name differs from the service label.

R2 layout and retention

Use a private R2 Standard bucket with explicit eu jurisdiction. Keep objects under separate UTC prefixes:

rolling/YYYY/MM/DD/HHMMSS/<backup-id>.dump.age
daily/YYYY/MM/DD/<backup-id>.dump.age
monthly/YYYY/MM/<backup-id>.dump.age

The local planner in src/heyaira/backup_retention.py proposes the finite policy: keep every verified rolling copy for 48 hours, promote one UTC daily representative for 30 days, and promote one UTC monthly representative for 180 days. Daily promotions are locked for at least 7 days. Incomplete and locked objects are never returned for deletion. Provider lifecycle rules are only a safety net; they do not replace the tested one-per-slot selection.

src/heyaira/backup_retention_runtime.py is the provider adapter. It accepts only a reviewed RetentionPlan, defaults to dry-run, copies encrypted bytes and rewrites the corresponding manifest for promotions, and deletes only the explicit object key plus its manifest for approved deletions. It never calls rclone sync or deletes a whole prefix.

Never use rclone sync for this history. Delete only explicit object keys returned by a reviewed retention plan. The plan must be dry-run reviewed against the actual bucket before any deletion capability is enabled.

Restore procedure

  1. Select a verified manifest and record its object key, backup ID, covered application-state marker, PostgreSQL version and service revision.
  2. Run RestoreWorker on a separate trusted executor with no route to the production volume or public port. The executor temporarily receives the private age identity; the application VPS does not receive it.
  3. Download the encrypted object with rclone copyto, verify its SHA-256 and byte size against the manifest, decrypt with age, then run pg_restore --exit-on-error --no-owner against an isolated PostgreSQL service.
  4. Verify schema, constraints, schema_migrations, and selected MCP reads using the code revision recorded in the manifest. Do not activate restored production tokens, OAuth grants or instance identity.
  5. Record elapsed time from object/key access to a usable isolated service, plus the exact manifest ID. A weekly restore is not considered active until an external executor has run this procedure successfully on schedule.

The repository CLI is heyaira-restore. It requires an explicit manifest, an isolated --restore-target, a protected age identity and a PostgreSQL service file for the isolated target. It prints only restore metadata; the private identity and R2 credentials never appear in its arguments or output.

Activation gate

Before production activation, an operator must provide the bounded approval and concrete non-secret identifiers for: R2 account/bucket with EU jurisdiction, sender credential scope, public age recipient, private-key custody and emergency copy, isolated restore executor, external monitor and alert recipient. Then verify the actual tool versions, service files, permissions, lock/lifecycle rules, and rollback. Only after a real encrypted copy and isolated restore should the independent monitor be enabled.

The monitor must alert on both an explicit failed run and a stale last-success manifest (working threshold: 30 minutes, to be calibrated by measurement). It must emit success only after upload and verification. A host-down signal must come from outside the VPS. The task remains incomplete until two real scheduled executions, a restore, and an alert delivery are evidenced.

The read-only heyaira-backup-health CLI reads only the durable heartbeat and terminal work-item outcomes. An independent host may invoke it with a database connection and use exit code 2 plus its JSON reasons for alerting. It is not itself the independent delivery channel; recipient and host-down transport remain operator configuration.

The provider-neutral decision logic is in src/heyaira/backup_monitor.py. It produces separate failure, stale-backup and stale-host reasons without delivering anything itself; the final adapter must use an independent monitor and an operator-supplied recipient.

Production heartbeat runtime

heyaira-backup runs two loops in the backup Compose service. The heartbeat polls every 10 seconds by default and only compares the current PostgreSQL application-state marker with the newest verified R2 manifest. A changed marker creates or coalesces one durable heyaira-backup work item in control_work_items. The worker claims that item independently and runs the dump/encrypt/upload pipeline. A heartbeat tick never performs a full dump merely because time passed. The interval is a responsiveness setting and may be raised after measuring the observed RPO and R2 list latency.

The backup service receives R2 credentials through the host-only backup.env file next to compose.yaml and maps them to rclone's environment-based S3 configuration. The file is not part of the repository. The application container never receives the R2 secret. Only the public age recipient is provided to the backup service; the private identity remains on the isolated restore executor.

The backup image pins PostgreSQL client 16 because the production PostgreSQL server is 16. The pg_dump and pg_restore major versions must match the production server for a portable custom-format dump and an isolated restore.

The rclone environment sets no_check_bucket=true. The R2 token is scoped to the selected bucket/object API, and this avoids a bucket-level preflight that Cloudflare rejects for object-scoped tokens.

The production host keeps backup.env at mode 0600. The repository's secrets/age/ directory is ignored by Git and is intended for the private restore identity only; it must never be copied to the application VPS.