Skip to content

HeyAira backup runbook

This runbook covers the change-driven PostgreSQL backup design for HeyAira. It is a repository reference, not proof that the production service is activated. The real bucket, credentials, private key custody, monitor and restore executor must be recorded separately without putting secrets in Git, HeyAira, chat or logs.

Operating model

The heartbeat is a cheap coordinator. It reads a PostgreSQL application-state marker (a digest of the insert/update/delete counters of the user tables and of PostgreSQL's statistics reset times) and compares it with the last externally verified manifest. If the marker is already covered, it does nothing. If it changed, it coalesces one heyaira-backup work item. A leased worker then runs:

pg_dump --format=custom --no-owner
        -> age --recipient <public-recipient>
        -> rclone copyto encrypted-object and manifest
        -> rclone size --json verification

The worker never copies the live PostgreSQL volume. Plaintext exists only in its bounded temporary directory and is removed before upload. The successful manifest is the external evidence used by the next heartbeat. Backups are change-driven but at most one per HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS (default 900, the 15-minute RPO; see "Minimum interval between backups"). It is not a blind full-dump timer: with no change, no backup is taken.

What moves the state marker

Only durable data moves the marker. Tables whose rows change without any user data changing are left out: STATE_MARKER_EXCLUDED_TABLES in src/heyaira/backup.py lists them. They are the shared work queue (control_work_items, control_work_events), the backup heartbeat evidence, Server Beat's check state, incidents, pulses, repair budgets and e-mail records, the notification queue and push deliveries (bridge_notification_events, bridge_mobile_deliveries), the log of refused Bridge requests (bridge_request_failures), Bridge liveness and executions (a running execution renews its lease every 20 seconds), self-renewing credentials (Bridge access tokens and rotations every 15 minutes, legacy Bridge credentials, per-execution MCP credentials, OAuth tokens and authorization flows), idempotency keys and audit_events, plus the unused alert tables of the 2026-09-20 release. If one of these moved the marker, the process that writes it would trigger backups by itself: an incident's push notification would cause a backup, and a running Bridge would keep causing them with every lease renewal.

Some of these tables also hold durable state in the same rows. Those changes are signalled through backup_state_signals (migration 036_backup_state_signals.sql), which is in the marker: the table stays empty, and each signal inserts and deletes one row so that its counters move.

  • Row triggers signal every change to bridge_executions except one that touches only lease_expires_at, updated_at or the unclaimed-offer alert's unclaimed_checked_at/unclaimed_alerted_at. Creating, claiming, finishing, cancelling and interrupting an execution, and its result, are backed up; a lease renewal is not.
  • Every change to bridge_nodes except the heartbeat's last_seen_at, last_sequence, health and updated_at: pairing and revoking a paired node are backed up, its heartbeats are not.
  • bridge_credentials: a new refresh credential and every update or deletion (revocations). The 15-minute access credential issued on each refresh is not.
  • The application calls backup_state_signal() for the two revocations a row trigger cannot tell from a rotation: an explicit OAuth revocation (/revoke) and a Bridge's own token revocation (/bridge/v1/tokens/revoke). Refresh-token rotation and Bridge token rotation are not backed up by themselves.
  • Revoking a project token (heyaira-admin revoke-token) or a Bridge identity (heyaira-admin revoke-bridge-identity) changes project_tokens or bridge_identities, which count.

audit_events stays excluded on purpose: a successful write's audit row comes with the change it records, but a rejected write's audit row alone does not trigger a backup.

Excluded tables are still in every dump; a change to them alone is only not a reason to take one. It is backed up with the next change that moves the marker. Everything else (memories, tasks and their history, workflows and their events, work sessions, projects and access, agent catalog, Bridge identities, devices and pairings, GitHub events) moves the marker. tests/test_backup_state_marker.py names every table on one side or the other, so a new table fails the suite until it is classified, and tests/test_backup_state_marker_durable.py drives the durable and the routine changes above through the real routes.

The backup worker claims only heyaira-backup items, and Server Beat's runner only server-beat-check and server-beat-action items, from the same control_work_items queue.

The service file is selected explicitly with service=<name> in the pg_dump/pg_restore arguments. This prevents libpq from interpreting the service label as a database name when the production database name differs from the service label.

Statistics resets. The counters only grow between two resets, and a reset (pg_stat_reset(), a per-table reset, or a PostgreSQL crash or immediate shutdown, which discards them) sets them back to zero. A marker built from the counters alone would then pass through values it had before the reset, match an earlier succeeded item and skip the backup. The marker therefore also digests the reset times: pg_stat_database.stats_reset (moved by pg_stat_reset() and by every per-table reset) and pg_stat_wal.stats_reset (moved when a crash discards the counters; the database's own reset time is then NULL). Neither moves with a write, so the backup's own bookkeeping still cannot trigger a backup. A reset alone changes the marker and costs one extra backup, and so does the first heartbeat after the release that added the reset times.

R2 layout and retention

Use a private R2 Standard bucket with explicit eu jurisdiction. Keep objects under separate UTC prefixes:

rolling/YYYY/MM/DD/HHMMSS/<backup-id>.dump.age
daily/YYYY/MM/DD/<backup-id>.dump.age
monthly/YYYY/MM/<backup-id>.dump.age

The local planner in src/heyaira/backup_retention.py proposes the finite policy: keep every verified rolling copy for 48 hours, promote one UTC daily representative for 30 days, and promote one UTC monthly representative for 180 days. Daily promotions are locked for at least 7 days. It plans per physical copy (one object plus its manifest), because a promotion keeps the backup ID: once a representative has its daily or monthly copy, its rolling copy past 48 hours is deleted; an existing promotion is recognised and not repeated, so its lock is not renewed; and a daily or monthly copy that is no longer its slot's representative, or is past its tier's window, is deleted. Copies younger than 48 hours, incomplete and locked copies are never returned for deletion. Provider lifecycle rules are only a safety net; they do not replace the tested one-per-slot selection.

A lifecycle rule must never delete a rolling copy before retention has promoted it. Retention promotes and deletes only in apply mode (HEYAIRA_BACKUP_RETENTION_APPLY=true; RcloneRetentionApplier.apply copies and deletes only when it is not a dry run). A review-only run, a stopped retention service and a failing or refused run promote nothing. Everything below assumes retention runs successfully in apply mode.

A backup becomes a daily or monthly representative only once it is older than the 48-hour rolling window (build_retention_plan picks representatives from backups whose created_at is before now - rolling_hours), and retention runs only every HEYAIRA_BACKUP_RETENTION_INTERVAL_SECONDS (six hours by default). With successful apply-mode runs, a rolling copy is therefore promoted between 48 and 54 hours after it was made; every run that fails, is refused or finds the lock taken pushes that back by one more interval. A 48-hour rule on rolling/ races every promotion and can delete a day's representative before its daily or monthly copy exists. The minimum is the rolling window plus one retention interval, 54 hours with the defaults; R2 counts lifecycle age in whole days, so never set rolling/ below 3 days. Use 7 days, which leaves room for a day of failed runs to be noticed.

The risk remains whenever retention is not applying: while it is in review-only mode (the state until deletion is enabled), stopped or failing, nothing is promoted, and a 7-day rolling/ rule deletes every rolling copy older than 7 days with no daily or monthly copy to replace it. Only the newest 7 days of history then exist. Do not add a rolling/ lifecycle rule before retention runs in apply mode, and watch its retention_review lines while the rule is in place.

Rules on daily/ and monthly/, if any, must not be shorter than those tiers' windows (30 and 180 days). With retention applying, their copies are written at promotion, after the backup's created_at that retention measures from, so such a rule fires only after retention has already deleted the copy; without it, the rule is what deletes them.

src/heyaira/backup_retention_runtime.py is the provider adapter. It accepts only a reviewed RetentionPlan, defaults to dry-run, and deletes only the explicit object key plus its manifest for approved deletions. All promotions run before any deletion. Retention never writes to a key that already exists under rolling/, daily/ or monthly/; that is what lets those prefixes carry an R2 bucket lock (below).

A promotion copies the encrypted object to the target key and writes the target's manifest. If the target object is already there with the source's size (a promotion retried after a crash), it is not copied again; with any other size the run stops (PromotionConflict). The target manifest is written only when it is absent.

A deletion is bound to its own mark object, retention-marks/<object_key>.json, holding object_key, backup_id, a fresh deletion_id and marked_at. Marks live in R2 next to the copies, not in PostgreSQL, because retention is deliberately independent of the database. The protocol:

  1. write the mark;
  2. read it back and require the same deletion_id, then delete the object;
  3. read it back again and require the same id, then delete the manifest;
  4. delete the mark.

A replaced or removed mark stops the deletion (StaleDeletion). The flock run lock (below) is what keeps runs apart; the read-back only makes each deletion act on its own mark. The next run finishes whatever an interrupted deletion left:

Left in the bucket Next run
mark, object and manifest an ordinary verified copy: it counts in that run's safeguards, locks and slot selection; if the plan deletes it, the protocol runs again with a new id, otherwise unmark deletes the mark only (the manifest is not touched)
mark and manifest, object gone clean-manifest: requires the same id, deletes the manifest, then the mark
mark only clean-mark deletes it
manifest, object gone, no mark missing-object problem: the run is refused
mark and object, no manifest marked-object-without-manifest problem: the run is refused
a manifest with retention_state or retention_deletion_id (an in-manifest mark of the code before this protocol) legacy-mark problem: the run is refused; see the migration below

It never calls rclone sync or deletes a whole prefix.

The heartbeat (see "Production heartbeat runtime") lists retention-marks/ after it has read the manifests and treats a marked copy as one retention is deleting.

Never use rclone sync for this history. Delete only explicit object keys returned by a reviewed retention plan. The plan must be dry-run reviewed against the actual bucket before any deletion capability is enabled.

heyaira-backup-retention lists rolling/, daily/, monthly/ and retention-marks/ with object sizes and R2 upload times, pairs every manifest with its encrypted object and its mark, and plans only over verified copies: the manifest names its own key, and the object is present with the manifest's encrypted_size_bytes. It prints retention_review backups=… copies=… keep=… promote=… delete=… marked=… clean=… problems=… orphans=… (marked counts marked copies that still exist, clean the clean-manifest and clean-mark actions), and one retention_problem line per inconsistency (a manifest without its object, a size or key mismatch, an unreadable manifest or mark, a marked object without its manifest, an in-manifest legacy mark, copies of one backup that disagree). An object without a manifest or a mark (an interrupted upload or promotion) is counted as an orphan and ignored. With --show it also prints one retention_action line per planned clean-manifest, clean-mark, unmark, promote and delete (object keys only, at most 500). It changes nothing unless --apply is given, and it refuses (retention_refused, exit 2) when a listing is incomplete, when the inventory has any problem, when the plan would keep no verified backup or delete the newest one, and when the policy would delete inside the bucket lock rules below (for example --rolling-hours 24).

--apply takes an exclusive flock on HEYAIRA_BACKUP_RETENTION_LOCK_FILE before it lists the bucket and holds it until the run ends. In Compose that file is /var/lib/heyaira-retention/apply.lock in the named volume retention-lock, which the retention service and every docker compose run retention … of the same Compose project mount. Every rclone process the run starts inherits the lock's file descriptor, and a flock is released only when the last process holding it exits. A run whose Python process dies while an rclone call is still running therefore keeps the lock until that call has finished, and a container that stops takes all of its processes, and the lock, with it. A run that cannot take the lock, or has no HEYAIRA_BACKUP_RETENTION_LOCK_FILE configured, prints retention_locked and exits 3 without reading or changing anything. Before every bucket write the run also checks that the lock file is still the file it locked: never delete apply.lock or the volume while retention is deployed. A review without --apply only reads the bucket and takes no lock.

What the lock does and does not cover:

  • It serializes --apply runs on one Docker host that use this Compose project's retention-lock volume: the service and docker compose run --rm retention heyaira-backup-retention --show --apply with the deploy's project name (app) and Compose files.
  • It does not cover a run on another host, a run under another Compose project name (a different volume), a heyaira-backup-retention started outside these containers with its own lock file, or any direct rclone (or R2 console/API) change to the bucket. Don't make such changes while retention may apply.
  • An HTTP request that rclone had already sent when it was killed can still complete at R2 shortly after the lock is released. Each deletion additionally reads its own mark's deletion_id back before deleting, which stops a stale deletion of a replaced or un-marked copy, but a check followed by a delete is not atomic on its own; the lock is what makes it safe.

Production runs it in the retention Compose service, which a plain up does not start (profile retention). It needs only backup.env (the R2 settings), retention.env (its own R2 key, see "R2 credentials" below) and its lock volume: not PostgreSQL and not the server. It runs every six hours (HEYAIRA_BACKUP_RETENTION_INTERVAL_SECONDS in backup.env changes that) and, until HEYAIRA_BACKUP_RETENTION_APPLY=true is set in backup.env, only logs the reviewed plan. Both variables are read from backup.env only; their defaults live in the container's shell command.

Deploying it with today's ops/deploy-heyaira.sh: the script starts the named services with --no-deps and then always checks the revision of the running server container. A retention-only release (--services retention) builds and starts the container but then fails with runtime revision mismatch unless the server already runs the same commit, and it does not move the current link. Until the script checks each named service's own revision, deploy retention together with the server from the same main commit:

./ops/deploy-heyaira.sh \
  --source /secure/heyaira-source \
  --sha <full-40-character-main-commit-sha> \
  --app-root /srv/heyaira \
  --config-dir /srv/heyaira/config \
  --compose-project app \
  --compose-override ops/openbao-production-compose.yaml \
  --compose-extra /srv/heyaira/config/compose.restart.yaml \
  --services "server retention"

Prerequisites: the commit is the main commit the server should run anyway, because the server is rebuilt and recreated with it (a short restart; PostgreSQL and the other services stay as they are). /srv/heyaira/config/backup.env exists with the R2 settings (the deploy links it into the release), and HEYAIRA_BACKUP_RETENTION_APPLY is absent or false for the first release. Run the same command with --dry-run first. Once per-service verification lands in the deploy script, --services retention alone is enough.

Enabling deletion: read at least one logged plan against the bucket, confirm that it reports no problems and that every listed deletion is a rolling copy older than 48 hours whose backup has a daily or monthly copy, or a tier copy that is superseded or past its window, then set the variable in backup.env and recreate the container. The environment is fixed when the container is created: docker compose restart keeps the old values and does not pick up a changed backup.env. Recreate it with the deploy's Compose invocation (project app, --env-file /srv/heyaira/config/.env, the release's compose.yaml, the OpenBao override and the extra files): … --profile retention up -d --force-recreate --no-deps retention. A recreate stops the old container and every process in it before the new one starts, so an apply run cut short there is finished or replanned by the next run.

R2 bucket lock and lifecycle rules

r2_bucket_lock_rules() in src/heyaira/backup_retention.py is the lock configuration, checked against the policy:

Prefix Bucket lock Earliest deletion by retention
rolling/ 1 day 48 hours (the rolling window)
daily/ 6 days 7 days (daily_lock_days: the locked_until a promotion writes, counted from when it writes the manifest)
monthly/ none at once, when a newer representative supersedes it
retention-marks/ none at the end of every deletion

An R2 lock is age-based from each object's upload, and every lock must end before retention may delete under it, or R2 refuses the delete (ObjectLockedByBucketPolicy) and every run stops there. The check refuses any lock on monthly/ or retention-marks/ and any lock not strictly shorter than the earliest deletion. Add the rules in the R2 dashboard (the bucket's lock rules) only after the migration below.

Two further guards keep a deletion out of the lock even when the run itself is late or its clock is off:

  • A daily copy's locked_until is the time its manifest is really written, after the copy, plus daily_lock_days; a run that stalls between planning and writing does not shorten it.
  • Retention lists the bucket with rclone lsf --use-server-modtime --format pst (R2's LastModified, in UTC) and never plans to delete a rolling/ or daily/ copy whose object or manifest R2 stored less than that tier's lock plus one hour ago.

--now sets the clock for a review only; with --apply the command refuses it (retention_refused, exit 2), because a backdated plan would delete copies R2 still locks.

r2_lifecycle_rules() returns the lifecycle minimums, never the selection: rolling/ 3 days (a whole day past the rolling window, and never less than 3 days), daily/ 30, monthly/ 180, and incomplete multipart uploads after 1 day. For rolling/ use 7 days and add it only once retention applies, as explained at the start of this section.

Migrating to retention marks

Deployed retention from before this protocol keeps its marks inside the manifests, and the new code refuses any such manifest (legacy-mark). In this order:

  1. While the bucket lock is still off, let the current code finish its interrupted deletions: run it with --apply (or wait for the service's next applying runs) until its review shows marked=0 clean=0.
  2. Deploy the new code and confirm its review shows problems=0.
  3. Only then add the lock rules above in the R2 dashboard.

Never add a lock while a release before this protocol may still apply: it rewrites manifests when it deletes, and the lock refuses that.

Restore procedure

  1. Select a verified manifest and record its object key, backup ID, covered application-state marker, PostgreSQL version and service revision.
  2. Run RestoreWorker on a separate trusted executor with no route to the production volume or public port. The executor temporarily receives the private age identity; the application VPS does not receive it.
  3. Download the encrypted object with rclone copyto, verify its SHA-256 and byte size against the manifest, decrypt with age, then run pg_restore --exit-on-error --no-owner against an isolated PostgreSQL service.
  4. Verify schema, constraints, schema_migrations, and selected MCP reads using the code revision recorded in the manifest. Do not activate restored production tokens, OAuth grants or instance identity.
  5. Record elapsed time from object/key access to a usable isolated service, plus the exact manifest ID. A weekly restore is not considered active until an external executor has run this procedure successfully on schedule.

Credentials after a production restore

A restored database knows only what was in its dump. Before a restored service is activated in place of production:

  • Credentials issued or rotated after the dump are unknown to it: OAuth clients authorize again, Bridges recover or re-enroll and legacy nodes pair again.
  • Revocations after the dump are not in it. Revocations move the state marker, so they are in every backup taken after them, but one made after the newest backup (within the backup's RPO) is lost; so is a token that a rotation revoked after the dump, which looks active again (OAuth refresh tokens live 30 days; Bridge access tokens expire 15 minutes after the dump at the latest).
  • Re-check every revocation since the dump's created_at and revoke again (heyaira-admin revoke-token, heyaira-admin revoke-bridge-identity, heyaira-admin revoke-bridge-node), using the failed instance's audit trail and logs where they can still be read and the owner's own record. The restored audit_events ends at the dump too. If that cannot be established, revoke every OAuth grant and Bridge credential and let clients authorize again.

The repository CLI is heyaira-restore. It requires an explicit manifest, an isolated --restore-target, a protected age identity and a PostgreSQL service file for the isolated target. It prints only restore metadata; the private identity and R2 credentials never appear in its arguments or output.

Activation gate

Before production activation, an operator must provide the bounded approval and concrete non-secret identifiers for: R2 account/bucket with EU jurisdiction, sender credential scope, public age recipient, private-key custody and emergency copy, isolated restore executor, external monitor and alert recipient. Then verify the actual tool versions, service files, permissions, lock/lifecycle rules, and rollback. Only after a real encrypted copy and isolated restore should the independent monitor be enabled.

The monitor must alert on both an explicit failed run and a stale last-success manifest (working threshold: 30 minutes, to be calibrated by measurement). It must emit success only after upload and verification. A host-down signal must come from outside the VPS. The task remains incomplete until two real scheduled executions, a restore, and an alert delivery are evidenced.

The read-only heyaira-backup-health CLI reads only the durable heartbeat and terminal work-item outcomes. An independent host may invoke it with a database connection and use exit code 2 plus its JSON reasons for alerting. It is not itself the independent delivery channel; recipient and host-down transport remain operator configuration.

The provider-neutral decision logic is in src/heyaira/backup_monitor.py. It produces separate failure, stale-backup and stale-host reasons without delivering anything itself; the final adapter must use an independent monitor and an operator-supplied recipient.

Production heartbeat runtime

heyaira-backup runs two loops in the backup Compose service. The heartbeat polls every 10 seconds by default and only compares the current PostgreSQL application-state marker with the newest verified R2 manifest. A changed marker creates or coalesces one durable heyaira-backup work item in control_work_items. The worker claims that item independently and runs the dump/encrypt/upload pipeline. A heartbeat tick never performs a full dump merely because time passed. The interval is a responsiveness setting and may be raised after measuring the observed RPO and R2 list latency.

Minimum interval between backups

User tables change almost all the time while agents and CI are busy (tasks, task versions, GitHub webhook events and workflow runs), so a backup on every uncovered heartbeat meant a full dump and R2 upload many times an hour (about 1,500 a day on 2026-10-02/03, roughly 50 GB a day into R2). The heartbeat therefore does not queue a change-driven backup while a backup of the scope finished successfully less than HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS ago (default 900, accepted 1 to 21600; the runtime refuses to start outside that range). The age is now() - finished_at on PostgreSQL's clock; the runtime's own clock is never used, and a success dated in the future (the database clock was set back) does not count. A queued backup needs no wait: the heartbeat coalesces the new marker into it (a coalesced work event), so it dumps the newest state when a worker claims it. A running backup is left as it is: its manifest records the marker the item held when it was claimed (src/heyaira/control_loop.py, PostgresWorkStore.ensure updates only a queued item), the heartbeat queues nothing beside it (one item per scope), and once it succeeds a marker that moved meanwhile is a change like any other and waits for the interval, counted from that success. A change made while a backup runs can therefore wait up to the running backup's duration plus the interval. The queue checks the interval again when it would insert the item, under a per-scope advisory lock that completing an item also takes, so a backup that succeeds between the heartbeat's check and the insert still holds the next one back. Once the interval has passed and the marker is still uncovered, the next heartbeat queues one backup of the state as it is then, so every change made during the wait is in it.

Two cases never wait:

  • The first backup of a scope, or the first after a quiet period: no backup succeeded within the interval.
  • A replacement for a newest copy that R2 lacks: its object is missing or has the wrong size, or the newest backup that had succeeded before the listing has no copy at all (the supersession described below). This applies only when the R2 listing that found it started after every recent success; the heartbeat reads the recent successes just before each listing. A cached listing that may predate the replacement (or any newer backup) therefore never queues another one; it waits for the interval like a change.

backup.fresh in Server Beat allows for the wait: it never fails an uncovered state sooner than the interval plus 15 minutes (see "Alerts"), so Server Beat's service gets the same HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS in Compose.

Set it in the deploy's Compose .env (/srv/heyaira/config/.env, the file passed with --env-file), not in backup.env. compose.yaml gives both the backup and the server-beat-runner service an environment entry HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS: ${HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS:-900}, interpolated from that .env, and an environment entry overrides the same variable from an env_file: a value in backup.env is silently replaced by 900 and never reaches Server Beat at all. After changing it, recreate both containers (… up -d --force-recreate --no-deps backup, and the same for server-beat-runner with --profile server-beat), so that the backup runtime and backup.fresh use the same value.

The heartbeat lists rolling/, daily/ and monthly/ with sizes (rclone lsf --recursive --format ps), reads the manifests, then lists retention-marks/ and ignores every copy that has a deletion mark there (a deletion that began before a manifest was read still has its mark then: retention deletes the mark last), deduplicates by backup ID and uses the newest recorded created_at to find the covered marker even after retention removes its rolling copy. A copy counts only when its encrypted object is in the same listing with the encrypted_size_bytes its manifest records; a manifest whose object vanished is not coverage. Retention may run while the heartbeat reads: a manifest that no longer downloads, or a copy marked for deletion that outranks every counted backup, makes the heartbeat list all tiers once more. A manifest that second listing lacks was deleted by retention and is skipped (never missing-object, never a failed tick); one it still has fails the read as before; and a copy retention promoted after the first listing (it promotes before it deletes the source, and never deletes the newest backup) is read and counted, so the newest backup never looks lost and nothing is replaced or superseded. When the newest retained backup has no such copy (missing-object or size-mismatch), or its copy is gone altogether, the heartbeat queues a new backup of the current state even though the marker did not change: the job that once succeeded for that marker is superseded (its dedupe key gets a :superseded:<id> suffix and a superseded event; its history stays) when it had already succeeded before the R2 listing started. That is decided in PostgreSQL, not by clocks: before each new listing the heartbeat reads the ids of the jobs that have succeeded for the current state, and that listing can supersede only those exact jobs; a job that succeeds later (a replacement, or one finishing out of order) never is. At most one backup job per scope can be queued or running (a unique partial index, migration 041). "Newest" in R2 is lineage, not wall time: each manifest records its job's seq (job_seq, the job's enqueue order), which ranks above created_at, so a replacement made while the clock runs behind still outranks the damaged copy it replaces; manifests without job_seq rank below all that have it. A database restored from an older dump numbers its jobs from that dump's sequence again, below copies R2 already holds, and its queued or running backups keep their old seq. So each tick, before it can queue a job, reconciles the queue with the highest job_seq the last R2 listing found, in one transaction under an advisory lock that every runtime takes: the control_work_items sequence moves to the greater of its value and that watermark (never back), and every queued or running backup at or below it gets a new seq above it. While it changes anything it also holds a SHARE ROW EXCLUSIVE lock on control_work_items, so no insert draws a value in between; the usual tick has nothing to change and takes no table lock. The worker claims no job before the first such reconciliation, and it publishes a manifest under the seq its item has at that moment, so the first backup after a restore ranks newest. Each tick records what the last listing found, with PostgreSQL's time and a check number from the backup_r2_check_seq sequence, both taken before the listing, for backup.fresh. Evidence only moves to a higher check number, never by timestamp (two runtimes overlapping in a rollout, or a database clock reset), while the heartbeat time always moves. backup.fresh ages the check on PostgreSQL's clock and fails closed on a check dated more than a minute in the future. It re-reads R2 at most every HEYAIRA_BACKUP_MANIFEST_POLL_SECONDS (default 300) and uses the last result in between. After a backup that completed successfully, that stale value does not cause a second backup of the same state: the new observation has the same state:<marker> dedupe key as the item that already succeeded, and the queue keeps one item per key. The listing is read whole (the reader's runner has no output limit; the worker's runner keeps its 64 KiB bound), and a listing that comes back incomplete (marked truncated, or not ending with rclone's final newline) is refused: the tick fails and writes no heartbeat evidence, so Server Beat's backup.fresh fails once the heartbeat is five minutes old, instead of the heartbeat comparing against an older manifest.

Known limits of the work queue

The one-item-per-marker rule holds for items that completed successfully. It does not cover these cases, which predate the current runtime and are detected, not prevented:

  • A worker that dies after it published the object and manifest but before it marked the item succeeded leaves the item running. A running worker renews its 20-minute lease every 5 minutes, so a slow but healthy job (up to six tools of up to 15 minutes each) is never reaped. A renewal that finds the item no longer its own (reaped, or taken by another runtime) cancels the job: the running pg_dump, age or rclone is killed, and the worker re-checks that it still holds the lease right before it uploads the object and again before the manifest, so it publishes nothing after losing it. Once a dead worker's lease expires the item is queued again and uploads the same state a second time under a new backup ID. Both copies are valid; retention treats them as two backups.
  • An item that failed on its last attempt stays failed. The same marker maps to the same item, so neither the heartbeat nor a restart of the backup service retries it; backup.fresh fails (a failure newer than the last success) until the next change moves the marker and a new item succeeds.
  • The unencrypted dump exists on disk only in the job's private heyaira-backup-* directory under HEYAIRA_BACKUP_WORK_DIR (default /tmp/heyaira-backup inside the container), until age has encrypted it. A worker killed in between (OOM, SIGKILL, host crash) leaves it there; the runtime deletes every such leftover directory when it starts, before its worker claims a job. The work directory must therefore belong to one runtime.

The backup service receives R2 credentials through the host-only backup.env file next to compose.yaml and maps them to rclone's environment-based S3 configuration. The file is not part of the repository. The application container never receives the R2 secret. Only the public age recipient is provided to the backup service; the private identity remains on the isolated restore executor.

The backup image pins PostgreSQL client 16 because the production PostgreSQL server is 16. The pg_dump and pg_restore major versions must match the production server for a portable custom-format dump and an isolated restore.

The rclone environment sets no_check_bucket=true. The R2 token is scoped to the selected bucket/object API, and this avoids a bucket-level preflight that Cloudflare rejects for object-scoped tokens.

The production host keeps backup.env at mode 0600. The repository's secrets/age/ directory is ignored by Git and is intended for the private restore identity only; it must never be copied to the application VPS.

Custody of the private age key: the emergency custody copy is the owner's entry in Apple Passwords, synced through iCloud. The file ~/.config/heyaira/backup-age-identity.txt (mode 600) on the owner's Mac is only the working copy the weekly restore rehearsal (ops/restore-rehearsal/) reads; if that Mac is lost, recreate the file from the Apple Passwords entry. Neither copy is ever put on the production server, in Git, in HeyAira or in a chat.

R2 credentials: one key writes, another deletes

Two R2 keys, one per role:

Role Variables File Read by Needs
Backup writer HEYAIRA_BACKUP_R2_ACCESS_KEY_ID, HEYAIRA_BACKUP_R2_SECRET_ACCESS_KEY backup.env backup (always running), restore, and the retention fallback put, list, get
Retention HEYAIRA_RETENTION_R2_ACCESS_KEY_ID, HEYAIRA_RETENTION_R2_SECRET_ACCESS_KEY (optional HEYAIRA_RETENTION_R2_ENDPOINT) retention.env (see retention.env.example) retention only list, get, put (manifests, promotions), delete

Both files sit next to compose.yaml in the config directory at mode 0600; every deploy links both into the release even before they exist (Compose skips a link whose file is missing), and --redeploy adds links an older release lacks, so a retention.env created after a deploy is used as soon as the retention container is recreated. Retention uses its own key when both retention variables are set, refuses to start with only one of them, and otherwise falls back to the backup key and prints retention_credentials_shared on every run. The backup service logs backup_sees_retention_credentials if the retention key ever reaches it. Every rclone child gets only its own role's key.

Creating the tokens in Cloudflare (R2 > Manage R2 API Tokens > Create API token, one token per role, each limited to the backup bucket only, with "Apply to specific buckets only" set to heyaira-backups-prod-eu, and TTL "Forever" unless rotation is planned):

  • Backup writer: permission Object Read & Write, bucket heyaira-backups-prod-eu only. R2 has no permission that writes without deleting: Object Read & Write is the narrowest write permission and also allows DeleteObject. What the split buys is that the delete-capable retention key is no longer on the always-running service. The key can be rotated or revoked on its own, and it can move to another host. Bucket lock rules are listed under "R2 bucket lock and lifecycle rules" and are added only after "Migrating to retention marks". Lifecycle rules are the outer bound, with the minimums in "R2 layout and retention" (rolling/ at least 3 days, 7 recommended, and only once retention runs in apply mode).
  • Retention: permission Object Read & Write, same single bucket. It needs delete. Put it only in retention.env. Later, the retention service can run on a separate host that holds only this key and needs no PostgreSQL.
  • Restore executor (optional, recommended): Object Read only, same bucket. The restore host never needs to write.

After creating the retention token: write retention.env, recreate only the retention container (see "Enabling deletion" for the command), and confirm that its next retention_review line comes without retention_credentials_shared. Then replace the backup writer's token with a new one so the old shared key, which retention used until now, is revoked:

  1. Create the new backup writer token (as above) and put its key into backup.env in place of the old one.
  2. Recreate the backup container, which reads backup.env only when it is created (docker compose restart keeps the old key): the deploy's Compose invocation (see "Enabling deletion") with … up -d --force-recreate --no-deps backup. The restore rehearsal's server-helper.sh reads R2 through this container too, so it uses the new key from then on.
  3. Confirm the new key works before revoking the old one: the new container's first heartbeat lists R2 (a new process has no cached listing), so r2_checked_at in backup_runtime_heartbeats moves past the recreate time and r2_problem stays empty, and the backup logs show no heartbeat_tick_failed.
  4. Only then revoke the old token in Cloudflare. Revoking it first leaves the running container with a dead key: every R2 listing fails and backup.fresh fails until the container is recreated.

Alerts

Backup alerts come from Server Beat, not from the backup service. Its backup.fresh check (every 5 minutes) fails when there is no verified backup evidence, when the last backup run failed after the last success, when the backup heartbeat is older than 5 minutes, or when the state is not covered and the last success is older than HEYAIRA_SERVER_BEAT_BACKUP_MAX_AGE_SECONDS (default 1800), or than HEYAIRA_BACKUP_MIN_INTERVAL_SECONDS plus 15 minutes when that is longer: the backup runtime waits that interval between backups, so an uncovered state is normal for that long. "Covered" means R2 holds the encrypted object, not that a job once succeeded: Server Beat has no R2 credentials, so it reads the R2 evidence the backup heartbeat records in backup_runtime_heartbeats (r2_checked_at, r2_covered_marker, r2_problem) and also fails when that evidence is older than HEYAIRA_SERVER_BEAT_BACKUP_R2_MAX_AGE_SECONDS (default 900; keep it above HEYAIRA_BACKUP_MANIFEST_POLL_SECONDS), when no backup with its object exists, or when the newest retained manifest's object is missing or has the wrong size. A failure opens an incident and its recovery closes it; both are delivered to the owner as an iPhone push first and by SES e-mail when there is no push route or the push is not confirmed within HEYAIRA_SERVER_BEAT_PUSH_CONFIRM_SECONDS (default 60). Each incident event is e-mailed once, under a reclaimable claim in control_email_events (a crash between sending and recording can repeat it once). See docs/SERVER_BEAT_RUNBOOK.md for the configuration (server-beat and server-beat-runner services, HEYAIRA_SERVER_BEAT_SES_CONFIG_FILE). Because Server Beat runs outside the backup container, it also reports a backup service that has stopped.

Status reports: retention and the restore rehearsal

Retention and the owner's restore rehearsal have no database access, so each leaves its outcome in the bucket's status/ prefix, which neither retention nor the backup reader treats as a tier:

  • status/retention.json: every --apply run that held the lock writes {"at": …, "result": "ok" | "refused" | "failed"} at its end (a review writes nothing). A refusal is what one bad object (a manifest without its object, a size or key mismatch, an unreadable manifest, conflicting copies) causes on every run until someone repairs or removes it; the retention_problem lines in the retention log name it.
  • status/restore-rehearsal.json: the weekly rehearsal on the owner's Mac reports PASS or FAIL through server-helper.sh report (ops/restore-rehearsal/README.md).

The backup runtime lists status/ at most every HEYAIRA_BACKUP_MANIFEST_POLL_SECONDS and records what it found in backup_runtime_heartbeats.status_reports (migration 047), merged with what it saw before: a report that was seen once and is gone now is kept as missing. A listing that fails, comes back truncated or has an unreadable line changes no report; it is recorded as a failed read (status_failed_at, next to status_checked_at, the last complete read). A status/ prefix that does not exist yet lists as empty. Once backups are otherwise fresh, backup.fresh turns WARN, which notifies the owner like disk.free and tls.expiry do, when no read of status/ has succeeded for an hour after a failed one, when a report has disappeared, when the last retention run was not ok or is more than 2 days old, or when the last rehearsal was not PASS or is more than 8 days old. A report that was never written (the job is not deployed) does not warn. The backup writer's R2 key reads status/ and the rehearsal writes through it, and retention writes with its own key, so both keys need object read and write on status/ as well as on the tiers. A report that cannot be written is logged (retention_status_unwritten) and never changes the run's own result.

The 2026-09-20 release had its own SES fallback inside the backup service (backup_alerts.py, branch codex/backup-production-close). It is not on main and has not run in production since the backup service was recreated from main on 2026-09-28. Its tables, backup_alert_states and backup_alert_events (migrations 013_backup_alerts.sql and 014_backup_alert_email_leases.sql, kept under their original names and contents), exist so that a fresh database has the same schema as production; nothing writes them.