Drum beat / control loop

Purpose

The drum beat is HeyAira's reusable control loop for cheap observation and durable work coordination. It is not a backup timer, a model router, or a notification heartbeat.

The existing Bridge heartbeat remains a liveness signal for a Bridge node. The control loop is a separate domain-neutral primitive that future consumers may use, including the backup worker.

Contract

heartbeat tick
    -> observe a cheap durable marker
    -> ensure(work_type, scope_key, dedupe_key)
    -> claim with a lease
    -> execute in a separate worker
    -> verify the result
    -> record outcome and covered marker
    -> retry or notify on failure

An observation contains:

• work_type: the domain consumer, for example heyaira-backup;
• scope_key: the protected resource or project;
• dedupe_key: one logical state change or coalesced range;
• observed_marker: the exact durable state the worker is expected to cover;
• bounded, non-secret payload metadata.

The unique (work_type, scope_key, dedupe_key) key makes repeated heartbeat ticks idempotent. For coalescing consumers such as backup, the store also keeps at most one active item per (work_type, scope_key): a queued item is advanced to the newest marker, while a running item keeps the marker of its already-started snapshot. A worker lease prevents two workers from executing the same job concurrently. An expired lease returns a job to the queue. Completion, failure, and coalescing are recorded as work-item events; a terminal failure is not reported as success.

Backup consumer

The backup observer compares the current HeyAira state marker with the marker covered by the last verified external manifest. If the marker is unchanged, the heartbeat does no work. If it changed, it creates one coalesced backup job. The worker performs pg_dump -> age -> R2, verifies the object and manifest, and only then records the marker as covered.

The observer may run every few seconds or every minute. That controls reaction latency and lease recovery; it does not cause a dump on every tick. A burst of writes produces one job for the selected coalescing boundary, subject to the accepted RPO and measured dump/upload time.

The backup ledger and manifest must remain outside the PostgreSQL database being protected. The control-loop tables may record job state, but an R2 manifest is the recovery evidence if PostgreSQL is unavailable.

Safety requirements

• No secrets, dump contents, private keys, or authorization headers in payloads, results, events, or logs.
• Workers must verify ownership before completing or failing a leased item.
• Retries use the same dedupe key and never overwrite a successful result.
• A worker must not mark a state marker covered before upload and verification.
• The loop must distinguish idle state from an overdue backup after a change.
• Backup retention is handled by the R2 retention design, not by the tick rate.

Current implementation boundary

src/heyaira/control_loop.py provides the contract, PostgreSQL queue/lease store, tick coordinator, and one-worker dispatch. Migration 010_control_loop.sql creates the durable work-item and event ledger. src/heyaira/backup.py adds the first consumer boundary: a PostgreSQL application-state marker observer (DML counters excluding control tables) and secret-free manifest builder. The R2 worker, lifecycle configuration, restore and monitoring remain subsequent stages of BACKUP-2 through BACKUP-5.

src/heyaira/backup_worker.py now implements the local worker pipeline with fixed argument-list execution: pg_dump reads a protected PostgreSQL service file, age encrypts the dump for a public recipient, rclone copyto uploads the encrypted object and manifest, and rclone size --json verifies both remote object sizes. The plaintext dump exists only inside a bounded temporary directory and is removed before upload; command output and secrets are not logged. Real tool paths, R2 credentials, and production execution remain deployment configuration rather than repository contents.