Skip to content

Server Beat / Bridge Beat

Purpose

HeyAira uses one reusable Server Beat rhythm for cheap observation and durable work coordination. A Bridge Beat is the liveness signal sent by a Bridge node (the iPhone channel); it is an input to Server Beat, not a second server scheduler. The server beat only knocks: it records a pulse and creates bounded, durable work. A separate runner performs checks/actions under leases.

The beat is not a backup timer, model router, or notification heartbeat.

The Bridge heartbeat remains a liveness signal for a Bridge node. A transition to a failing check opens one incident and queues one notification action. A return to OK closes that incident and queues one recovery notification. Repeated ticks do not duplicate incidents, actions, or mobile deliveries.

Contract

heartbeat tick
    -> observe a cheap durable marker
    -> ensure(work_type, scope_key, dedupe_key)
    -> claim with a lease
    -> execute in a separate worker
    -> verify the result
    -> record outcome and covered marker
    -> retry or notify on failure

An observation contains:

  • work_type: the domain consumer, for example heyaira-backup;
  • scope_key: the protected resource or project;
  • dedupe_key: one logical state change or coalesced range;
  • observed_marker: the exact durable state the worker is expected to cover;
  • bounded, non-secret payload metadata.

The unique (work_type, scope_key, dedupe_key) key makes repeated heartbeat ticks idempotent. For coalescing consumers such as backup, the store also keeps at most one active item per (work_type, scope_key): a queued item is advanced to the newest marker, while a running item keeps the marker of its already-started snapshot. A worker lease prevents two workers from executing the same job concurrently. An expired lease returns a job to the queue. Completion, failure, and coalescing are recorded as work-item events; a terminal failure is not reported as success.

Backup consumer

The backup observer compares the current HeyAira state marker with the marker covered by the last verified external manifest. If the marker is unchanged, the heartbeat does no work. If it changed, it creates one coalesced backup job. The worker performs pg_dump -> age -> R2, verifies the object and manifest, and only then records the marker as covered.

The observer may run every few seconds or every minute. That controls reaction latency and lease recovery; it does not cause a dump on every tick. A burst of writes produces one job for the selected coalescing boundary, subject to the accepted RPO and measured dump/upload time.

The backup ledger and manifest must remain outside the PostgreSQL database being protected. The control-loop tables may record job state, but an R2 manifest is the recovery evidence if PostgreSQL is unavailable.

Safety requirements

  • No secrets, dump contents, private keys, or authorization headers in payloads, results, events, or logs.
  • Workers must verify ownership before completing or failing a leased item.
  • Retries use the same dedupe key and never overwrite a successful result.
  • A worker must not mark a state marker covered before upload and verification.
  • The loop must distinguish idle state from an overdue backup after a change.
  • Backup retention is handled by the R2 retention design, not by the tick rate.

Current implementation boundary

src/heyaira/control_loop.py provides the contract, PostgreSQL queue/lease store, tick coordinator, and one-worker dispatch. Migration 010_control_loop.sql creates the durable work-item and event ledger. src/heyaira/backup.py adds the first consumer boundary: a PostgreSQL application-state marker observer (DML counters excluding control tables) and secret-free manifest builder. The R2 worker, lifecycle configuration, restore and monitoring remain subsequent stages of BACKUP-2 through BACKUP-5.

src/heyaira/backup_worker.py now implements the local worker pipeline with fixed argument-list execution: pg_dump reads a protected PostgreSQL service file, age encrypts the dump for a public recipient, rclone copyto uploads the encrypted object and manifest, and rclone size --json verifies both remote object sizes. The plaintext dump exists only inside a bounded temporary directory and is removed before upload; command output and secrets are not logged. Real tool paths, R2 credentials, and production execution remain deployment configuration rather than repository contents.

Server Beat registry and escalation

The first Server Beat package keeps the layers explicit:

  1. L0 supervisor: Compose restart policies and user systemd supervise crashed processes; the higher layers handle live-but-unhealthy processes.
  2. L1 local checks/actions: application health, PostgreSQL readiness, configured containers, disk, TLS expiry, backup freshness, OpenBao seal status and liveness pulses. OpenBao sealed is alert-only; it is never auto-unsealed. Bounded actions include one allow-listed container restart, only when an operator explicitly enables it. The default Compose services receive no Docker socket. The activation overlay exposes only the private host repair broker, which validates exact container and Compose labels, then rechecks health. Budget is shared across incidents/processes; exhaustion stops repair until the target has been continuously healthy for one hour (unobserved time does not count), then it re-arms itself.
  3. L2 on-host Watchdog: an independent user systemd service with no database access. It watches the pulse files of the Beat and of the runner (a shared host directory) and restarts the stale container through that broker; after three attempts it persists stop and e-mails the owner directly. When the Beat is alive but reports that PostgreSQL is unreachable, it e-mails that instead of restarting anything. It writes its own pulse file; the runner restores it through the fixed user-systemd broker operation.
  4. L3 external Watchdog: a Cloudflare Worker receives periodic pulse-out, independently probes HTTP on a one-minute cron and requires three failed probes. Healthy HTTP never triggers reboot for a missing pulse. Maximum 1 attempt/hour and 2/day, then persistent stop and independent SES alert. Reboot remains disabled by default.
  5. L4 human: iPhone push first; the push counts only once the phone delivery is confirmed. If it is not confirmed within 60 s (the push sender lives in the application, which may be what is down) or there is no push route, the event is e-mailed. Notification jobs retry until delivered and are never counted as repair escalations. A sleeping Mac does not suppress the phone queue. Missing channels fail visibly, not silently.

An incident opens after two consecutive bad observations (one blip neither pages nor restarts) and closes after two minutes of continuous health, so a flapping check is one incident with one "down" and one "recovered". While a check is unsettled it is re-probed every 30 s, also the daily TLS check. Warnings (disk under 15 %, TLS under 14 days) open a warning incident and notify once; a later failure of the same incident sends "down".

The primary human channel is a queued Bridge notification delivered to the connected iPhone. The project is never inferred from “the first row” in the database: HEYAIRA_SERVER_BEAT_NOTIFICATION_PROJECT_ID must explicitly bind Server Beat to the target project. If the binding is absent or invalid, Bridge fan-out is skipped and the configured fallback is attempted. No credential, authorization header, APNs token or repository content is stored in beat state, jobs, incidents, logs or HeyAira.

The registry is in src/heyaira/server_beat.py:

Check Cadence Default action
openbao.sealed 30 s notification on incident; never auto-unsealed
app.healthz 30 s notification; container restart when repair is activated
postgres.ready 30 s notification on incident
containers.running 30 s broker failure notifies; a stopped container is informational (app.healthz covers the app, the on-host Watchdog the Beat and runner)
disk.free 5 min warning under 15 %, incident under 5 %
tls.expiry 1 day warning under 14 days, incident under 3 days
backup.fresh 5 min notification on incident; warning when retention or the restore rehearsal reports a problem (docs/BACKUP_RUNBOOK.md)
watchdog.alive 30 s notification; Watchdog restart when repair is activated
bridge.online 30 s alerts after prolonged loss of contact outside an announced sleep; a sleeping Mac is normal; OK when no Mac is configured

bridge.pickup and bridge.polling are retired (S2, decisions a9f0e60e and 6cab5c63); historical check states remain untouched.

Adding a check or action means adding one bounded registry entry and tests; it does not create another scheduler. The Server Beat service and runner have separate Compose resource limits. The runner claims only server-beat-check and server-beat-action work types, so it cannot steal backup or unrelated control-loop work. The backup runner likewise claims only heyaira-backup work. Migration 026 adds cross-incident repair budgets, email dedup and a pending-check constraint; 028 adds the incident lifecycle columns, e-mail claims and the indexes that keep every tick an index lookup. The runner deletes finished Server Beat work after 14 days and recovered incidents after 180 days. See the limits (approved by the owner on 2026-09-27; tuned from measured data).

Migration 025_server_beat.sql adds check state, incident/event, and pulse tables. heyaira-server-beat --mode beat is the cheap scheduler and heyaira-server-beat --mode runner is the bounded worker. The on-host Watchdog is heyaira-server-beat-watchdog with its unit template at ops/systemd/heyaira-server-beat-watchdog.service. The external Worker template is in ops/cloudflare/server-beat-watchdog/. See the Server Beat runbook for activation gates and the FLOW-08 evidence checklist.