Retry scan fairness and expiry: worker evidence

Task: ce8e2cf3-0bc4-4493-a88e-f89c0d50944c, frozen version 3. Execution: 77ddd928-2162-4fd2-b643-fd7fd3b1c1b7. Repository: h8v6/heyaira; branch: heyaira/ce8e2cf3. Source and starting HEAD: 422ee9a9f04ff89a04b8dbbadcdc3b9e5bf8dec9. Working directory for every command below: /Users/jablonskib/HeyAira-agents/.heyaira-worktrees/heyaira/ce8e2cf3-worker. The resulting commit SHA is recorded in the execution's RESULT/receipt; this file is part of that commit. Subagents: none.

Cause and change

The base repeatedly selected the earliest 32 due rows before discovering that their tasks were locked. Neither their scan order nor their eligibility changed. It also excluded expired rows when their next retry time was still in the future.

Migration 060 adds retry_checked_at and orders the partial retry index by that timestamp (nulls first), then retry_at,id. Selection uses the same persistent ordering. Each visited row is stamped by an autocommitted pool UPDATE before its per-item transaction, following the waiting-corrections pattern. A skipped task lock or rolled-back item cannot undo the rotation. Transactional task SKIP LOCKED is not used. The per-item lock order remains subject, task, binding, capacity, identity.

Both candidate selection and the recheck under the task lock admit due retries or rows past their deadline/24-hour failure window. _offer_retry already checks expiry before authority, node availability and capacity; that check and park() are unchanged. _defer_retry and the retry policy remain unchanged. Unvisited future retries eligible only for expiry precede already visited rows.

The regressions create genuine preparation failures through MCP/Bridge ASGI, then hold the first 32 task locks on another PostgreSQL connection across both passes. T1 checks the 33rd row is queued on the second pass and the 32 locked rows remain waiting. T2 places two future retries behind the locked tasks, expires one by deadline and one by first-failure age, and makes their node offline. It checks both park with the exact reason, clear retry_at, retain their execution IDs, and each emit one parked notification.

K1: T1 base failure and fixed retry suite

Frozen method: python scripts/test_postgres.py tests/test_workflow_retry.py -p no:cacheprovider on 422ee9a9 (with the new test) and HEAD; RESULT states both.

Base run: HEAD was 422ee9a9 with only the new regression tests added; workflow.py and migration 060 were unchanged.

TMPDIR="$PWD/.venv/task-tmp" python scripts/test_postgres.py tests/test_workflow_retry.py -p no:cacheprovider

Exit 1, actual output excerpts (synthetic fixture headers omitted):

test_retry_scan_promotes_33rd_behind_32_locked_tasks
> assert await workflow.offer_retries(pool) == 1
E assert 0 == 1

test_retry_scan_expires_future_retries_behind_32_locked_tasks
> assert parked['phase'] == 'blocked'
E AssertionError: assert 'retry_wait' == 'blocked'

FAILED tests/test_workflow_retry.py::test_retry_scan_promotes_33rd_behind_32_locked_tasks
FAILED tests/test_workflow_retry.py::test_retry_scan_expires_future_retries_behind_32_locked_tasks
2 failed, 24 passed in 65.01s (0:01:05)

Same command against the fixed candidate's code, before the local commit: exit 0, actual final output:

26 passed in 141.93s (0:02:21)
postgres_acceptance=passed: selected tests, zero skipped tests

The tested runtime, migration and regression bytes are committed unchanged. Their SHA-256 digests were checked before committing:

src/heyaira/workflow.py: 805e69d5115706acbdfb013489aa0ebb06bfc1ea7d4cf406e7df77c23104b36c
migrations/060_work_queue_retry.sql: 3f6832d3abd9cf50d230afdb44a3b04f56be81cbd63be8205b95d35431b7fb25
tests/test_workflow_retry.py: 7990ad7a05b20921aac308351f9e437be929d48ecdb0d0e1f4c1d3262d116698

criterion_id: K1 outcome: pass evidence_path: docs/reports/2026-10-12-retry-scan-result.md feedback: Base command exited 1 with T1's 0 != 1 assertion; the same retry suite against the fixed code exited 0, 26 passed, zero skipped. The timestamp is committed before per-item locks/transactions; lock order is unchanged.

K2: future retries expire behind locked tasks

Frozen method: targeted test via scripts/test_postgres.py passes; RESULT states base outcome.

The base failure is recorded above: T2 remained retry_wait after two passes. The fixed full retry suite above also includes T2. Additional focused command:

TMPDIR="$PWD/.venv/task-tmp" python scripts/test_postgres.py tests/test_workflow_retry.py -k retry_scan_expires_future_retries_behind_32_locked_tasks -p no:cacheprovider

Exit 0, actual final output:

1 passed, 25 deselected in 38.49s
postgres_acceptance=passed: selected tests, zero skipped tests

criterion_id: K2 outcome: pass evidence_path: docs/reports/2026-10-12-retry-scan-result.md feedback: T2 failed on the base at retry_wait != blocked. Focused fixed PostgreSQL test exited 0, 1 passed, zero skipped; both future retries park within two passes with parked: retry window expired despite the offline node and 32 held task locks.

K3: required suites

Frozen method: python scripts/test_postgres.py tests/test_workflow_retry.py tests/test_capacity_queue.py tests/test_capacity_queue_reads.py -p no:cacheprovider.

TMPDIR="$PWD/.venv/task-tmp" python scripts/test_postgres.py tests/test_workflow_retry.py tests/test_capacity_queue.py tests/test_capacity_queue_reads.py -p no:cacheprovider

Exit 0, actual final output:

40 passed in 285.30s (0:04:45)
postgres_acceptance=passed: selected tests, zero skipped tests

The full named suites ran without deselection. PostgreSQL used the prepared Unix-socket connection and randomly named disposable databases; requests used ASGI transport. This establishes high confidence in the targeted fix and the existing retry, shared capacity FIFO, concurrency and queue-read behavior.

criterion_id: K3 outcome: pass evidence_path: docs/reports/2026-10-12-retry-scan-result.md feedback: Required three-suite command exited 0, 40 passed, zero skipped; no tests were deselected.

K4: scope and ancestry

Frozen methods: git diff --name-only 422ee9a9..HEAD; git merge-base --is-ancestor 422ee9a9 HEAD.

Before committing, the staged candidate was checked using the existing Xcode Git executable (the system launcher restriction is recorded below):

/Applications/Xcode.app/Contents/Developer/usr/bin/git diff --cached --name-only 422ee9a9

Exit 0, actual paths:

docs/reports/2026-10-12-retry-scan-result.md
migrations/060_work_queue_retry.sql
src/heyaira/workflow.py
tests/test_workflow_retry.py

git merge-base --is-ancestor 422ee9a9 HEAD exited 0, without output. git diff --check and git diff --cached --check each exited 0, without output. The starting HEAD is the source commit and its existing history is preserved. Only one new local commit is intended; no migration 061 is added. The frozen range/ancestry commands are run again after committing and their actual results, final SHA and clean-worktree check go into RESULT/receipt.

criterion_id: K4 outcome: pass evidence_path: docs/reports/2026-10-12-retry-scan-result.md feedback: Staged scope check exited 0 and listed only the four authorized paths above; ancestry exited 0. Final committed range and ancestry checks are recorded in RESULT/receipt.

Execution limits and unrun work

The existing Python dependencies and PostgreSQL Unix-socket connection were used without installation. ASGI transport requires no listening socket. TMPDIR was set to an ignored writable directory inside the worktree. The system Git launcher initially reported a denied xcrun cache write; subsequent Git checks use the existing Xcode Git binary directly: /Applications/Xcode.app/Contents/Developer/usr/bin/git. It warns that the global Git ignore file cannot be read; repository checks still complete successfully. These are execution limitations, not code defects.

The complete repository suite was not run. Independent verification remains Bridge's next step; this report is worker evidence, not acceptance. No task status or criteria were changed, and no publication or deployment was performed.