Workflow retry rotation evidence¶

Task: 60a4cb37-70dc-48d3-9047-236e9c3ce260, frozen version 3. Execution: 578fc2a2-a154-44ae-95c6-51f7c858e1cc. Repository: h8v6/heyaira; branch: heyaira/60a4cb37. Working directory: /Users/jablonskib/HeyAira-agents/.heyaira-worktrees/heyaira/60a4cb37-worker. Source and starting HEAD: 8c8286d2a1040acda1bff8066fe0cab2f0219db3. No subagents used. Independent acceptance belongs to the Bridge verifier.

Cause and change¶

The old scan selected up to 32 retries before checking workflow locks. Its unbounded, autocommitted timestamp update then waited on a locked workflow. Cancellation rolled back that timestamp, leaving the same rows at the front.

Candidate selection now uses FOR NO KEY UPDATE OF w SKIP LOCKED, so the batch limit applies to unlocked workflows. A separate short transaction rotates the selected rows and commits before per-item processing. Only those rows enter the per-item transaction; its existing lock order is preserved.

The new parametrized regression creates 34 genuine transient preparation failures through MCP/Bridge ASGI against real PostgreSQL. The first 32 rows remain locked across two recovery calls, followed by a ready and an expired retry. T1a holds task and workflow locks; T1b holds workflow locks only. Each call has a two-second timeout and an elapsed-time assertion. Both modes assert promotion, the exact parking reason, unchanged locked workflow rows, and no locked workflow reaching _offer_retry.

K1: base regression¶

Command (source implementation unchanged; only new tests added):

python scripts/test_postgres.py tests/test_workflow_retry.py -p no:cacheprovider

Exit: 1. Actual output excerpts (fixture data omitted):

src/heyaira/workflow.py:1925: in offer_retries
    await pool.execute("""UPDATE task_workflows SET retry_checked_at=clock_timestamp()
E               TimeoutError
FAILED tests/test_workflow_retry.py::test_retry_scan_skips_32_locked_workflows[T1a-task-and-workflow]
FAILED tests/test_workflow_retry.py::test_retry_scan_skips_32_locked_workflows[T1b-workflow-only]
2 failed, 26 passed in 200.02s (0:03:20)

Both required modes timed out on the first recovery call while all 32 locks were held. The test harness cleaned up its disposable databases and returned exit 1. No checkout, reset, rebase or amendment was used for the base run.

K1: fixed regression¶

Same command, with the fix and new tests in the candidate tree above the starting commit:

python scripts/test_postgres.py tests/test_workflow_retry.py -p no:cacheprovider

Exit: 0. Actual output:

28 passed in 158.76s (0:02:38)
postgres_acceptance=passed: selected tests, zero skipped tests

This includes T1a and T1b and all existing retry tests. Both modes completed two calls under the enforced two-second bound, promoted the ready retry, parked the expired retry with parked: retry window expired, and left all 32 locked workflows untouched and outside the per-item handler.

K2: combined PostgreSQL suites¶

Command on the same fixed candidate tree:

python scripts/test_postgres.py tests/test_workflow_retry.py tests/test_capacity_queue.py tests/test_capacity_queue_reads.py -p no:cacheprovider

Exit: 0. Actual output:

42 passed in 318.25s (0:05:18)
postgres_acceptance=passed: selected tests, zero skipped tests

The existing configured maintenance database was used over its Unix socket. The harness created and removed disposable databases. No installation or production database operation was performed.

Execution limitations¶

The default /usr/bin/git launcher reported denied Xcode cache writes and Git reported denied access to the user's global ignore file. Repository commands completed using the existing Git binary at /Applications/Xcode.app/Contents/Developer/usr/bin/git. These warnings did not block the tests or repository checks. No required runtime check was skipped. Independent verification and publication remain Bridge actions.

The first local commit also emitted packed-refs.lock: Operation not permitted; the commit command nevertheless exited 0. Read-back confirmed HEAD e367eada9f4f0c44835a4931f5964f9ed23b2929, its source-commit parent, and a clean worktree. The sandbox restriction did not prevent the branch update or local commit.

K3: scope and ancestry¶

Commands after the implementation commit (using the Git binary above):

git diff --name-only 8c8286d2..HEAD
git merge-base --is-ancestor 8c8286d2 HEAD
git log -1 --format='%H %P'
git status --short
git diff --check

Scope output, exit 0:

docs/reports/60a4cb37-workflow-retry-evidence.md
src/heyaira/workflow.py
tests/test_workflow_retry.py

Ancestry: exit 0, no output. Implementation commit and parent:

e367eada9f4f0c44835a4931f5964f9ed23b2929 8c8286d2a1040acda1bff8066fe0cab2f0219db3

Worktree status: empty, exit 0. Whitespace check before the implementation commit: empty, exit 0. The final evidence-only commit adds no further paths; the same scope and ancestry commands are rechecked after it and recorded in the task RESULT with the final SHA. Existing history is preserved.

K1: committed candidate confirmation¶

The required retry command was repeated on committed implementation HEAD e367eada9f4f0c44835a4931f5964f9ed23b2929:

python scripts/test_postgres.py tests/test_workflow_retry.py -p no:cacheprovider

Exit: 0. Actual output:

28 passed in 46.81s
postgres_acceptance=passed: selected tests, zero skipped tests

The following evidence-only commit changes no runtime code or tests. Confidence is high: both modes reproduced the precise base failure and all required real-PostgreSQL suites pass with the fix. No required check remains unrun; Bridge independent verification remains pending.

Criterion records¶

criterion_id: K1
outcome: pass
evidence_path: docs/reports/60a4cb37-workflow-retry-evidence.md
feedback: Required retry command; base exit 1 (T1a and T1b TimeoutError, 26 passed); fixed committed implementation exit 0 (28 passed, zero skipped).

criterion_id: K2
outcome: pass
evidence_path: docs/reports/60a4cb37-workflow-retry-evidence.md
feedback: Required combined PostgreSQL command; exit 0; 42 passed; zero skipped.

criterion_id: K3
outcome: pass
evidence_path: docs/reports/60a4cb37-workflow-retry-evidence.md
feedback: Scope command exit 0 lists only workflow.py, the regression file and this report; ancestry command exit 0; source is the implementation commit's parent.