Recovery & Retry

orchcore handles the messy realities of long-running AI agent processes: rate limits, stalls, git dirty trees, and partial failures. This guide covers the recovery and retry system.

Typed Failure Categories

Failures travel as data. Every failed AgentResult carries an error_category (an AgentErrorCategory value such as rate_limit, nonzero_exit, stream_error, timeout, or cancelled) populated where the failure is first observed — see the stream-events reference for the full population table. The engine's retry decision keys on this category alone: a result with error_category == AgentErrorCategory.RATE_LIMIT is retried under the phase's RetryPolicy, using the rate_limit_reset_seconds the runner already parsed; everything else is returned as-is. No error-string matching happens at the engine level, so stdout noise that merely looks like a rate limit no longer triggers retries.

Rate-Limit Detection

The RateLimitDetector uses regex patterns to identify rate-limit messages in agent output across all supported CLIs. It acts as the runner's fallback classifier: when a CLI exits non-zero without emitting a typed rate_limit stream event, a matching stderr tail upgrades the result's category from nonzero_exit to rate_limit.

Detected patterns include:

from orchcore.recovery import RateLimitDetector

detector = RateLimitDetector()
if detector.is_rate_limited(output_text):
    message = detector.extract_message(output_text)
    print(f"Rate limited: {message}")

Reset Time Parsing

The ResetTimeParser extracts wait times from rate-limit messages, supporting multiple formats:

The parser is timezone-aware — it correctly handles messages like "resets at 3:00 AM PST" regardless of the local timezone.

Retry Policy

The RetryPolicy model controls retry behavior:

from orchcore.recovery import RetryPolicy, FailureMode

policy = RetryPolicy(
    max_retries=3,
    backoff_schedule=[120, 300, 900, 1800],  # seconds
    max_wait=21600,  # 6 hours
    failure_mode=FailureMode.FAIL_FAST,
    min_count=1,  # for REQUIRE_MINIMUM mode
)

Fields

Backoff Schedule

The default backoff schedule uses exponential steps:

Random jitter (0-30 seconds) is added to each wait to avoid thundering-herd effects.

Failure Modes

For multi-agent parallel phases, the FailureMode controls how failures are evaluated:

FAIL_FAST

Stop the phase on the first agent failure. The phase is marked as failed.

phase = Phase(
    name="review",
    agents=["claude", "codex"],
    parallel=True,
    failure_mode=FailureMode.FAIL_FAST,
)

CONTINUE

Run all agents regardless of individual failures. Report results per agent.

• All succeed → done
• Some succeed, some fail → partial
• All fail → failed

phase = Phase(
    name="review",
    agents=["claude", "codex", "gemini"],
    parallel=True,
    failure_mode=FailureMode.CONTINUE,
)

REQUIRE_MINIMUM

Require at least min_count agents to succeed. Useful when you want redundancy but don't need every agent to finish.

from orchcore.recovery import RetryPolicy, FailureMode

phase = Phase(
    name="review",
    agents=["claude", "codex", "gemini"],
    parallel=True,
    failure_mode=FailureMode.REQUIRE_MINIMUM,
    retry_policy=RetryPolicy(
        failure_mode=FailureMode.REQUIRE_MINIMUM,
        min_count=2,  # At least 2 of 3 must succeed
    ),
)

Git Recovery Policy

Git recovery is disabled by default. orchcore never runs git or mutates the consumer's repository unless the phase opts in through RetryPolicy.git_recovery.

from orchcore.recovery import RetryPolicy

policy = RetryPolicy(git_recovery="auto_commit")

Available modes:

Git commands run in git_recovery_cwd when set, otherwise the explicit agent working directory. If neither exists, recovery is skipped with a warning and an on_git_recovery("skipped_no_cwd", ...) callback.

Stall and Runtime Enforcement

stall_timeout and deep_tool_timeout detect silence and emit STALL events. Enforcement is separate:

Timeout and stall kills may leave partial output artifacts; output_empty reports whether anything useful was written.

Per-Phase Retry Configuration

Retry policies can be set per phase:

from orchcore.pipeline import Phase
from orchcore.recovery import RetryPolicy, FailureMode

# Critical phase — retry aggressively
implementation = Phase(
    name="implementation",
    agents=["claude"],
    retry_policy=RetryPolicy(
        max_retries=5,
        backoff_schedule=[60, 120, 300, 600, 1200],
    ),
)

# Review phase — fail fast, no retries
review = Phase(
    name="review",
    agents=["claude", "codex"],
    parallel=True,
    failure_mode=FailureMode.CONTINUE,
    retry_policy=RetryPolicy(max_retries=0),
)

UICallback Integration

The recovery system communicates through the UICallback protocol:

Related

• Configuration Reference — max_retries, max_wait, stall_timeout settings
• ADR-008: Partial failure semantics with retry
• Stream Pipeline — stall detection stage