When a model call fails: the Healer, the Medic Protocol and what recovery never does
How Bids2Bank retries a failed drafting step across model providers, when it refuses to retry, and why recovery can never turn a held sentence into a shipped one.
A Bids2Bank draft draws on three model providers. Any of them can time out, hit a rate limit or return something that does not parse. This post describes the layers of recovery in the code today, and the one rule that sits above all of them: recovery may make a run slower or move it to another provider, but it never changes whether a sentence passed the compliance checks.
Layer 1: every model call has somewhere else to go
Model calls go through one router. It asks the requested provider first (Anthropic by default) and, if that call throws, tries the others in a fixed order: Anthropic, then OpenAI, then Google. A fallback uses that provider's own default model; today those are claude-opus-4-6, gpt-4o and gemini-2.5-pro.
Layer 2: each AI step has a rule-based backup
Seven steps in the pipeline use a model: normalizing the solicitation's section headings, drafting win themes, building the outline, writing the technical, management and past-performance sentences, and drafting the cover letter, executive summary and transmittal letter.
Each one validates the model's answer against a schema and cross-checks it against the solicitation. For example, a win theme that cites a requirement the solicitation does not contain is dropped. If the model path fails, the step falls back to a deterministic version and records a warning event, so a degraded run is visible rather than silent. A rule-based draft is plainer than a model's, and its sentences still go through the same compliance checks.
Layer 3: the Healer re-runs a failed stage
The pipeline has six stages: Intake, Triage, Capture, Generation, Compliance and Submission. When a stage fails, a component we call the Healer reads the error and puts it in one of three classes:
| Class | Typical cause | What the Healer does |
|---|---|---|
| Provider | rate limit, overloaded provider, every provider failed | re-runs the stage on the next provider in the order |
| Transient | timeout, dropped connection, a 502, 503 or 504 | re-runs the stage once as it was |
| Permanent | failed validation, missing signing key, permission denied | does not re-run |
An error it cannot classify is treated as permanent: it does not retry blind. It allows at most one re-run per strategy and two per stage, and it reports a recovery only when a re-run actually returned success. Every attempt and its result is written to the workspace's event log.
When the pipeline runs on Inngest, each stage is also its own durable step, so a stage that throws can be retried by Inngest without repeating the stages before it.
After a stage succeeds, a check we call the Guardian looks at what it produced: the hand-off record must be signed and of the kind that stage is supposed to produce, and the stage's required outputs must be present. The Compliance stage, for example, must report how many sentences passed each check. Missing pieces are logged as warnings.
The Medic Protocol
The Medic Protocol is a general five-tier failover around a task the caller supplies:
| Tier | Model | Limits on the attempt |
|---|---|---|
| 1 | Claude | none |
| 2 | GPT | at most 2,500 ms |
| 3 | Gemini | at most $0.08 per call and 3,000 ms |
| 4 | Claude, marked as degraded | at most $0.05 per call and 5,000 ms |
| 5 | none: reject and escalate to a person | nothing runs |
The tiers run in order with a 500 ms pause between them and stop at the first real success. A tier succeeds only when the task itself ran again and reported success inside that tier's limits; Medic never invents an outcome. A cost the task could not report is recorded as unknown, not as zero. Tier 5 records a critical event that says every tier was exhausted and claims nothing else.
To be precise about where it runs: the Medic Protocol wraps calls made through the multi-model router's agent-call function. The proposal pipeline does not go through it today; the pipeline's stage recovery is the Healer described above.
What recovery never does
- It never changes a compliance verdict. The checks have their own narrow fallback, used only when a model gave no verdict at all, and every sentence that passes that way is flagged for a person. The post on the three checks covers this in detail.
- It never retries an error it cannot classify. Unknown means stop, not try again.
- It never reports a recovery it did not observe. Success comes from a re-run that returned success, nothing else.
- It never approves anything. Approval of a package belongs to a signed-in person on your team.
Limits
- Recovery costs time. A stage that is re-run on another provider finishes later than one that ran clean.
- Classification reads error messages. An error worded in an unexpected way can be classed wrongly: retried when it should not be, or stopped when a retry would have worked.
- A fallback draft is plainer. When a step falls back to its rule-based version, the result is predictable but less capable, and the run is recorded as degraded so you can see it happened.