Synthetic tool-error recovery (CORE-227)

symfonic.evals.error_recovery.run_trial(provider, "native", agent_factory) runs public Agent.run once with fresh tools and a fresh in-process ledger; agent_factory(provider, tools) returns the Agent under test. The module's run_pair adds a second, counted legacy route: it calls a caller-supplied legacy_factory(provider, tools) and counts entries into the returned host's _legacy_run_impl, so the factory must return a SymfonicAgent-shaped host (run(...).final_response, flush_background_tasks()). The module imports and builds no SymfonicAgent itself; the repository's paired comparison supplies a SymfonicAgent factory, and that composition is kept in the source repository, not shipped. From 11.3.0 SymfonicAgent(...) refuses at construction, so an adopter evaluates the native route only. read_value raises a synthetic ValueError on its first attempt, then returns 42. record_value applies the value once; attempted duplicate records are counted even when rejected.

The report retains final answer, read/record attempts, applied/duplicate effects, recorded values, model-call count, exposed stop reason and the actual synthetic error observation delivered on a later model round. passed requires the read retry, one applied record, zero duplicates, correct final value, error evidence at the second provider call and correct route/body count. Exactly four calls and read rounds (1, 2) / record round (3,) are required; two reads batched before observing the failure do not count as recovery. A correct-looking final answer alone cannot pass. On the repository's legacy route, stop reason is not exposed: null means UNKNOWN.

Offline instrumentation check

This runs on an installed package with no provider and no network. The scripted model replies the same way whatever it observes, so a pass here proves only that the harness counts reads, records, rounds and the error observation; it says nothing about how a real model recovers.

import asyncio

from langchain_core.messages import AIMessage

from symfonic import Agent
from symfonic.core.testing.scripted import ScriptedChatModel, ScriptedModelProvider
from symfonic.evals.error_recovery import run_trial


class BoundedScript(ScriptedChatModel):
    max_tokens: int = 256
    max_retries: int = 0


def call(name, args, call_id):
    return AIMessage(content="", tool_calls=[{"name": name, "args": args, "id": call_id}])


def native_factory(provider, tools):
    return Agent(provider, tools=tools, max_model_rounds=4)


provider = ScriptedModelProvider()
provider.model = BoundedScript(
    script=[
        call("read_value", {}, "first"),
        call("read_value", {}, "retry"),
        call("record_value", {"value": 42}, "record"),
        AIMessage(content="42", response_metadata={"finish_reason": "stop"}),
    ],
    calls=[],
)
evidence = asyncio.run(run_trial(provider, "native", native_factory))
assert evidence.passed, evidence
print(evidence)

Bounded invocation, pending approval

No approved provider/model configuration was available for this work. No real model calls ran, no .env was sourced, and real recovery outcomes are UNKNOWN. The following is an invocation template, not an existing approved provider:

import asyncio

from approved_local_eval import create_provider  # owner-supplied, not shipped

from symfonic import Agent
from symfonic.evals.error_recovery import run_trial


def native_factory(provider, tools):
    return Agent(provider, tools=tools, max_model_rounds=4)


evidence = asyncio.run(run_trial(create_provider(), "native", native_factory))
print(evidence)  # synthetic answers/errors only; never print provider/config

The owner must approve the provider/endpoint/model and configure its ordinary LangChain chat model with max_tokens=256, max_retries=0, cache disabled and list/absent callbacks. Call the factory once per trial so every trial gets a fresh provider and model state with the same approved model/config, irrespective of the framework's model-config argument. Do not source ambient credentials or invent a fallback model. Model construction itself must not make network calls. Run from a clean eval environment with symfonic-core installed.

The adapter refuses missing/unverified token, retry, callback or cache controls. A raising chat-start callback blocks request five; each trial has a 60-second asyncio timeout. Native also limits model rounds to four. Counts are allowed chat-boundary requests, not hidden HTTP attempts. Before approving real calls, verify the particular provider honours token/retry controls, invokes callbacks before network, and has no hidden SDK retries/cache/alternate model path. The outer timeout relies on cooperative async cancellation. The caller owns provider clients and their cleanup. No production/staging/data access is needed.

Evidence limits and decision

Offline scripts prove adapter instrumentation and oracle controls only. They deliver fixed replies regardless of error observation. The routes also use different framework system prompts when the repository's paired comparison runs; that is a route comparison, not a causal experiment isolating error wording. Only run is covered here, not stream or typed projection, constructor-warning parity or default admission. Native uses the public Agent composition directly; its zero local compatibility counter is not a global probe for legacy-body unreachability. On the repository's legacy route, the actual legacy instance's body is wrapped and counted without replacing its result. Failed or cancelled trials do not prove background-task/client cleanup; isolated provider clients and post-trial cleanup remain caller responsibilities.

One real trial per route is exploratory, not proof of superiority/equivalence. Before choosing native v11 error observations over scoped compatibility, predeclare outcome criteria for retry/recovery, final correctness, attempted and duplicate effects, call counts and exhausted budgets. Repeat sufficiently for the intended models and add controlled same-prompt error-wording trials if attributing causality. This harness authorizes neither option: CORE-227 remains blocked on provider approval, outcome evidence and the explicit contract decision. No envelope widening or RET-6 reduction follows from it.