symfonic.evals.provider_evidence

provider_evidence

Opt-in capture of the messages delivered at the chat-model boundary.

ProviderEvidenceRecorder

ProviderEvidenceRecorder(provider: Any)

Wrap a model provider and retain what its chat models actually receive.

This is evaluation instrumentation, not inference logic. It captures in a LangChain callback at the provider boundary, after prompt assembly and attachment encoding. Reading the original eval prompt would be easier but would turn a dropped context block into a false green.

Source code in src/symfonic/evals/provider_evidence.py

def __init__(self, provider: Any) -> None:
    if not callable(getattr(provider, "get_chat_model", None)):
        raise TypeError("ProviderEvidenceRecorder requires get_chat_model()")
    self._provider = provider
    self._calls: list[tuple[Any, ...]] = []
    self._capture = _Capture(self._calls)

calls property

calls: Sequence[tuple[Any, ...]]

All captured batches, for diagnostics owned by the evaluation.

calls_since

calls_since(cursor: int) -> tuple[tuple[Any, ...], ...]

Actual message batches delivered after cursor.

Source code in src/symfonic/evals/provider_evidence.py

def calls_since(self, cursor: int) -> tuple[tuple[Any, ...], ...]:
    """Actual message batches delivered after ``cursor``."""
    if not isinstance(cursor, int) or isinstance(cursor, bool) or cursor < 0:
        raise ValueError("an evidence cursor must be a non-negative integer")
    return tuple(self._calls[cursor:])

cursor

cursor() -> int

Position before a turn; later reads cannot include earlier calls.

Source code in src/symfonic/evals/provider_evidence.py

def cursor(self) -> int:
    """Position before a turn; later reads cannot include earlier calls."""
    return len(self._calls)

get_chat_model

get_chat_model(config: Any) -> Any

Return the provider's model with the capture callback installed.

Source code in src/symfonic/evals/provider_evidence.py

def get_chat_model(self, config: Any) -> Any:
    """Return the provider's model with the capture callback installed."""
    return self._with_capture(self._provider.get_chat_model(config))