Skip to content

symfonic.evals.model

model

Public values for repeatable agent behaviour evaluations.

AssertionResult dataclass 

AssertionResult(name: str, passed: bool, reason: str = '')

One assertion verdict with a payload-free explanation.

EvalAssertion

Bases: Protocol

A deterministic check over one turn observation.

EvalOperation

Bases: StrEnum

What a step asks the target to do.

Two, because a turn and a redemption are different operations against different seams. An approval evaluation that sent the person's answer as another prompt would prove the model can be told about an answer; only the deployment's own resume operation proves the paused run was rebuilt from its checkpoint and that the token was spent exactly once.

EvalProfile

Bases: StrEnum

Standard cost and infrastructure tiers for evaluation suites.

EvalReport dataclass 

EvalReport(scenarios: tuple[ScenarioResult, ...], framework_version: str = '')

Complete result of an evaluation suite.

EvalStatus

Bases: StrEnum

Stable aggregate and trial outcomes.

EvalStep dataclass 

EvalStep(prompt: str = '', assertions: tuple[EvalAssertion, ...] = (), conversation: str = 'default', scope: str = 'default', restart_before: bool = False, expected_error: str | None = None, output_type: type[Any] | None = None, state: Mapping[str, Any] = (lambda: MappingProxyType({}))(), attachments: tuple[Any, ...] = (), operation: EvalOperation = EvalOperation.OBSERVE, answer: Mapping[str, Any] | None = None)

One prompt and its required evidence assertions.

EvalSuite dataclass 

EvalSuite(scenarios: tuple[Scenario, ...], target_factory: Callable[[], Any] | Mapping[str, Callable[[], Any]], pack_factory: Callable[[CapabilityEvidence, str], Sequence[PackResolution]] | None = None)

Discoverable suite definition consumed by :command:symfonic eval.

target_for 

target_for(profile: str | None = None) -> Callable[[], Any]

Resolve the target explicitly; never guess among infrastructure tiers.

Source code in src/symfonic/evals/model.py 
282
283
284
285
286
287
288
289
290
291
def target_for(self, profile: str | None = None) -> Callable[[], Any]:
    """Resolve the target explicitly; never guess among infrastructure tiers."""
    if callable(self.target_factory):
        return self.target_factory
    if profile is None:
        raise ValueError("this eval suite requires --profile to select its target")
    try:
        return self.target_factory[profile]
    except KeyError as exc:
        raise ValueError(f"the eval suite has no {profile!r} target") from exc

Evidence dataclass 

Evidence(events: tuple[Any, ...] = (), attributes: Mapping[str, Any] = (lambda: MappingProxyType({}))())

Payload-free events and attributes loaded from a deployed application.

Observation dataclass 

Observation(response: str, events: tuple[Any, ...] = (), attributes: Mapping[str, Any] = (lambda: MappingProxyType({}))())

One response and the safe evidence produced while obtaining it.

Scenario dataclass 

Scenario(name: str, steps: tuple[EvalStep, ...], policy: TrialPolicy = TrialPolicy(), tags: frozenset[str] = frozenset())

An ordered behaviour specification executed as one isolated trial.

ScenarioResult dataclass 

ScenarioResult(name: str, status: EvalStatus, trials: tuple[TrialResult, ...], required_passes: int, tags: frozenset[str] = frozenset())

Aggregate verdict across all configured trials.

StepResult dataclass 

StepResult(assertions: tuple[AssertionResult, ...], duration_ms: float)

Safe result of one scenario step.

TrialPolicy dataclass 

TrialPolicy(trials: int = 1, pass_threshold: float = 1.0, timeout_seconds: float = 30.0)

How often a scenario runs and how many successful trials it requires.

required_passes property 

required_passes: int

Smallest integer number of passes satisfying the threshold.

TrialResult dataclass 

TrialResult(index: int, status: EvalStatus, steps: tuple[StepResult, ...] = (), reason: str = '', duration_ms: float = 0.0)

Result of one isolated scenario attempt.