Whether an optional evaluation pack applies to the agent that was compiled.
An absent pack must be reported by name and an applicable pack must contribute
scenarios. Both states are explicit so neither can become a silent skip.
Applicability is a value derived from what the agent actually folded:
* :class:CapabilityEvidence reads the compiled agent or the payload-free
evidence :class:~symfonic.evals.targets.AgentTarget publishes.
* :class:PackResolution is the answer, and it is always an answer.
applicable packs must carry scenarios and name no missing evidence;
not-applicable packs must carry no scenarios and must name what was missing.
There is no third shape, so a pack cannot resolve to nothing.
A capability name is only the coarsest evidence. Narrow traits, attributed
tools and target operations prove what it actually wired; the pack still has
to prove the behaviour on a turn.
CapabilityEvidence
dataclass
CapabilityEvidence(capabilities: frozenset[str] = frozenset(), turn_inputs: frozenset[str] = frozenset(), traits: frozenset[str] = frozenset(), operations: frozenset[str] = frozenset(), capability_tools: frozenset[tuple[str, str]] = frozenset(), evidence_channels: frozenset[str] = frozenset())
What one compiled agent demonstrably offers an evaluation.
from_agent
classmethod
from_agent(agent: Any, *, traits: Iterable[str] = (), operations: Iterable[str] = (), evidence_channels: Iterable[str] = ()) -> CapabilityEvidence
Read names and attributed tools off a compiled Agent.
Traits come from :func:symfonic.evals.traits.compiled_evidence, which
reads the plan rather than accepting them here as a claim.
Source code in src/symfonic/evals/applicability.py
| @classmethod
def from_agent(
cls,
agent: Any,
*,
traits: Iterable[str] = (),
operations: Iterable[str] = (),
evidence_channels: Iterable[str] = (),
) -> CapabilityEvidence:
"""Read names and attributed tools off a compiled ``Agent``.
Traits come from :func:`symfonic.evals.traits.compiled_evidence`, which
reads the plan rather than accepting them here as a claim.
"""
names = getattr(agent, "capabilities", None)
if names is None:
raise TypeError(
"capability evidence requires a compiled agent exposing "
"capabilities; an object without it cannot say what it folded"
)
manifest = getattr(agent, "composition_manifest", None) or {}
from symfonic.evals.traits import attested_traits
return cls(
frozenset(names),
turn_inputs_of(agent),
attested_traits(agent) | frozenset(traits),
frozenset(operations),
frozenset(tuple(row) for row in manifest.get("tools", ())),
frozenset(evidence_channels),
)
|
from_observation
classmethod
from_observation(observation: Observation) -> CapabilityEvidence
Read the evidence a target published alongside one answer.
A target that publishes neither attribute yields empty evidence, and
every optional pack then resolves to not-applicable by name. That is
the honest reading: nothing about that turn showed a capability.
Source code in src/symfonic/evals/applicability.py
| @classmethod
def from_observation(cls, observation: Observation) -> CapabilityEvidence:
"""Read the evidence a target published alongside one answer.
A target that publishes neither attribute yields empty evidence, and
every optional pack then resolves to not-applicable *by name*. That is
the honest reading: nothing about that turn showed a capability.
"""
return cls(
frozenset(_names(observation.attributes.get("capabilities"))),
frozenset(_names(observation.attributes.get("turn_inputs"))),
frozenset(_names(observation.attributes.get("traits"))),
frozenset(_names(observation.attributes.get("operations"))),
frozenset(_tool_names(observation.attributes.get("capability_tools"))),
frozenset(_names(observation.attributes.get("evidence_channels"))),
)
|
missing
missing(*, capabilities: Iterable[str] = (), turn_inputs: Iterable[str] = (), traits: Iterable[str] = (), operations: Iterable[str] = (), capability_tools: Iterable[tuple[str, str]] = (), evidence_channels: Iterable[str] = ()) -> tuple[str, ...]
Required evidence this agent did not produce, in stable order.
Each kind is reported with its own prefix, because "the capability is
absent", "the capability composed nothing", and "this target cannot
deliver it" send an operator to three different places.
Source code in src/symfonic/evals/applicability.py
| def missing(
self,
*,
capabilities: Iterable[str] = (),
turn_inputs: Iterable[str] = (),
traits: Iterable[str] = (),
operations: Iterable[str] = (),
capability_tools: Iterable[tuple[str, str]] = (),
evidence_channels: Iterable[str] = (),
) -> tuple[str, ...]:
"""Required evidence this agent did not produce, in stable order.
Each kind is reported with its own prefix, because "the capability is
absent", "the capability composed nothing", and "this target cannot
deliver it" send an operator to three different places.
"""
absent = {name for name in capabilities if name not in self.capabilities}
absent |= {f"turn input {name}" for name in turn_inputs if name not in self.turn_inputs}
absent |= {
f"evidence {name}" for name in evidence_channels if name not in self.evidence_channels
}
absent |= {f"trait {name}" for name in traits if name not in self.traits}
absent |= {f"operation {name}" for name in operations if name not in self.operations}
absent |= {
f"tool {owner}:{name}"
for owner, name in capability_tools
if owner in self.capabilities and (owner, name) not in self.capability_tools
}
return tuple(sorted(absent))
|
PackResolution
dataclass
PackResolution(pack: str, applicable: bool, scenarios: tuple[Scenario, ...] = (), missing: tuple[str, ...] = ())
One pack's verdict: scenarios to run, or the evidence that was absent.
reason
property
Why the pack did not apply, naming evidence and never payloads.
applicability_report
applicability_report(resolutions: Iterable[PackResolution]) -> dict[str, object]
A JSON-safe row per pack, so a not-applicable pack is published.
Reported next to :func:~symfonic.evals.reporters.report_dict, because a
pack that resolved to not-applicable contributes no scenario and would
otherwise leave no trace at all in the run's evidence.
Source code in src/symfonic/evals/applicability.py
| def applicability_report(
resolutions: Iterable[PackResolution],
) -> dict[str, object]:
"""A JSON-safe row per pack, so a not-applicable pack is published.
Reported next to :func:`~symfonic.evals.reporters.report_dict`, because a
pack that resolved to not-applicable contributes no scenario and would
otherwise leave no trace at all in the run's evidence.
"""
rows = [
{
"pack": resolution.pack,
"applicable": resolution.applicable,
"scenarios": [scenario.name for scenario in resolution.scenarios],
"missing": list(resolution.missing),
"reason": resolution.reason,
}
for resolution in resolutions
]
return {
"schema_version": 1,
"packs": rows,
"not_applicable": [row["pack"] for row in rows if not row["applicable"]],
}
|
applicable_scenarios
applicable_scenarios(resolutions: Iterable[PackResolution]) -> tuple[Scenario, ...]
Every scenario the applicable packs contributed, in resolution order.
Source code in src/symfonic/evals/applicability.py
| def applicable_scenarios(
resolutions: Iterable[PackResolution],
) -> tuple[Scenario, ...]:
"""Every scenario the applicable packs contributed, in resolution order."""
return tuple(scenario for resolution in resolutions for scenario in resolution.scenarios)
|
resolve_pack
resolve_pack(pack: str, evidence: CapabilityEvidence, build: Callable[[], Sequence[Scenario]], *, capabilities: Iterable[str] = (), turn_inputs: Iterable[str] = (), traits: Iterable[str] = (), operations: Iterable[str] = (), capability_tools: Iterable[tuple[str, str]] = (), evidence_channels: Iterable[str] = ()) -> PackResolution
Build pack's scenarios only when its required evidence is present.
build is a callable rather than a built sequence so that an
inapplicable pack never constructs steps for a capability nothing
composed -- the steps would be unrunnable, and holding them would invite a
caller to run them anyway.
Source code in src/symfonic/evals/applicability.py
| def resolve_pack(
pack: str,
evidence: CapabilityEvidence,
build: Callable[[], Sequence[Scenario]],
*,
capabilities: Iterable[str] = (),
turn_inputs: Iterable[str] = (),
traits: Iterable[str] = (),
operations: Iterable[str] = (),
capability_tools: Iterable[tuple[str, str]] = (),
evidence_channels: Iterable[str] = (),
) -> PackResolution:
"""Build ``pack``'s scenarios only when its required evidence is present.
``build`` is a callable rather than a built sequence so that an
inapplicable pack never constructs steps for a capability nothing
composed -- the steps would be unrunnable, and holding them would invite a
caller to run them anyway.
"""
absent = evidence.missing(
capabilities=capabilities,
turn_inputs=turn_inputs,
traits=traits,
operations=operations,
capability_tools=capability_tools,
evidence_channels=evidence_channels,
)
if absent:
return PackResolution(pack, applicable=False, missing=absent)
return PackResolution(pack, applicable=True, scenarios=tuple(build()))
|
turn_inputs_of(agent: Any) -> frozenset[str]
Which of :data:TURN_INPUTS this compiled agent's turn accepts.
Read from the public entry point rather than assumed: an evaluation target
that speaks a narrower protocol -- the shipped JSON chat API accepts a
query and nothing else -- must not be credited with a channel it would
refuse, because a pack whose steps cannot be delivered has to report that
rather than fail on the delivery.
Source code in src/symfonic/evals/applicability.py
| def turn_inputs_of(agent: Any) -> frozenset[str]:
"""Which of :data:`TURN_INPUTS` this compiled agent's turn accepts.
Read from the public entry point rather than assumed: an evaluation target
that speaks a narrower protocol -- the shipped JSON chat API accepts a
query and nothing else -- must not be credited with a channel it would
refuse, because a pack whose steps cannot be delivered has to report that
rather than fail on the delivery.
"""
stream = getattr(agent, "stream", None)
if stream is None:
return frozenset()
try:
parameters = inspect.signature(stream).parameters
except (TypeError, ValueError): # pragma: no cover - exotic callables
return frozenset()
return frozenset(name for name in TURN_INPUTS if name in parameters)
|