symfonic.evals.traits

traits

Feature traits: what a compiled plan wired, not what its capabilities are called.

A capability name is an umbrella. prompting is folded by PromptingCapability(sources=[]), which compiles nothing and offers a knowledge evaluation no corpus at all; memory and tools are folded by every deployment that remembers anything and runs any tool, none of which implies a procedure was ever learned, reviewed, or allowed to refuse a call. Gating an optional pack on the umbrella therefore admits deployments whose behaviour the pack cannot judge, and the pack then fails for the one reason a regression suite must never fail: the feature is not there.

A trait is narrower and is read from evidence rather than declared:

• :func:plan_traits folds the same capability configs the Agent folds, through the kernel's own public :func:fold_contributions, and reads the plan that comes back. Its payload-free canonical digest must reproduce the manifest retained by the actual agent. The prompting stage carries its compiled digest and region count; a procedural gate requires the shipped implementation identity as well as its public name.
• :func:probed_traits grants a trait only when a public operation of the deployment answers. The procedural review door is the case: whether a reviewer can see a queue is not in the plan at all -- it is whether MemoryAdminService was built with the layer the promotion phase writes into -- and the only honest way to know is to ask it.

Neither route accepts a declaration, and both are necessary rather than sufficient: a pack that resolves applicable still has to prove the behaviour on a turn.

TraitProbe dataclass

TraitProbe(available: bool, operation: Callable[[], Any])

A public operation and whether its feature door exists.

available=False is the explicit feature-absent answer. Once the door exists, an exception is an operational failure and propagates; collapsing it into absence would silently remove the pack from a broken deployment.

attested_traits

attested_traits(agent: Any) -> frozenset[str]

Read narrow traits from the immutable manifest the agent retained.

Unlike :func:plan_traits, this needs no second fold. It is therefore the route for targets that receive an already-built agent from an :class:~symfonic.platform.AgentHost and do not own its capability config objects. Only payload-free fields attested by composition_manifest participate.

Source code in src/symfonic/evals/traits.py

def attested_traits(agent: Any) -> frozenset[str]:
    """Read narrow traits from the immutable manifest the agent retained.

    Unlike :func:`plan_traits`, this needs no second fold.  It is therefore the
    route for targets that receive an already-built agent from an
    :class:`~symfonic.platform.AgentHost` and do not own its capability config
    objects.  Only payload-free fields attested by ``composition_manifest``
    participate.
    """
    manifest = getattr(agent, "composition_manifest", None)
    if not isinstance(manifest, Mapping) or not manifest.get("digest"):
        return frozenset()
    found: set[str] = set()
    for row in manifest.get("stages", ()):
        if not isinstance(row, (tuple, list)) or len(row) < 7 or row[0] != _PROMPTING:
            continue
        counters = dict(row[6]) if isinstance(row[6], (tuple, list)) else {}
        if any(_count(counters.get(key)) > 0 for key in _SOURCE_COUNTERS):
            found.add(KNOWLEDGE_SOURCES)
    for row in manifest.get("preconditions", ()):
        if (
            isinstance(row, (tuple, list))
            and len(row) == 2
            and tuple(row) == (PROCEDURAL_PRECONDITION_NAME, _SKILL_PRECONDITION)
        ):
            found.add(PROCEDURAL_PRECONDITION)
    return frozenset(found)

compiled_evidence

compiled_evidence(agent: Any, capabilities: Sequence[Any], *, effect_grants: Iterable[str], target: Any = None, traits: Iterable[str] = (), options: Mapping[str, Any] | None = None) -> CapabilityEvidence

Evidence for agent, with traits read off the fold it compiled.

A second fold must reproduce the payload-free canonical digest retained by the actual Agent. Matching capability names alone is insufficient: two prompting capabilities may compile different sources, and two extension bundles may contribute different executable tools under the same umbrella.

Parameters:

Name	Type	Description	Default
traits	Iterable[str]	additional traits established elsewhere, such as the ones :func:probed_traits obtained from a public operation.	()

Source code in src/symfonic/evals/traits.py

def compiled_evidence(
    agent: Any,
    capabilities: Sequence[Any],
    *,
    effect_grants: Iterable[str],
    target: Any = None,
    traits: Iterable[str] = (),
    options: Mapping[str, Any] | None = None,
) -> CapabilityEvidence:
    """Evidence for ``agent``, with traits read off the fold it compiled.

    A second fold must reproduce the payload-free canonical digest retained by
    the actual ``Agent``. Matching capability names alone is insufficient: two
    prompting capabilities may compile different sources, and two extension
    bundles may contribute different executable tools under the same umbrella.

    Args:
        traits: additional traits established elsewhere, such as the ones
            :func:`probed_traits` obtained from a public operation.
    """
    folded = frozenset(getattr(agent, "capabilities", ()) or ())
    actual = getattr(agent, "composition_manifest", None)
    if not isinstance(actual, Mapping) or not actual.get("digest"):
        raise TypeError("compiled evidence requires an Agent exposing its composition_manifest")
    from symfonic.kernel.contracts.contributions import fold_contributions
    from symfonic.kernel.contracts.diagnostics import composition_manifest

    stages, _handlers, _grants, tools, names, preconditions = fold_contributions(
        tuple(capabilities),
        effect_grants=frozenset(effect_grants),
        options=options,
    )
    candidate = composition_manifest(stages, tools, names, preconditions)
    if candidate["digest"] != actual["digest"]:
        raise ValueError(
            "this fold did not reproduce the agent's composition digest, so "
            "the traits read from it describe something the agent did not "
            "compile. Pass the same capabilities and effect grants the agent "
            "was built with."
        )
    return CapabilityEvidence(
        capabilities=folded,
        turn_inputs=turn_inputs_of(agent),
        traits=_traits_of(stages, preconditions) | frozenset(traits),
        operations=operations_of(target) if target is not None else frozenset(),
        capability_tools=frozenset(tuple(row) for row in actual.get("tools", ())),
    )

operations_of

operations_of(target: Any) -> frozenset[str]

Which non-turn operations this target publishes.

Read from the target rather than from the agent, because an operation such as resume is a deployment seam: the compiled agent pauses, and something else redeems the token. A target that cannot resume must make an approval pack not-applicable by name rather than fail it on delivery.

Source code in src/symfonic/evals/traits.py

def operations_of(target: Any) -> frozenset[str]:
    """Which non-turn operations this target publishes.

    Read from the target rather than from the agent, because an operation such
    as resume is a *deployment* seam: the compiled agent pauses, and something
    else redeems the token. A target that cannot resume must make an approval
    pack not-applicable by name rather than fail it on delivery.
    """
    declared = getattr(target, "operations", None)
    if declared is not None and not isinstance(declared, (str, bytes)):
        try:
            return frozenset(str(name) for name in declared)
        except TypeError:  # pragma: no cover - exotic attribute
            return frozenset()
    return frozenset()

plan_traits

plan_traits(capabilities: Sequence[Any], *, effect_grants: Iterable[str], options: Mapping[str, Any] | None = None) -> frozenset[str]

Traits readable from one fold of capabilities.

Parameters:

Name	Type	Description	Default
capabilities	Sequence[Any]	the same config objects handed to Agent(...). contribute() is a declaration by contract -- "anything it needs to do happens in its handlers, at dispatch" -- so folding them a second time to read the plan performs nothing.	required
effect_grants	Iterable[str]	the grants the invocation holds. Stated rather than defaulted: a capability may decide what to contribute from them, so folding with a wider set than the agent was given would credit traits the compiled plan does not have. :func:compiled_evidence catches a mismatch against the agent's canonical manifest digest.	required

Source code in src/symfonic/evals/traits.py

def plan_traits(
    capabilities: Sequence[Any],
    *,
    effect_grants: Iterable[str],
    options: Mapping[str, Any] | None = None,
) -> frozenset[str]:
    """Traits readable from one fold of ``capabilities``.

    Args:
        capabilities: the same config objects handed to ``Agent(...)``.
            ``contribute()`` is a declaration by contract -- "anything it needs
            to *do* happens in its handlers, at dispatch" -- so folding them a
            second time to read the plan performs nothing.
        effect_grants: the grants the invocation holds. Stated rather than
            defaulted: a capability may decide what to contribute from them, so
            folding with a wider set than the agent was given would credit
            traits the compiled plan does not have. :func:`compiled_evidence`
            catches a mismatch against the agent's canonical manifest digest.
    """
    from symfonic.kernel.contracts.contributions import fold_contributions

    stages, _handlers, _grants, _tools, _names, preconditions = fold_contributions(
        tuple(capabilities),
        effect_grants=frozenset(effect_grants),
        options=options,
    )
    return _traits_of(stages, preconditions)

probed_traits async

probed_traits(probes: Mapping[str, TraitProbe]) -> frozenset[str]

Grant each trait whose named public operation answers.

probes maps a trait to a public operation and its public availability signal. Absence skips the call. A call that raises is a broken available feature and propagates rather than disguising itself as not-applicable.

Deliberately not a truthiness test on the result. An empty review queue is a wired review door with nothing in it yet, which is exactly the state a procedural pack starts from.

Source code in src/symfonic/evals/traits.py

async def probed_traits(
    probes: Mapping[str, TraitProbe],
) -> frozenset[str]:
    """Grant each trait whose named public operation answers.

    ``probes`` maps a trait to a public operation and its public availability
    signal.  Absence skips the call.  A call that raises is a broken available
    feature and propagates rather than disguising itself as not-applicable.

    Deliberately not a truthiness test on the result. An empty review queue is
    a wired review door with nothing in it yet, which is exactly the state a
    procedural pack starts from.
    """
    granted: set[str] = set()
    for trait, probe in probes.items():
        if not trait:
            raise ValueError("a trait probe requires a trait name")
        if not isinstance(probe, TraitProbe):
            raise TypeError("probed_traits requires TraitProbe values")
        if not probe.available:
            continue
        result = probe.operation()
        if inspect.isawaitable(result):
            await result
        granted.add(trait)
    return frozenset(granted)