symfonic.evals.content_packs

content_packs

Optional packs for what a turn carries in and what it must hand back.

Three families, and only one of them is gated on a capability name:

• knowledge grounding needs prompting and the knowledge.sources trait. Knowledge is deliberately not its own capability -- sources are what the prompting capability compiles -- so a pack that required a knowledge name would report a capability no deployment can compose and would never run. But the name alone is the umbrella the review caught: PromptingCapability(sources=[]) folds prompting and compiles nothing, and a grounding pack run against it asserts that answers came from a corpus that does not exist. The trait is the compiled region count off the agent-attested fold, so an empty capability resolves to not-applicable by name.
• structured output and multimodal attachments are gated on the turn inputs the compiled agent accepts, because neither composes a capability at all: output_type and attachments are parameters of the public turn. Inventing a capability name for either would claim the default scaffold composes something it does not.

The gate is therefore whether the target can deliver the step. The shipped JSON chat API accepts a query and nothing else, so both packs resolve to not-applicable against it by name, rather than failing on the delivery and looking like a behaviour regression.

knowledge_grounding_pack

knowledge_grounding_pack(evidence: CapabilityEvidence, *, grounded_prompt: str, grounded_fragments: Sequence[str], source_fragment: str, unsupported_prompt: str, refusal_fragments: Sequence[str], context_attribute: str = 'prompt_context', conversation: str = 'knowledge', policy: TrialPolicy = _DEFAULT_POLICY, source_trait: str | None = None) -> PackResolution

Answer from the composed sources, and decline outside them.

Two steps, because either one alone is passable by an agent that is broken in the other direction: a corpus answer proves nothing about fabrication, and a refusal proves nothing about retrieval. The first also requires the source text in the compiled context -- an answer that was right while the source never reached the prompt was right from the model's weights, and will stop being right when the corpus changes.

Parameters:

Name	Type	Description	Default
source_fragment	str	text the composed knowledge source contributes, as it must appear in the model's input.	required
refusal_fragments	Sequence[str]	how this deployment declines. Its own wording, not a phrase invented here.	required

Source code in src/symfonic/evals/content_packs.py

def knowledge_grounding_pack(
    evidence: CapabilityEvidence,
    *,
    grounded_prompt: str,
    grounded_fragments: Sequence[str],
    source_fragment: str,
    unsupported_prompt: str,
    refusal_fragments: Sequence[str],
    context_attribute: str = "prompt_context",
    conversation: str = "knowledge",
    policy: TrialPolicy = _DEFAULT_POLICY,
    source_trait: str | None = None,
) -> PackResolution:
    """Answer from the composed sources, and decline outside them.

    Two steps, because either one alone is passable by an agent that is
    broken in the other direction: a corpus answer proves nothing about
    fabrication, and a refusal proves nothing about retrieval. The first also
    requires the source text in the compiled *context* -- an answer that was
    right while the source never reached the prompt was right from the model's
    weights, and will stop being right when the corpus changes.

    Args:
        source_fragment: text the composed knowledge source contributes, as it
            must appear in the model's input.
        refusal_fragments: how this deployment declines. Its own wording, not
            a phrase invented here.
    """
    if not grounded_fragments or not refusal_fragments:
        raise ValueError(
            "a knowledge pack needs both the grounded answer and the refusal "
            "it must give outside the corpus"
        )

    def build() -> tuple[Scenario, ...]:
        return (
            Scenario(
                "knowledge-source-grounding",
                (
                    EvalStep(
                        grounded_prompt,
                        (
                            PromptRecallContains(source_fragment, attribute=context_attribute),
                            ResponseContains(*grounded_fragments),
                            CitationsSupported(),
                        ),
                        conversation=conversation,
                    ),
                    EvalStep(
                        unsupported_prompt,
                        (
                            ResponseContains(*refusal_fragments),
                            CitationsSupported(minimum=0),
                        ),
                        conversation=conversation,
                    ),
                ),
                policy=policy,
                tags=frozenset({"knowledge", "pack"}),
            ),
        )

    return resolve_pack(
        "knowledge-grounding",
        evidence,
        build,
        capabilities=("prompting",),
        traits=(KNOWLEDGE_SOURCES, *((source_trait,) if source_trait else ())),
        evidence_channels=(context_attribute,),
    )

multimodal_attachment_pack

multimodal_attachment_pack(evidence: CapabilityEvidence, *, prompt: str, attachments: Sequence[Any], delivered_media_types: Sequence[str], expected_fragments: Sequence[str] = (), delivery_attribute: str = 'attachment_delivery', conversation: str = 'multimodal', policy: TrialPolicy = _DEFAULT_POLICY) -> PackResolution

Require the attachment to reach the provider, not the call to be accepted.

Agent.run(attachments=[...]) returning without raising proves the signature; it does not prove a block was built or that it carried the bytes, and a dropped attachment produces a confident answer about a picture the model never saw. So the assertion is on delivery evidence the target publishes from the outgoing side of the turn, and the response check is additional rather than sufficient.

Parameters:

Name	Type	Description	Default
delivered_media_types	Sequence[str]	exactly the media types the wire must carry, in attachment order. Retained as a declaration check; delivery itself is asserted with ordered payload-safe content digests.	required
delivery_attribute	str	where the target publishes what actually went out.	'attachment_delivery'

Source code in src/symfonic/evals/content_packs.py

def multimodal_attachment_pack(
    evidence: CapabilityEvidence,
    *,
    prompt: str,
    attachments: Sequence[Any],
    delivered_media_types: Sequence[str],
    expected_fragments: Sequence[str] = (),
    delivery_attribute: str = "attachment_delivery",
    conversation: str = "multimodal",
    policy: TrialPolicy = _DEFAULT_POLICY,
) -> PackResolution:
    """Require the attachment to reach the provider, not the call to be accepted.

    ``Agent.run(attachments=[...])`` returning without raising proves the
    signature; it does not prove a block was built or that it carried the
    bytes, and a dropped attachment produces a confident answer about a
    picture the model never saw. So the assertion is on delivery evidence the
    target publishes from the outgoing side of the turn, and the response
    check is additional rather than sufficient.

    Args:
        delivered_media_types: exactly the media types the wire must carry, in
            attachment order. Retained as a declaration check; delivery itself
            is asserted with ordered payload-safe content digests.
        delivery_attribute: where the target publishes what actually went out.
    """
    if not attachments:
        raise ValueError("a multimodal pack requires at least one attachment")
    if len(delivered_media_types) != len(attachments):
        raise ValueError(
            "declare one delivered media type per attachment; a shorter list "
            "would pass while an attachment was silently dropped"
        )
    expected_delivery = attachment_delivery_manifest(attachments)
    expected_media = tuple(row[2] for row in expected_delivery)
    if tuple(delivered_media_types) != expected_media:
        raise ValueError("declared delivered media types do not match the attachments under test")

    def build() -> tuple[Scenario, ...]:
        assertions: list[Any] = [
            AttributeEquals(delivery_attribute, expected_delivery),
        ]
        if expected_fragments:
            assertions.append(ResponseContains(*expected_fragments))
        return (
            Scenario(
                "multimodal-attachment-delivery",
                (
                    EvalStep(
                        prompt,
                        tuple(assertions),
                        conversation=conversation,
                        attachments=tuple(attachments),
                    ),
                ),
                policy=policy,
                tags=frozenset({"multimodal", "pack"}),
            ),
        )

    return resolve_pack(
        "multimodal-attachments",
        evidence,
        build,
        turn_inputs=("attachments",),
        evidence_channels=(delivery_attribute,),
    )

structured_output_pack

structured_output_pack(evidence: CapabilityEvidence, *, prompt: str, output_type: type[Any], expected: Mapping[str, Any] | None = None, conversation: str = 'structured', policy: TrialPolicy = _DEFAULT_POLICY) -> PackResolution

Require a validated instance of the declared schema, not prose about it.

StructuredOutputMatches reads the target's validated value, so a fluent JSON-shaped answer that never reached the schema fails here.

Source code in src/symfonic/evals/content_packs.py

def structured_output_pack(
    evidence: CapabilityEvidence,
    *,
    prompt: str,
    output_type: type[Any],
    expected: Mapping[str, Any] | None = None,
    conversation: str = "structured",
    policy: TrialPolicy = _DEFAULT_POLICY,
) -> PackResolution:
    """Require a validated instance of the declared schema, not prose about it.

    ``StructuredOutputMatches`` reads the target's *validated* value, so a
    fluent JSON-shaped answer that never reached the schema fails here.
    """

    def build() -> tuple[Scenario, ...]:
        return (
            Scenario(
                "structured-output-contract",
                (
                    EvalStep(
                        prompt,
                        (StructuredOutputMatches(output_type, expected),),
                        conversation=conversation,
                        output_type=output_type,
                    ),
                ),
                policy=policy,
                tags=frozenset({"pack", "structured-output"}),
            ),
        )

    return resolve_pack("structured-output", evidence, build, turn_inputs=("output_type",))

viewable_tool_result_pack

viewable_tool_result_pack(evidence: CapabilityEvidence, *, tool_name: str, first_prompt: str, follow_up_prompt: str, expected_fragments: Sequence[str], arguments: dict[str, object] | None = None, conversation: str = 'viewable-result', policy: TrialPolicy = _DEFAULT_POLICY) -> PackResolution

A tool result the model can look at must still be lookable at later.

One turn proves nothing: compaction rewrites results that have settled, so the failure is on the second turn. It replaced an image block with a text stub and offered recall returning str, and the model then answers about the picture from a description of it, which reads exactly like an answer about the picture (#144).

Hence two steps in one conversation: the first calls the tool, the second asks something only the image can settle. expected_fragments must name something legible only from the image and never a value the first answer already stated, or the model can reconstruct it from its own prose.

Source code in src/symfonic/evals/content_packs.py

def viewable_tool_result_pack(
    evidence: CapabilityEvidence,
    *,
    tool_name: str,
    first_prompt: str,
    follow_up_prompt: str,
    expected_fragments: Sequence[str],
    arguments: dict[str, object] | None = None,
    conversation: str = "viewable-result",
    policy: TrialPolicy = _DEFAULT_POLICY,
) -> PackResolution:
    """A tool result the model can look at must still be lookable at later.

    One turn proves nothing: compaction rewrites results that have *settled*,
    so the failure is on the second turn. It replaced an image block with a
    text stub and offered recall returning ``str``, and the model then answers
    about the picture from a description of it, which reads exactly like an
    answer about the picture (#144).

    Hence two steps in one conversation: the first calls the tool, the second
    asks something only the image can settle. ``expected_fragments`` must name
    something legible only from the image and never a value the first answer
    already stated, or the model can reconstruct it from its own prose.
    """
    if not tool_name:
        raise ValueError("a viewable-result pack requires the tool name")
    if not expected_fragments:
        raise ValueError(
            "declare what only the image can settle; without it the follow-up "
            "passes on any fluent answer, which is the failure under test"
        )

    def build() -> tuple[Scenario, ...]:
        return (
            Scenario(
                "viewable-tool-result-survives-compaction",
                (
                    EvalStep(
                        first_prompt,
                        (
                            ToolCalled(tool_name, times=1, arguments=arguments),
                            ToolSucceeded(tool_name, times=1),
                        ),
                        conversation=conversation,
                    ),
                    EvalStep(
                        follow_up_prompt,
                        (ResponseContains(*expected_fragments),),
                        conversation=conversation,
                    ),
                ),
                policy=policy,
                tags=frozenset({"multimodal", "tools", "pack"}),
            ),
        )

    # Both, not just the tool: ``missing`` only reports an absent tool when its
    # owning capability is composed, so a pack naming the tool alone is
    # applicable to a target with no tools at all and would run steps nothing
    # can execute.
    return resolve_pack(
        "viewable-tool-results",
        evidence,
        build,
        capabilities=("tools",),
        capability_tools=(("tools", tool_name),),
    )