symfonic.evals.procedural_pack

procedural_pack

The optional pack for a procedure a deployment learned, reviewed and enforces.

Five facts have to be true before a learned procedure governs anything, and each of them is a place a deployment silently has nothing:

1. a consolidation cycle promoted a draft out of what the scope actually did;
2. while still a draft it was absent from the prompt and did not govern tools;
3. a reviewer saw that draft through the public review door and approved it, naming the tool it governs and the state that must hold;
4. the approved procedure is persisted, so it survives the process that approved it;
5. the tool gate reads that same procedure on both enforcement turns, admits the call whose state holds and refuses the one whose state does not.

So this pack takes the deployment's own promotion and review evidence rather than a boolean, and every assertion below is against a value the deployment read back out of its store. procedural_review_rows is the projection: it is spelled once, here, so a target cannot invent a shape that happens to match what a pack expects.

The gate is tools plus two traits, and not the memory name. Memory is the umbrella the review caught: a procedure is promoted into a procedural store and reviewed through an admin door, neither of which is the memory capability an agent folds -- the shipped chain composes neither on the agent -- so requiring the name would make a working deployment not-applicable while still admitting one that merely remembers.

Per-step state is explicit and required. The two enforcement turns differ only in the state the precondition is judged against, so a pack that carried no state would be sending the same turn twice and calling the second one a refusal. satisfied_state and blocked_state must differ, and the pack refuses to build if they do not.

procedural_review_rows

procedural_review_rows(procedures: Any) -> tuple[tuple[str, str], ...]

What the review door holds, as (status, governed tool) per row.

The projection a target publishes so the pack can assert on the review queue without the queue's content reaching a report. Sorted, so two runs of the same deployment produce the same evidence.

A draft nobody reviewed projects as ("draft", "") -- it has no status a gate acts on and governs no tool -- which is exactly what distinguishes it from the approved row this pack requires.

Source code in src/symfonic/evals/procedural_pack.py

def procedural_review_rows(procedures: Any) -> tuple[tuple[str, str], ...]:
    """What the review door holds, as ``(status, governed tool)`` per row.

    The projection a target publishes so the pack can assert on the review
    queue without the queue's content reaching a report. Sorted, so two runs
    of the same deployment produce the same evidence.

    A draft nobody reviewed projects as ``("draft", "")`` -- it has no status
    a gate acts on and governs no tool -- which is exactly what distinguishes
    it from the approved row this pack requires.
    """
    rows: list[tuple[str, str]] = []
    for procedure in procedures or ():
        metadata = getattr(procedure, "metadata", None) or {}
        rows.append(
            (
                str(metadata.get("status", "") or ""),
                str(metadata.get("action_tool", "") or ""),
            )
        )
    return tuple(sorted(rows))

procedural_skill_pack

procedural_skill_pack(evidence: CapabilityEvidence, *, demonstrations: Sequence[str], review_prompt: str, draft_prompt: str, satisfied_prompt: str, blocked_prompt: str, satisfied_state: Mapping[str, Any], blocked_state: Mapping[str, Any], draft_state: Mapping[str, Any], action_tool: str, procedure_fragment: str, ledger: SideEffectLedger, promotion_attribute: str = 'procedural_promoted', review_attribute: str = 'procedural_review', review_identity_attribute: str = 'procedural_review_ids', enforcement_identity_attribute: str = 'procedural_enforcement_ids', precondition_stage: str = PRECONDITION_STAGE, restart_before_enforcement: bool = True, conversation: str = 'procedural', policy: TrialPolicy = _DEFAULT_POLICY) -> PackResolution

Demonstrate, promote, approve, persist, then enforce one procedure (SC-16).

Parameters:

Name	Type	Description	Default
demonstrations	Sequence[str]	the turns that give the consolidation something to generalise. The last one is where the promotion becomes visible, so the target must have run its cycle by then and must publish the cycle's own promoted count -- not a turn counter.	required
review_prompt	str	the operator turn under which the draft is reviewed. The evidence is the review door's listing, so an approval that did not reach the store fails here even when the answer says it did.	required
draft_prompt	str	a tool-calling turn before review; it proves the draft is absent from the prompt and inert at the enforcement gate.	required
satisfied_state	Mapping[str, Any]	the turn state under which the approved procedure's precondition holds.	required
blocked_state	Mapping[str, Any]	the same request with it unmet. Must differ from satisfied_state: two identical states would make the refusal step a repeat of the one before it.	required
draft_state	Mapping[str, Any]	explicit state for the pre-approval tool call.	required
action_tool	str	the tool the approved procedure is allowed to call, and the tool the review row must name.	required
procedure_fragment	str	text that must appear in the recall block once the procedure is approved. Checked against model input, because an answer that repeats a procedure proves only that the model can repeat a procedure.	required
ledger	SideEffectLedger	the fixture the action tool applies its effect through.	required
promotion_attribute	str	where the target publishes the promoted count its consolidation cycle reported.	'procedural_promoted'
review_attribute	str	where the target publishes :func:procedural_review_rows over the public review door.	'procedural_review'
review_identity_attribute	str	approved procedure identity digests read through the review door.	'procedural_review_ids'
enforcement_identity_attribute	str	identity digests from the procedures the actual precondition callback read during this turn.	'procedural_enforcement_ids'
restart_before_enforcement	bool	rebuild the deployment before the turn that enforces. On by default, because everything above the store is process state: a procedure that only governs until the next restart is not one a deployment can rely on.	True

The last step is the one that discriminates. A refused call still reaches the gate, so the pack requires the round to show a rejected precondition stage, a tool call, zero successful executions, and a side-effect ledger whose applied count did not move from the previous step. A tool that ran anyway moves that count, and no wording in the answer can hide it.

Source code in src/symfonic/evals/procedural_pack.py

def procedural_skill_pack(
    evidence: CapabilityEvidence,
    *,
    demonstrations: Sequence[str],
    review_prompt: str,
    draft_prompt: str,
    satisfied_prompt: str,
    blocked_prompt: str,
    satisfied_state: Mapping[str, Any],
    blocked_state: Mapping[str, Any],
    draft_state: Mapping[str, Any],
    action_tool: str,
    procedure_fragment: str,
    ledger: SideEffectLedger,
    promotion_attribute: str = "procedural_promoted",
    review_attribute: str = "procedural_review",
    review_identity_attribute: str = "procedural_review_ids",
    enforcement_identity_attribute: str = "procedural_enforcement_ids",
    precondition_stage: str = PRECONDITION_STAGE,
    restart_before_enforcement: bool = True,
    conversation: str = "procedural",
    policy: TrialPolicy = _DEFAULT_POLICY,
) -> PackResolution:
    """Demonstrate, promote, approve, persist, then enforce one procedure (SC-16).

    Args:
        demonstrations: the turns that give the consolidation something to
            generalise. The last one is where the promotion becomes visible,
            so the target must have run its cycle by then and must publish the
            cycle's *own* promoted count -- not a turn counter.
        review_prompt: the operator turn under which the draft is reviewed.
            The evidence is the review door's listing, so an approval that did
            not reach the store fails here even when the answer says it did.
        draft_prompt: a tool-calling turn before review; it proves the draft is
            absent from the prompt and inert at the enforcement gate.
        satisfied_state: the turn state under which the approved procedure's
            precondition holds.
        blocked_state: the same request with it unmet. Must differ from
            ``satisfied_state``: two identical states would make the refusal
            step a repeat of the one before it.
        draft_state: explicit state for the pre-approval tool call.
        action_tool: the tool the approved procedure is allowed to call, and
            the tool the review row must name.
        procedure_fragment: text that must appear in the recall block once the
            procedure is approved. Checked against model *input*, because an
            answer that repeats a procedure proves only that the model can
            repeat a procedure.
        ledger: the fixture the action tool applies its effect through.
        promotion_attribute: where the target publishes the promoted count its
            consolidation cycle reported.
        review_attribute: where the target publishes
            :func:`procedural_review_rows` over the public review door.
        review_identity_attribute: approved procedure identity digests read
            through the review door.
        enforcement_identity_attribute: identity digests from the procedures
            the actual precondition callback read during this turn.
        restart_before_enforcement: rebuild the deployment before the turn that
            enforces. On by default, because everything above the store is
            process state: a procedure that only governs until the next restart
            is not one a deployment can rely on.

    The last step is the one that discriminates. A refused call still reaches
    the gate, so the pack requires the round to show a *rejected* precondition
    stage, a tool call, zero successful executions, and a side-effect ledger
    whose applied count did not move from the previous step. A tool that ran
    anyway moves that count, and no wording in the answer can hide it.
    """
    if not demonstrations:
        raise ValueError("a procedural pack needs at least one demonstration turn")
    if not action_tool:
        raise ValueError("a procedural pack requires the tool the procedure governs")
    if dict(satisfied_state) == dict(blocked_state):
        raise ValueError(
            "the satisfied and blocked states are the same, so the two "
            "enforcement turns would differ in nothing and the refusal would "
            "prove no precondition was read"
        )
    if not satisfied_state or not blocked_state:
        raise ValueError(
            "both enforcement turns need explicit state: a precondition judged "
            "against nothing refuses everything, which passes this pack for a "
            "gate that is not reading anything"
        )
    if not draft_prompt or not draft_state:
        raise ValueError("the pre-approval draft probe needs a prompt and state")
    approved_rows = ((APPROVED, action_tool),)

    def build() -> tuple[Scenario, ...]:
        steps: list[EvalStep] = [
            EvalStep(prompt, conversation=conversation)
            for prompt in demonstrations[:-1]
        ]
        steps.append(
            EvalStep(
                demonstrations[-1],
                (AttributeCount(promotion_attribute, minimum=1),),
                conversation=conversation,
            )
        )
        steps.append(
            EvalStep(
                draft_prompt,
                (
                    AttributeEquals(review_attribute, (("draft", ""),)),
                    AttributeEquals(enforcement_identity_attribute, ()),
                    PromptRecallExcludes(procedure_fragment),
                    ToolSucceeded(action_tool, times=1),
                    SideEffectCount(ledger, attempts=1, effects=1, duplicates=0),
                ),
                conversation=conversation,
                state=draft_state,
            )
        )
        steps.append(
            EvalStep(
                review_prompt,
                (AttributeEquals(review_attribute, approved_rows),),
                conversation=conversation,
            )
        )
        steps.append(
            EvalStep(
                satisfied_prompt,
                (
                    # Re-read after the restart: the row has to still be there.
                    AttributeEquals(review_attribute, approved_rows),
                    AttributesSameNonEmpty(
                        review_identity_attribute, enforcement_identity_attribute
                    ),
                    PromptRecallContains(procedure_fragment),
                    StageObserved(
                        precondition_stage, minimum_counts={"checked": 1}
                    ),
                    ToolSucceeded(action_tool, times=1),
                    SideEffectCount(ledger, attempts=2, effects=2, duplicates=0),
                ),
                conversation=conversation,
                state=satisfied_state,
                restart_before=restart_before_enforcement,
            )
        )
        steps.append(
            EvalStep(
                blocked_prompt,
                (
                    AttributesSameNonEmpty(
                        review_identity_attribute, enforcement_identity_attribute
                    ),
                    StageObserved(
                        precondition_stage,
                        outcome="rejected",
                        minimum_counts={"refused": 1},
                    ),
                    ToolCalled(action_tool, times=1),
                    ToolSucceeded(action_tool, times=0),
                    # Unchanged from the step above: the refused call reached
                    # the gate and performed nothing.
                    SideEffectCount(ledger, attempts=2, effects=2, duplicates=0),
                ),
                conversation=conversation,
                state=blocked_state,
            )
        )
        return (
            Scenario(
                "procedural-review-and-enforcement",
                tuple(steps),
                policy=policy,
                tags=frozenset({"pack", "procedural"}),
            ),
        )

    return resolve_pack(
        "procedural-skills",
        evidence,
        build,
        capabilities=("tools",),
        traits=(PROCEDURAL_PRECONDITION, PROCEDURAL_REVIEW),
    )