Skip to content

symfonic.evals.pause_evidence

pause_evidence

How a stopped turn and its redemption become evidence.

Both shapes read only identity off values the deployment produced, and neither carries what a person wrote. A pause payload is the question somebody was asked and a resume response is their answer; a report an operator reads needs to know that the run stopped and that it came back, never what was said.

paused_observation 

paused_observation(paused: tuple[Any, Any], events: tuple[Any, ...], evidence: Mapping[str, Any]) -> Observation

A run that stopped to ask a person, as evidence and not as a failure.

A pause is a terminal event rather than a raised error, so without this an approval evaluation could only see "no result arrived" -- the same shape a crashed run has. The token and the question are deliberately left out: the token is a credential and the payload is the person's content, and neither belongs in a report an operator reads.

Source code in src/symfonic/evals/pause_evidence.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def paused_observation(
    paused: tuple[Any, Any],
    events: tuple[Any, ...],
    evidence: Mapping[str, Any],
) -> Observation:
    """A run that stopped to ask a person, as evidence and not as a failure.

    A pause is a terminal event rather than a raised error, so without this an
    approval evaluation could only see "no result arrived" -- the same shape a
    crashed run has. The token and the question are deliberately left out: the
    token is a credential and the payload is the person's content, and neither
    belongs in a report an operator reads.
    """
    event, interrupt = paused
    return Observation(
        response="",
        events=events,
        attributes={
            "run_id": getattr(interrupt, "run_id", ""),
            "paused": True,
            "pause_kind": getattr(event, "kind", ""),
            "pause_name": getattr(interrupt, "name", ""),
            # Measured by the capability's own checkpoint write, never asserted.
            "pause_resumable": bool(getattr(interrupt, "resumable", False)),
            **evidence,
        },
    )

resume_observation 

resume_observation(outcome: Any, evidence: Mapping[str, Any] | None = None) -> Observation

One redemption, as payload-free evidence.

Reads only identity and shape off the deployment's outcome. The validated answer and the recorded question are both left out: they are the person's content, and an operator reading a report needs to know that the turn came back, not what was said.

resume_continued is the fact an approval evaluation exists for. A deployment can validate an answer, spend the token, and still have nothing to continue -- the paused turn's state was never recorded, or could not be rebuilt -- and every one of those failures reports a successful redemption. So it is reported separately from the outcome, and separately again from the checkpoint the token was bound to.

Source code in src/symfonic/evals/pause_evidence.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def resume_observation(
    outcome: Any, evidence: Mapping[str, Any] | None = None
) -> Observation:
    """One redemption, as payload-free evidence.

    Reads only identity and shape off the deployment's outcome. The validated
    answer and the recorded question are both left out: they are the person's
    content, and an operator reading a report needs to know that the turn came
    back, not what was said.

    ``resume_continued`` is the fact an approval evaluation exists for. A
    deployment can validate an answer, spend the token, and still have nothing
    to continue -- the paused turn's state was never recorded, or could not be
    rebuilt -- and every one of those failures reports a successful redemption.
    So it is reported separately from the outcome, and separately again from
    the checkpoint the token was bound to.
    """
    return Observation(
        response="",
        attributes={
            "resume_outcome": "resumed",
            "resume_name": str(getattr(outcome, "name", "") or ""),
            "resume_run_id": str(getattr(outcome, "run_id", "") or ""),
            "resume_checkpoint": bool(getattr(outcome, "checkpoint_id", "")),
            "resume_continued": getattr(outcome, "turn", None) is not None,
            **(evidence or {}),
        },
    )