symfonic.core.tool_offload_gate

tool_offload_gate

v8.6.0 net-saving gate for large-tier tool-output offload.

This is the load-bearing economic piece of the size-tiered tool-output policy (design: .claude/docs/2026-06-17-design-size-tiered-tool- output-policy.md §3.3). It is what was MISSING in v7.12.0 — that release fired offload on size alone with no net-saving check and lost money (+94.8% cost, cache-hit 58% → 32% on Jarvio's A/B).

The gate fires offload for a large tool output only when it provably saves money::

fire_offload  ⇔  reprefill_saving
                 − (stub_carry_cost + expected_recall_hop_cost) > 0

Every term is priced from the resolved model's MODEL_PRICING row (src/symfonic/core/observability/pricing.py). For opus-4-6: input = $5/M, output = $25/M.

Conservatism (so default-on is safe and the +94.8% failure mode cannot recur):

• A cheap pre-filter (large_offload_threshold_tokens) gates out small outputs entirely — they are the ones likely to be recalled, and the recall hop dominates their tiny re-prefill saving (worked example B). The threshold sits well above tool_result_compaction_size_chars.
• The horizon H is an underestimate of remaining turns (fewer turns of saving → the gate does not over-fire). Per the locked product decision it is a fixed, configurable expected_conversation_ depth prior — NOT the live iteration index.
• The recall hop is priced as a single expected hop (recall is on-demand, not per-turn), weighted by p_recall.
• An unknown model (no pricing row) cannot prove a saving → does not fire.

The module is intentionally a pure, side-effect-free function so it is unit-testable in isolation (the worked examples A/B in the design doc are reproduced as tests in tests/agent/test_tool_offload_gate.py).

OffloadGateDecision dataclass

OffloadGateDecision(
    fire: bool,
    net_saving_usd: float,
    reprefill_saving_usd: float,
    stub_carry_cost_usd: float,
    expected_recall_hop_cost_usd: float,
    reason: str,
)

Result of the gate: the fire/no-fire verdict plus the priced terms (so telemetry can record WHY it fired or did not).

OffloadGateInputs dataclass

OffloadGateInputs(
    output_tokens: float,
    horizon: int,
    recall_probability: float,
    model_name: str,
    large_offload_threshold_tokens: int = DEFAULT_LARGE_OFFLOAD_THRESHOLD_TOKENS,
    stub_tokens: int = DEFAULT_STUB_TOKENS,
    recall_input_tokens: int = DEFAULT_RECALL_INPUT_TOKENS,
    recall_output_tokens: int = DEFAULT_RECALL_OUTPUT_TOKENS,
)

Inputs to the net-saving gate. All economic; no I/O.

output_tokens is the size of the candidate tool output (chars / 4). horizon is the fixed expected_conversation_depth prior (remaining turns the output would otherwise be carried). recall_probability is the configurable p_recall prior. model_name resolves the pricing row.

evaluate_offload_gate

evaluate_offload_gate(
    inputs: OffloadGateInputs,
) -> OffloadGateDecision

Decide whether a large tool output should be offloaded.

Returns an :class:OffloadGateDecision whose fire field is the verdict. Pure function — no I/O, no mutation; safe to call on the hot path and trivial to unit-test against the design's worked examples.

Source code in src/symfonic/core/tool_offload_gate.py

def evaluate_offload_gate(inputs: OffloadGateInputs) -> OffloadGateDecision:
    """Decide whether a large tool output should be offloaded.

    Returns an :class:`OffloadGateDecision` whose ``fire`` field is the
    verdict.  Pure function — no I/O, no mutation; safe to call on the
    hot path and trivial to unit-test against the design's worked
    examples.
    """
    # Cheap pre-filter: below the Large threshold the gate never fires
    # (these are the recall-heavy small outputs that lost money in
    # v7.12.0).  This is a hard gate independent of the math.
    if inputs.output_tokens < inputs.large_offload_threshold_tokens:
        return OffloadGateDecision(
            fire=False,
            net_saving_usd=0.0,
            reprefill_saving_usd=0.0,
            stub_carry_cost_usd=0.0,
            expected_recall_hop_cost_usd=0.0,
            reason=(
                f"below large_offload_threshold "
                f"({inputs.output_tokens:.0f} < "
                f"{inputs.large_offload_threshold_tokens} tokens)"
            ),
        )

    pricing = MODEL_PRICING.get(inputs.model_name)
    if pricing is None:
        # No pricing row → cannot prove a saving → conservatively do
        # not fire (never lose money on an unpriced model).
        return OffloadGateDecision(
            fire=False,
            net_saving_usd=0.0,
            reprefill_saving_usd=0.0,
            stub_carry_cost_usd=0.0,
            expected_recall_hop_cost_usd=0.0,
            reason=f"no MODEL_PRICING row for {inputs.model_name!r}",
        )

    input_rate = _per_token(float(pricing.get("input", 0.0)))
    output_rate = _per_token(float(pricing.get("output", 0.0)))

    h = max(0, int(inputs.horizon))

    # The uncached re-prefill we avoid by stubbing — the observed
    # turn-4 dynamic (cache_created=0 proves the region is uncached, so
    # price at the input rate, not cache_read).
    reprefill_saving = inputs.output_tokens * input_rate * h

    # The stub still rides along in the history every turn.
    stub_carry_cost = inputs.stub_tokens * input_rate * h

    # The rehydrate is +1 LLM hop, weighted by p_recall.  Priced as a
    # single expected hop (recall is on-demand, not per-turn): the
    # recalled slice re-enters as input, plus the model's response.
    p_recall = max(0.0, min(1.0, float(inputs.recall_probability)))
    expected_recall_hop_cost = p_recall * (
        inputs.recall_input_tokens * input_rate
        + inputs.recall_output_tokens * output_rate
    )

    net = reprefill_saving - (stub_carry_cost + expected_recall_hop_cost)
    fire = net > 0.0

    reason = (
        f"net=${net:.4f} "
        f"(saving=${reprefill_saving:.4f} "
        f"− stub=${stub_carry_cost:.4f} "
        f"− recall=${expected_recall_hop_cost:.4f}); "
        f"H={h} p_recall={p_recall:.2f} model={inputs.model_name}"
    )

    return OffloadGateDecision(
        fire=fire,
        net_saving_usd=net,
        reprefill_saving_usd=reprefill_saving,
        stub_carry_cost_usd=stub_carry_cost,
        expected_recall_hop_cost_usd=expected_recall_hop_cost,
        reason=reason,
    )