Skip to content

Tool-Output Offload (Size-Tiered Tool-Output Policy)

On cached, tool-heavy, multi-turn sessions, a large tool result can be re-sent to the model on every react iteration and every subsequent turn. If it lands on the uncached side of the prompt-cache boundary, you pay full input price to re-prefill it each time — and the model attends over all of it on every forward pass. For a single large output (e.g. a big query result or a full export) this can dominate a turn's cost.

Tool-output offload collapses a genuinely large, settled tool result to a small deterministic stub and lets the agent pull back exactly the slice it needs on demand via the recall_tool_result tool. It is opt-in and default-OFF in v8.6.0.

Status (v8.6.0): This ships the large-tier offload, default-OFF, behind a net-saving gate. Turn it on per-deployment, measure, then keep it. The default-ON flip and the medium-tier "advancing anchor" optimization are planned follow-ups.

The size-tiered policy

Tool results are handled by size, so you only pay the offload machinery where it actually saves money:

Tier Size Handling
Hot the keep_last_n most-recent results kept full, inline — the freshest data is never stubbed
Small ≤ tool_result_compaction_size_chars (~8K) kept full, inline — too small to bother
Large ≥ tool_result_large_offload_threshold_chars (~60K) and the gate fires offloaded to a stub + on-demand recall_tool_result

A medium tier ("cache the bytes" via an advancing cache anchor) is designed but not yet shipped — see Status.

How to enable

from symfonic.agent import FrameworkConfig

config = FrameworkConfig(
    tool_result_compaction_enabled=True,   # the umbrella compaction switch
    tool_result_offload_enabled=True,      # turn ON large-tier offload (default False)
    # optional tuning (defaults shown):
    tool_result_large_offload_threshold_chars=60_000,
    tool_result_offload_horizon_default=5,       # expected conversation depth (turns)
    tool_result_offload_recall_probability=0.5,  # prior: how likely the model re-reads
    tool_result_recall_mode="both",              # "window" | "structural" | "both"
)

With tool_result_offload_enabled=False (the default), behavior is byte-identical to v8.5.0 — the existing size-only compaction path runs unchanged, and nothing is offloaded.

Config reference

Field Default What it does
tool_result_offload_enabled False Master switch for large-tier offload.
tool_result_large_offload_threshold_chars 60_000 Cheap pre-filter: only outputs at/above this size are even considered (the gate decides the rest).
tool_result_offload_horizon_default 5 Fixed prior for "remaining turns" — how long the savings accrue. Conservative; tune to your typical conversation depth.
tool_result_offload_recall_probability 0.5 Prior probability the model will re-read the output (priced as a recall round-trip in the gate).
tool_result_recall_mode "both" Retrieval styles the recall tool exposes: substring window, structural (JSON-path), or both.
tool_result_compaction_size_chars 8000 Small↔Large floor (existing).
keep_last_n 1 How many most-recent results stay hot/full (existing).

The net-saving gate (why it's safe)

Offload does not fire on size alone. After the pre-filter, a deterministic gate fires only when it is actually cheaper:

fire  ⟺  reprefill_saving − (stub_carry_cost + expected_recall_hop_cost) > 0

priced from the resolved model's rates. Concretely, for a ~321K-token result over a 5-turn horizon the gate computes a net saving (~+$8) and fires; for an 8K result it does not (the pre-filter blocks it, and even past it the recall round-trip would erase the saving). An unknown model (no pricing) never fires. This gate is what makes the feature safe to enable — it cannot trade a prefill saving for a more expensive recall loop.

Reading offloaded data: recall_tool_result

When a result is offloaded, the model sees a compact stub carrying a mem_id (trm_*), a structural index (keys / item counts / line count), a short head preview, and an instruction to call recall_tool_result. The full body is never deleted — it stays in the ledger. The tool:

recall_tool_result(
    mem_id,                 # from the stub
    query="order-4821",     # substring → returns ±window around each match
    # or:
    offset=0, max_chars=2000,   # a byte range
    # or:
    json_path="items[3].price", # structural addressing (recall_mode "structural"/"both")
    # no slice args → the whole body, capped
)

Retrieval is deterministic (no semantic search), so it never destabilizes the prompt cache. The structural index in the stub means the model can often answer without recalling at all (it sees row counts, columns, a preview).

When to enable it

Good fit: - Tool-heavy, multi-turn sessions with prompt caching on, where a large tool output (big query result, large document, full export) would otherwise ride uncached and be re-prefilled every turn.

Before enabling in production: - Measure. It's default-OFF on purpose. Enable it, run your representative workload, and confirm cost goes down without a quality regression (the model must reliably recall_tool_result when it genuinely needs the full data). - The keep_last_n hot window guarantees the freshest result is always full, so the most-likely-needed data is never stubbed. - Tune tool_result_offload_horizon_default to your typical conversation depth — deeper conversations make offload more favorable.

Status & roadmap

  • v8.6.0 (this release): large-tier offload, default-OFF, net-saving-gated; keyed recall (window + structural).
  • Planned: decision-quality eval (offload on vs off) gating a default-ON flip; the medium-tier advancing cache anchor ("cache the bytes" for outputs the model re-reads in full) — see the design record at .claude/docs/2026-06-17-design-size-tiered-tool-output-policy.md.