symfonic.core.prompt.messages_cache_rolling

messages_cache_rolling

v8.7.0 -- rolling-ladder messages-region cache strategy.

Problem

The default :class:LastStableTurnBoundaryStrategy (v7.15.1 sticky anchor) pins ONE cache_control marker at the first stable boundary of a turn and never advances it -- deliberately, because v7.15.0 showed that MOVING a single marker forward re-bills the whole prefix as cache_creation on every React iteration (~$1.7K/mo/brand overpay).

The cost of never advancing: every tool exchange appended AFTER the pinned anchor (get_block_schema -> add_block -> connect ...) is re-sent UNCACHED on every subsequent iteration. For an agentic graph-builder that runs ~64 iterations, the uncached tail grows ~N**2 (the flow-builder cost driver: ~1.4M uncached tokens / build).

Approach

Anthropic honors up to 4 cache_control markers per request and its lookback is INCREMENTAL: a marker at a longer prefix reads the cached shorter prefix automatically. So instead of MOVING one marker (rebill) we HOLD the prior frontier marker(s) and ADD one new marker at the newest closed boundary each iteration:

• every held rung was already marked last iteration -> served as a cache-READ (its prefix is frozen and cached),
• the single newly-added rung is the ONLY cache_creation write, and it writes just the delta since the previous frontier.

Dropping the OLDEST rung when the ladder is full is safe: a strictly longer rung is always still marked, so the dropped rung's (shorter) prefix remains a cache-read via lookback. We never MOVE a marker, so the v7.15.0 rebill cannot recur.

Result: the uncached tail collapses from ~N**2 to ~linear.

Budget

The ladder needs >= 2 messages-region marker slots (one to HOLD the frontier as a read, one to EXTEND). The available budget is ANTHROPIC_MAX_CACHE_BREAKPOINTS minus the system-prefix markers (stratigraphic L0+L1 == 2 by default) minus the tools marker (1 iff tools_cache_ttl is set). Default stratigraphic config leaves exactly 4 - 2 - 0 = 2 free -> a 2-rung ladder fits. When an adopter sets tools_cache_ttl only 1 slot remains, so this strategy DEGRADES to the single sticky marker (byte-identical to "default") for that turn.

Verification

Unit tests pin the placement math; the token-billing win itself is only observable on live Anthropic -- see tests/integration/live_llm/test_cache_mechanics.py before enabling messages_cache_policy="rolling" in production.

RollingLadderStrategy

RollingLadderStrategy()

v8.7.0 rolling-ladder implementation of both :class:MessagesRegionCacheStrategy (single) and :class:MultiBreakpointCacheStrategy (plural).

Lifecycle: ONE instance per agent (registered in engine.py when messages_cache_policy == "rolling"). The same instance is consulted on every React iteration, so it persists the ids of the rungs it emitted last iteration to guarantee the never-move / only- add invariant.

Concurrency (v8.7.1 H1): the held-rung ids are isolated PER CONVERSATION in a bounded LRU keyed by conversation_id (the react run_id). ONE agent instance serves ALL concurrent requests in create_agent_router and shares this strategy, so a single _held_ids list was clobbered by interleaved React loops on different conversations — moving the frontier and re-billing the full prefix. Keying per conversation removes the shared mutable state; the never-move / only-add math is unchanged.

Source code in src/symfonic/core/prompt/messages_cache_rolling.py

def __init__(self) -> None:
    # Delegate the single-marker (degrade / back-compat) path to the
    # battle-tested v7.15.1 sticky strategy so its own memoization
    # stays authoritative there.
    self._single = LastStableTurnBoundaryStrategy()
    # v8.7.1 (H1): ids of the rungs emitted on the PREVIOUS plural
    # call, per conversation. Ascending by the index they occupied
    # then. Empty/absent when the ladder is inactive for a
    # conversation (degraded / non-Anthropic / below threshold).
    self._held_ids_by_conv: OrderedDict[str, list[str]] = OrderedDict()

place_breakpoint

place_breakpoint(
    messages: Sequence[Any],
    *,
    provider_family: ProviderFamily,
    model_id: str,
    thinking_enabled: bool = False,
    prior_breakpoint_indices: Sequence[int] = (),
    conversation_id: str | None = None,
) -> int | None

Single-marker fallback. Delegates to the sticky strategy so callers that never learned about the plural API (or a budget of exactly 1) get the exact v7.15.1 behavior.

Source code in src/symfonic/core/prompt/messages_cache_rolling.py

def place_breakpoint(
    self,
    messages: Sequence[Any],
    *,
    provider_family: ProviderFamily,
    model_id: str,
    thinking_enabled: bool = False,
    prior_breakpoint_indices: Sequence[int] = (),
    conversation_id: str | None = None,
) -> int | None:
    """Single-marker fallback.  Delegates to the sticky strategy so
    callers that never learned about the plural API (or a budget of
    exactly 1) get the exact v7.15.1 behavior."""
    return self._single.place_breakpoint(
        messages,
        provider_family=provider_family,
        model_id=model_id,
        thinking_enabled=thinking_enabled,
        prior_breakpoint_indices=prior_breakpoint_indices,
        conversation_id=conversation_id,
    )