Guide 14 — The Production-Adopter Pattern¶

This guide narrates the examples/production_adopter/ reference implementation that ships in v7.21.0. The example shows four "HARD-lever" tools wired up end-to-end so adopters can lift the shapes into their own application code.

Lift, don't import. The reference module is meant to be read and copied. Production adopters lift the wiring shapes into their own factories rather than importing this module — the example uses MockModelProvider and stub data so it boots without credentials, and adopters have their own real providers.

The 4-HARD-lever decision tree¶

The framework exposes four tools that adopters wire to enforce compliance, observability, and routing decisions. Each is a separate lever; a production adopter typically wires all four because they complement each other.

#	Lever	Knob	Concept doc
1	Procedural force-tool-choice	`FrameworkConfig.procedural_force_tool_choice="hard"`	`hard-lever-tools.md`
2	Diagnostic callbacks	`FrameworkConfig.callbacks=[DiagnosticCallbacks(...)]`	v6.1 `CallbackHandler` protocol
3	`on_tool_call_dispatch` rewriter	`agent.set_on_tool_call_dispatch(fn)`	v7.19.0 dispatch hook
4	MultiProviderRouter	`MultiProviderRouter(default=..., routes={...})`	v7.9.1 per-role routing

flowchart LR
  A[adopter request] --> B{1. force-tool?}
  B -- enum --> C[engine picks tool]
  C --> D{2. callbacks fire}
  D --> E{3. dispatch rewriter}
  E -- args mutated --> F[tool invoked]
  F --> G{4. router picks model}
  G --> H[LLM call]
  H --> D

When the diagnostic callbacks observe lever-1 firing AND the rewriter sees the resulting arguments, you have full visibility into the compliance decision chain. Lever-4 keeps cost predictable by routing the per-role models to the right backing provider.

Lever 1 — Procedural force-tool-choice¶

The v7.20.0 procedural_force_tool_choice enum has three values:

"off" — no forcing. Default. The LLM picks freely from tools[].
"soft" — force the tool only when the LLM also indicates intent to call it. Compatible with extended-thinking models that refuse hard tool_choice (v7.8.5 invariant).
"hard" — force the tool unconditionally. Requires the model to support tool_choice (verify via provider.supports_forced_tool_choice(config) — see hard-lever-tools.md for the full refusal matrix).

from symfonic.agent.config import FrameworkConfig

cfg = FrameworkConfig(
    procedural_force_tool_choice="hard",
)

The reference impl picks "hard" to demonstrate the strongest mode. Production adopters either:

Pick "hard" and ensure all routed models support it (use MultiProviderRouter to keep extended-thinking models off the action role).
Pick "soft" to accept the LLM's intent signal even on models that refuse the hard force.

Cross-ref: docs/concepts/hard-lever-tools.md defines the four-knob compliance framework: enum + MultiProviderRouter + supports_forced_tool_choice + refusal_reason.

Lever 2 — Diagnostic callback module¶

Bundle related observability hooks into a single class. The reference impl ships DiagnosticCallbacks in diagnostics.py with four hooks:

on_agent_start — pre-run hook for run_id correlation.
on_llm_end — post-LLM hook for cost / token accounting (the framework's TokenUsage event carries the input/output token counts and v7.20.0 cache_ttl).
on_force_choice_refusal — v7.19.2 per-turn refusal so adopter teams can correlate force-resolver telemetry across models.
on_procedure_selection — v7.20.0 ProcedureSelectionEvent for lever-1 observability.

Adopters swap the logger.info(...) lines for real telemetry sinks (CloudWatch, Datadog, Honeycomb, Grafana, ...).

CloudWatch namespace alignment. When emitting to CloudWatch, use a single namespace per lever-1 outcome:

Hook	Suggested CloudWatch metric name
`on_procedure_selection`	`symfonic_procedure_selected`
`on_force_choice_refusal`	`symfonic_force_refused_total`
`on_llm_end` (cost)	`symfonic_llm_cost_usd`

The aligned namespace makes the lever-1 vs lever-2 correlation legible in a single dashboard.

Lever 3 — `on_tool_call_dispatch` rewriter¶

The v7.19.0 hook lets adopters inspect and optionally rewrite tool call arguments before dispatch. The reference impl (rewriter.py) shows the simplest case: inject a locale="en-US" default when the tool accepts it and the caller did not supply.

Production patterns:

Argument normalisation — timezones, currency units, ISO formats.
Defaults injection — request-level context (locale, currency, tenant_branding).
Tenant-scoped guards — refuse a tool call whose tenant_id argument does not match the request scope (raises an exception that the framework's error layer surfaces).
Per-tool feature flags — gate experimental tools behind a rollout percentage.

The hook signature is:

def tool_call_dispatch_rewriter(
    tool_name: str,
    arguments: dict[str, Any],
    *,
    context: Any = None,
) -> dict[str, Any]:
    ...

Return the (possibly mutated) arguments dict. Returning None is NOT permitted; raise an exception to refuse a call.

Lever 4 — `MultiProviderRouter`¶

The v7.9.1 router satisfies the ModelProvider protocol and dispatches each call to a backing provider based on the ModelConfig.model_name prefix. Adopters compose it once at construction; SymfonicAgent never sees the multi-provider topology.

from symfonic.core.providers import (
    AnthropicProvider, MultiProviderRouter, OpenAIProvider,
)

provider = MultiProviderRouter(
    default=AnthropicProvider(),
    routes={
        "gpt-": OpenAIProvider(),
        "o1-": OpenAIProvider(),
        "o3-": OpenAIProvider(),
    },
)
agent = SymfonicAgent(
    model_provider=provider,
    config=FrameworkConfig(
        agent=AgentConfig(
            model=ModelConfig(model_name="claude-opus-4-6"),
        ),
        role_models={
            "action": ModelConfig(model_name="gpt-4.1-nano"),
            "reflection": ModelConfig(model_name="claude-opus-4-6"),
        },
    ),
)

Routes are matched in insertion order via startswith (first matching prefix wins). Unmatched model_name strings fall back to default.

Pair the router with supports_forced_tool_choice + refusal_reason introspection (v7.8.5 invariant) — see hard-lever-tools.md — to keep the hard-lever wiring sound across provider switches.

Procedural-skill authoring playbook¶

Production adopters seed skills directly into the ProceduralLayer at boot. The reference impl (procedures.py) returns a demo set of skill specs with the v7.20.0 force binding attached.

{
    "name": "lookup_customer_record",
    "steps": [...],
    "preconditions": ("customer_lookup_available",),
    "force_tool_choice": "customer_lookup",  # v7.20.0
},

preconditions is the state-predicate tuple introduced in v7.18.0 — see 12-state-predicate-preconditions.md. force_tool_choice is the v7.20.0 PROC -> TOOL binding consulted by the force-resolver.

Sweep + Tier-2 regression pattern¶

Production observability needs a Tier-2 regression sweep that runs the full skill catalogue against fixture inputs and asserts the lever-1 outcomes match the expected forced tool. The reference impl keeps the example surface small; adopters add their own sweep harness under tests/regression/tier2/ that:

Loads the production skill catalogue (via procedures.py's shape or your own loader).
For each skill + fixture input, drives the agent through one turn.
Asserts the on_procedure_selection event fires with the expected skill_id and the tool dispatch invokes the expected tool.

When a Tier-2 regression fails, the lever-2 telemetry tells you which hook diverged — a much shorter loop than re-running a full integration suite.

What this guide is NOT about¶

The topic-OUT scope is deliberate to keep the guide focused:

No cache-prefix invariant material — see hard-lever-tools.md and the multi-scope concept (v7.22).
No orthogonal-lever guidance — the four levers here are about enforcement / observability / routing. Cost-leverage knobs (pricing_overrides, _PRICING_SCHEMA_VERSION) live in v7.20.0's pricing-registry hardening concept.
No Docker / Postgres / Celery infrastructure — the production-adopter pattern is provider-agnostic; choose your own infrastructure stack.
No quickstart material — start with 01-quickstart.md if you are new.

Part B — A production-validated cost + latency recipe (v8.6.2)¶

Where Part A is the enforcement/observability/routing wiring, Part B is the cost + latency recipe an adopter ACTUALLY ran in production and measured. The levers below were validated end-to-end during a production engagement (an e-commerce assistant on Anthropic opus/sonnet/haiku). Each entry states the lever, WHY it works, the MEASURED evidence, and the honest caveat.

The validated lever set at a glance¶

Lever	Knob	Why it works	Measured evidence	Caveat
Concision directive	system-prompt instruction	Output verbosity is the dominant per-call cost (200-400 output tokens @ ~40 tok/s = 5-10s/hop)	~-15% p50, no quality loss	Adopter-authored prompt text, not a framework knob
Critic → haiku	`role_models["critic"]=claude-haiku-4-5`	Critic is the #2 wall-clock cost; haiku still revises	~4.5s (opus) → ~1s (haiku), ~half critic cost	Honoured since v7.24.2; pure adopter lever, zero core change
Action → sonnet	`role_models[ACTION]=claude-sonnet-4-5`	~3-5x cheaper than opus; more compliant on procedural tool calls	nano 6/6 vs opus 1/6 on a compliance probe	Splits the per-model cache pool — validate cache-hit on YOUR shape
Tool routing = observe	`tool_routing_mode="observe"`	`enforce` narrows the `tools[]` array per query, busting the Anthropic prefix cache	enforce dropped cache-hit 58%→15%, +7.1% cost; observe keeps the manifest cache-stable	Per-turn savings must come from `role_*`, not per-query tool narrowing
Metacognition critic	`metacognition_enabled=True`, threshold `0.7`	Post-draft reflection catches fabrication / bad claims	Genuine revises on claim-heavy turns	On high-confidence traffic the gate legitimately rarely fires — "0 critic spans" is not a bug
HMS stratigraphic prompt	`enable_hms_prompt=True`, `prompt_layer_mode="stratigraphic"`, `context_strategy="stratified"`	Cached L0/L1 prefix + uncached L2 task layer = warm-turn token burn collapses	15.0s cold vs 4.1s warm on identical ~29k-input opus turns (3.7x, cache-explained)	Requires the SystemMessage to actually reach the LLM (legacy mode renders none)
Hydration hops + recency	`hydration_max_hops=2`, `scorer_on_hot_path=True`, `scope_blend_mode="blend"`	Multi-hop hydration chains; recency-decay actually scores the hot path	recent graph nodes outrank stale at equal cosine	Recency scores graph-resident nodes only today
1h cache tier	`system_prefix_cache_ttl="1h"`, `tools_cache_ttl="1h"`	Interactive turns separated by >5-min gaps miss the default 5m tier	3.7x cold/warm latency gap closed for >5-min-gap traffic	Tools TTL must be ≥ system TTL (descending-order rule); pin core ≥7.24.1
Durability gate	precondition stack (`procedural_enforce_preconditions`, `..._render_preflight_in_l1`, `..._force_first_action_tool`)	Closes the zero-tool-use failure mode reactively AND proactively	WLM probe 0/5 → 4-5/5 tool_use blocks	`force_first_action_tool` 400s under thinking+forced choice unless the provider declines forcing
Tool-result offload	`tool_result_offload_enabled` (v8.6.0)	Collapses large prior-turn tool results to rehydratable refs	per-turn re-prefill of big observations reclaimed	Default OFF — env-gated canary; eval before flip
Per-request credentials	`credentials_resolver` (v8.5.0)	Per-tenant API keys resolved per request without rebuild	—	Adopter supplies the resolver
Hierarchical scope	`TenantScope` org→brand→conversation	Conversation-scoped query blends brand+org memories, distance-decayed	x1.0 → x0.5 → x0.25 blend	Prefix ISOLATION (the security boundary) is always-on regardless
Stream-liveness timeout	adopter stream watchdog	Bounds a hung provider stream	—	Adopter-side

Per-lever detail and evidence¶

1. Concision directive (−15% p50, the dominant term)¶

The 2026-06-15 latency teardown isolated output verbosity as the dominant per-call cost: a 150-300 word answer is 200-400 output tokens, and at ~40 tok/s that is 5-10s of pure generation per react hop. A concision instruction ("lead with the answer, omit preamble") in the authored system prompt measured ~-15% p50 with no quality loss. This is the highest-ROI latency lever for chat-style agents and it is free — it is prompt text, not a framework knob, so it lives in the adopter's SOUL / system prompt.

2. Critic → haiku (~half the critic cost)¶

The metacognition critic was measured as the #2 wall-clock cost on the adopter's battery: ~22% (~89.5s / 20 turns, ~4.5s/turn). Routing the critic role to claude-haiku-4-5 (no thinking) drops each fired round-trip from ~4.5s (opus) to ~1s (haiku) — roughly half the critic cost — and haiku still produces genuine revisions. Core has honoured role_models["critic"] since v7.24.2, so this is a pure adopter-side lever with zero core change.

3. Tool routing = observe (keep the manifest cache-stable)¶

enforce mode narrows the wire-level tools= array per query based on intent. This looks like a savings lever but Anthropic's prompt-cache prefix hash includes the entire tools= array, which precedes the system blocks where the cache breakpoints sit. A query-conditioned tools array invalidates the cached prefix every turn whose intent differs. The adopter's A/B confirmed: enforce dropped cache-hit 58%→15% and INCREASED cost +7.1% ($1.75→$1.88/turn) despite cutting raw input 40%. observe runs the same narrower for telemetry but binds the full manifest, keeping the prefix byte-stable. Per-turn savings come from role_models + role_tools, not from per-query tool narrowing.

4. Metacognition — and the confidence-gate reality (honest caveat)¶

metacognition_enabled=True turns on the post-draft LLM critic; the 0.7 confidence threshold gates when it fires. The honest caveat core got wrong and corrected: on production traffic that is mostly high-confidence and non-sensitive, the critic legitimately almost never fires. "0 metacognition_critic spans in prod" is the gate working as designed, NOT a wiring bug. The critic only surfaces as the #2 cost on cold / synthetic / claim-heavy batteries. Measure your own fire-rate before tuning the threshold.

5-7. HMS prompt, hydration, and the 1h cache tier¶

The stratigraphic prompt (cached L0/L1 prefix + uncached L2 task layer) is what makes warm turns cheap: identical ~29k-input opus turns measured 15.0s cold (cache_read=0) vs 4.1s warm (cache_read=24,488) — a 3.7x gap entirely explained by cache state. The brain's interactive traffic (turns >5 min apart) hits the cold 5-minute-tier path constantly, so the 1h cache tier (system_prefix_cache_ttl="1h" + tools_cache_ttl="1h") covers the gap. Invariant: tools TTL must be ≥ system TTL (Anthropic enforces descending TTL across the prefix); pin core ≥7.24.1 for the descending-TTL normaliser. Hydration adds hydration_max_hops=2 for product→competitor→category chains and scorer_on_hot_path=True so the recency-decay term actually runs on the MEMORY_CONTEXT hot path.

8-12. Durability, offload, creds, scope, stream-liveness¶

Durability gate — the three-lever precondition stack (procedural_enforce_preconditions + ..._render_preflight_in_l1 + ..._force_first_action_tool) closes the zero-tool-use failure mode both reactively (gate) and proactively (forced first tool). WLM probe went 0/5 → 4-5/5 tool_use blocks. Note force_first_action_tool returns HTTP 400 under thinking+forced-choice unless the provider's supports_forced_tool_choice declines forcing.
Tool-result offload (v8.6.0) — collapses large prior-turn tool results to rehydratable references, reclaiming the per-turn re-prefill of big observations. Default OFF; ships as an env-gated canary — eval before flipping.
Per-request credentials_resolver (v8.5.0) — per-tenant API keys resolved per request without rebuilding the agent.
Hierarchical TenantScope (org→brand→conversation) — a conversation-scoped query blends its brand's and org's memories, down-weighted by hierarchy distance (x1.0 → x0.5 → x0.25). Prefix isolation (the security boundary) is always-on regardless of the blend.
Stream-liveness timeout — an adopter-side watchdog that bounds a hung provider stream.

The #7c data-suppression caveat (adopter-side)¶

One validated finding is NOT a framework lever and adopters must own it: the #7c data-suppression class — when the adopter's own tool layer truncates or suppresses large tool outputs (one adopter's wrapper capped a 7M-char tool result at 80K), the model can compound truncated context across iterations and exhaust the 200K window. The framework cannot see or fix adopter-side truncation; the fix is adopter-side (offload via v8.6.0, or a smarter cap). Surface it here so adopters do not mistake it for a core bug.

The eval-before-flip discipline¶

Every cost lever above interacts with the Anthropic prompt cache, and the cache effect frequently DOMINATES the nominal token saving — the two biggest reversals of the engagement (enforce mode and v7.12.0 tool-result compaction) both increased cost because they busted the cache. The discipline that emerged:

Default-OFF levers stay OFF until measured (offload, selective metacog gate are env-gated canaries).
A/B on YOUR conversation shape, not on a synthetic battery — cache warming depends on traffic inter-arrival time and intent distribution.
Watch cache-hit ratio, not just raw input tokens — a 40% input cut that halves cache-hit is a net loss.
One lever at a time, with the v8.4.0 node_name + v8.6.2 turn_index span attributes so cost is attributable per-subsystem and per-turn.

Guide 14 — The Production-Adopter Pattern¶

The 4-HARD-lever decision tree¶

Lever 1 — Procedural force-tool-choice¶

Lever 2 — Diagnostic callback module¶

Lever 3 — `on_tool_call_dispatch` rewriter¶

Lever 4 — `MultiProviderRouter`¶

Procedural-skill authoring playbook¶

Sweep + Tier-2 regression pattern¶

What this guide is NOT about¶

See also¶

Part B — A production-validated cost + latency recipe (v8.6.2)¶

The validated lever set at a glance¶

Per-lever detail and evidence¶

1. Concision directive (−15% p50, the dominant term)¶

2. Critic → haiku (~half the critic cost)¶

3. Tool routing = observe (keep the manifest cache-stable)¶

4. Metacognition — and the confidence-gate reality (honest caveat)¶

5-7. HMS prompt, hydration, and the 1h cache tier¶

8-12. Durability, offload, creds, scope, stream-liveness¶

The #7c data-suppression caveat (adopter-side)¶

The eval-before-flip discipline¶

See also (Part B)¶

Guide 14 — The Production-Adopter Pattern¶

The 4-HARD-lever decision tree¶

Lever 1 — Procedural force-tool-choice¶

Lever 2 — Diagnostic callback module¶

Lever 3 — on_tool_call_dispatch rewriter¶

Lever 4 — MultiProviderRouter¶

Procedural-skill authoring playbook¶

Sweep + Tier-2 regression pattern¶

What this guide is NOT about¶

See also¶

Part B — A production-validated cost + latency recipe (v8.6.2)¶

The validated lever set at a glance¶

Per-lever detail and evidence¶

1. Concision directive (−15% p50, the dominant term)¶

2. Critic → haiku (~half the critic cost)¶

3. Tool routing = observe (keep the manifest cache-stable)¶

4. Metacognition — and the confidence-gate reality (honest caveat)¶

5-7. HMS prompt, hydration, and the 1h cache tier¶

8-12. Durability, offload, creds, scope, stream-liveness¶

The #7c data-suppression caveat (adopter-side)¶

The eval-before-flip discipline¶

See also (Part B)¶

Lever 3 — `on_tool_call_dispatch` rewriter¶

Lever 4 — `MultiProviderRouter`¶