LLM timeouts: the two-tier liveness model¶

Status: Shipped in v7.25.0; Anthropic inter-chunk idle bound added in v7.26.0 via the SymfonicChatAnthropic transport-injection seam, and actually made to fire in v8.7.9 (the anthropic SDK ignores http_client.timeout, so the seam now adopts the adopter's httpx.Timeout as the SDK timeout — see "Anthropic" below). Closes: a production stream-stall failure mode (21-min hang on a raw 500, ~$1.51 burned with no inter-chunk idle bound) Related: ModelConfig.timeout_seconds, ModelConfig.http_client, src/symfonic/core/_anthropic_transport.py

Why two tiers¶

A single scalar timeout is not enough for streaming LLM calls.

A request_timeout=180s bounds the entire HTTP round-trip. When the upstream provider returns a malformed SSE stream that hangs on the first chunk, the call burns 180 seconds of wall-clock before it gives up.
An inter-chunk idle bound caps the silence between bytes. When the connection is healthy but the upstream stops emitting tokens (an observed failure mode), the read times out in seconds, not minutes.

v7.25.0 surfaces both tiers as additive ModelConfig fields. Defaults are None so existing adopters see byte-identical wire shape; opt-in is per-deployment.

Tier 1: `ModelConfig.timeout_seconds`¶

from symfonic.core.config import ModelConfig

cfg = ModelConfig(timeout_seconds=180.0)

Threaded to the provider's total-request timeout kwarg:

Provider	LangChain class	Kwarg
Anthropic	`ChatAnthropic`	`default_request_timeout`
OpenAI	`ChatOpenAI`	`timeout`
DeepSeek	`ChatOpenAI` (compat)	`timeout`
Kimi	`ChatOpenAI` (compat)	`timeout`
Google	`ChatGoogleGenerativeAI`	`timeout`
Ollama	`ChatOllama`	`timeout`

Coarse-grained but universal. Use this when you need a single number that bounds the worst-case spend per call.

Anthropic note. default_request_timeout becomes the anthropic SDK's httpx.Timeout(timeout_seconds) — and httpx has no separate "whole request" timeout, only per-phase (connect/read/write/pool). The read phase is exactly the gap-between-received-bytes, so timeout_seconds=180 already bounds inter-chunk idle at 180s (uniformly across phases). Tier 1 alone closes the stall; Tier 2 only adds finer-grained control (a distinct read vs connect).

Tier 2: `ModelConfig.http_client`¶

import httpx
from symfonic.core.config import ModelConfig

# Construct ONCE at app startup, share across requests.
shared_httpx = httpx.AsyncClient(
    timeout=httpx.Timeout(read=180.0, connect=10.0, write=10.0, pool=None),
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
)

cfg = ModelConfig(http_client=shared_httpx)

This is the fine-grained per-chunk idle bound: the read timeout applies to the gap between any two received bytes -- exactly the signal we need when a stream stalls mid-response -- while connect stays tight and independent (something a single timeout_seconds scalar cannot express).

Precedence (important). timeout_seconds and http_client are not additive on Anthropic or the OpenAI family — both resolve to a single SDK httpx.Timeout. If you set both, timeout_seconds wins and your http_client's own timeout is ignored. Set http_client's httpx.Timeout(read=…) without timeout_seconds when you want the fine-grained read bound to govern.

Why the framework has to intervene. The anthropic-python and openai-python SDKs share a _base_client lineage that stamps its own self.timeout on every request and never reads http_client.timeout. Left alone, passing http_client without timeout_seconds yields self.timeout=None → httpx applies no read bound → a stalled stream hangs forever. symfonic therefore adopts your client's httpx.Timeout as the SDK request timeout (guarding httpx's default Timeout(5.0) — a bare AsyncClient() is not imposed as a 5s bound; it falls back to 600s with a warning). (Historical note: this bound silently never fired before v8.7.9 on Anthropic and v8.7.10 on the OpenAI family.)

class App:
    async def startup(self) -> None:
        self.httpx = httpx.AsyncClient(
            timeout=httpx.Timeout(read=180.0, connect=10.0),
        )

    async def shutdown(self) -> None:
        await self.httpx.aclose()

    async def handle_request(self, query: str) -> str:
        # http_client's read=180 governs the inter-chunk idle bound.  Do NOT
        # also set timeout_seconds here -- it would override the client's
        # timeout (see "Precedence" above).
        cfg = ModelConfig(http_client=self.httpx)
        agent = SymfonicAgent(
            model_provider=AnthropicProvider(),
            config=FrameworkConfig(agent=AgentConfig(model=cfg)),
        )
        return await agent.run(query)

Do not construct a fresh httpx.AsyncClient per request -- you thrash the connection pool and lose every byte of HTTP/2 multiplexing benefit.

Provider support matrix for `http_client`¶

Provider	`http_client` accepted?	Mechanism
Anthropic	YES (since v7.26.0)	Routed through the `SymfonicChatAnthropic` transport-injection seam (see "Anthropic — inter-chunk idle bound" below).
OpenAI	YES (`http_async_client`)	Native pydantic field. Since v8.7.10 the client's `httpx.Timeout` is adopted as the SDK request timeout when `timeout_seconds` is unset (the openai SDK ignores `http_client.timeout` otherwise — see the Precedence note above).
DeepSeek	YES	Rides `ChatOpenAI` (same v8.7.10 timeout-adoption).
Kimi	YES	Rides `ChatOpenAI` (same v8.7.10 timeout-adoption).
Google	NO	`google-genai` SDK builds its own transport; LangChain integration does not expose the kwarg. `timeout_seconds` still threads. `HttpClientUnsupportedWarning` fires once per process.
Ollama	NO	Local-only provider; httpx interception is rarely needed in the local-deployment shape. `HttpClientUnsupportedWarning` fires once per process.

Filter the warning when the lever doesn't apply to your provider family:

import warnings
from symfonic.core.providers import HttpClientUnsupportedWarning

warnings.filterwarnings("ignore", category=HttpClientUnsupportedWarning)

Anthropic — inter-chunk idle bound via the v7.26.0 transport seam¶

The v7.25.0 release threaded ModelConfig.timeout_seconds to ChatAnthropic.default_request_timeout for the total-budget bound but emitted a HttpClientUnsupportedWarning and dropped ModelConfig.http_client because ChatAnthropic builds its own httpx client internally and does not expose the kwarg.

v7.26.0 added a transport-injection seam (src/symfonic/core/_anthropic_transport.py). When you set ModelConfig.http_client, AnthropicProvider.get_chat_model returns a SymfonicChatAnthropic subclass that overrides the two cached_property descriptors (_client / _async_client) that ChatAnthropic uses to build the anthropic-python SDK client. The override constructs anthropic.AsyncClient(http_client=your_client, **client_params) directly, bypassing _get_default_async_httpx_client entirely.

v8.7.9 fix — the read bound now actually fires. The anthropic-python SDK stamps its own self.timeout on every request (_base_client.BaseClient._build_request) and never reads http_client.timeout. So the v7.26.0 seam threaded the client but not a timeout — self.timeout resolved to None, httpx applied no read bound, and a stalled stream hung forever (the exact failure this seam was meant to close). v8.7.9 fixes this: when you have not set timeout_seconds, the seam adopts your client's httpx.Timeout as the SDK timeout, so your read (inter-chunk idle) bound governs the wire.

Footgun guard. httpx's out-of-the-box default is Timeout(5.0) (5s on every phase). A bare httpx.AsyncClient() passed purely for transport reasons is not adopted as a 5s inter-chunk bound (that would abort legitimate long generations); the seam detects the exact default, falls back to the anthropic SDK default (600s read), and logs a warning pointing you at timeout_seconds. Pass an explicit httpx.Timeout(read=…) to opt into a real bound.

Dormancy guarantee. When ModelConfig.http_client is None, the seam module is not imported, the subclass is not instantiated, and AnthropicProvider returns vanilla ChatAnthropic(**kwargs) byte-identical to v7.25.0. Adopters who have not opted in see zero behavioural change.

Version pin. The seam refuses to activate outside _TESTED_LANGCHAIN_ANTHROPIC_VERSIONS (currently {0.3.20, 0.3.21, 0.3.22}) and raises ConfigurationError with an issue-link rather than silently falling back. If you pin langchain-anthropic to a version we have not verified the override against, you will see the error at get_chat_model() time -- loud failure beats the bound not applying.

Recommended recipe:

import httpx
from symfonic.core.config import ModelConfig

shared_httpx = httpx.AsyncClient(
    timeout=httpx.Timeout(read=180.0, connect=10.0, write=10.0, pool=30.0),
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
)

cfg = ModelConfig(
    model_name="claude-sonnet-4-5",
    http_client=shared_httpx,  # read=180 -> inter-chunk idle bound (adopted as SDK timeout)
    max_retries=0,             # let timeouts surface immediately
    # NOTE: do NOT also set timeout_seconds here -- it overrides the client's
    # timeout, so read=180 would become read=timeout_seconds (see "Precedence").
)

This converts the 1248s incident into at most a ~180s incident — the read timeout fires after the first 180s of silence, the anthropic SDK raises APITimeoutError, and the symfonic engine's on_error path takes over. (The simpler timeout_seconds=180.0 alone achieves the same bound uniformly; use the http_client form only when you need a distinct connect vs read.)

Upstream parallel. A langchain-anthropic upstream PR is in flight that adds http_client as a constructor kwarg directly on ChatAnthropic (see .claude/docs/2026-06-12-langchain-anthropic-http-client-pr-draft.md for the draft). When that lands and we bump the pin past it, _thread_timeout_and_http_client resumes threading via _accepts_kwarg → True and the seam becomes dead code we can delete.

Architectural-line note¶

The framework constrains how the call is made (timeout, transport); the adopter owns the what (model, prompt). v7.25.0 stays on that line: the new fields are seams, not policy. No defaults change.

LLM timeouts: the two-tier liveness model¶

Why two tiers¶

Tier 1: ModelConfig.timeout_seconds¶

Tier 2: ModelConfig.http_client¶

Lifecycle: construct once, share, close on shutdown¶

Provider support matrix for http_client¶

Anthropic — inter-chunk idle bound via the v7.26.0 transport seam¶

Architectural-line note¶

Tier 1: `ModelConfig.timeout_seconds`¶

Tier 2: `ModelConfig.http_client`¶

Provider support matrix for `http_client`¶