Model tuning: length & sampling

ModelConfig controls how the model generates — how long it can talk and how it samples tokens. These are the knobs that fix "the agent won't stop talking" and shape how focused vs. creative answers are.

from symfonic.core.config import ModelConfig

ModelConfig(
    model_name="claude-haiku-4-5",
    max_tokens=2048,     # hard length ceiling
    temperature=0.4,     # focus vs. creativity
    top_p=0.9,           # nucleus sampling
    top_k=40,            # top-k sampling (provider-dependent)
)

The knobs

Field	Default	What it does
max_tokens	16384	Hard cap on reply length. The single most important knob for runaway output — the default is large for chat, so lower it (e.g. 1024–2048) for snappy responses.
temperature	1.0	Randomness. Lower (0.2–0.5) = focused, deterministic, less rambling. Higher = more varied/creative.
top_p	None	Nucleus sampling — restrict to the smallest set of tokens whose cumulative probability ≥ top_p (0–1). Threaded to all providers.
top_k	None	Top-k sampling — restrict to the k most likely tokens. Honoured by Anthropic, Google, Ollama; the OpenAI chat API has no top_k, so OpenAI/DeepSeek/Kimi ignore it (with a one-shot warning).

temperature or top_p

Most providers recommend tuning either temperature or top_p, not both at once. Pick one as your primary randomness lever.

None on top_p/top_k means "leave the provider's own default in place" — the construction wire-shape is byte-identical for adopters who don't set them.

Provider support

Provider	max_tokens	temperature	top_p	top_k
Anthropic	✅	✅	✅	✅
OpenAI / DeepSeek / Kimi	✅	✅	✅	⚠️ ignored (warns)
Google (Gemini)	✅	✅	✅	✅
Ollama	✅	✅	✅	✅

A top_k set against an OpenAI-family provider emits a one-shot UserWarning and is dropped; top_p and temperature still apply. Silence it with warnings.filterwarnings("ignore") if intended.

Where to set it

Anywhere you build a ModelConfig:

• Scaffolded app — the main agent's model in app/setup/agent.py (model_config = ModelConfig(...)), and per-role models in the same file.
• Sub-agents — each child's model in app/setup/sub_agents.py. The starter ships a researcher capped at max_tokens=512, temperature=0.4 so delegated answers stay brief. See Sub-agents.
• Per-role / per-dispatch — override sampling for specific roles via FrameworkConfig.role_models. See Model routing.

Worked example: keep a chat agent concise

FrameworkConfig(
    agent=AgentConfig(
        model=ModelConfig(
            model_name="claude-haiku-4-5",
            max_tokens=1024,   # ~750 words, not an essay
            temperature=0.3,   # focused
        ),
    ),
)

Combined with a system prompt that asks for brevity ("answer in one paragraph unless asked for more"), this reliably stops the model from over-explaining.