Skip to content

Troubleshooting Guide

Real failure modes seen in production adopter integrations, with one-line diagnostic checks and concrete fixes. Organized by symptom, not by subsystem — adopters typically describe what they observe (no edges, $N/month silent overpay, 401 from embedding endpoint) rather than which internal code path failed.

Quick health check

Before diving into specific symptoms, run this snippet right after building your SymfonicAgent — it surfaces the three most common misconfigurations in one call:

async def quick_audit(agent: SymfonicAgent) -> None:
    """Print a one-shot HMS audit.  Run this after agent construction
    to confirm every layer you expect to write to is actually wired."""
    orch = agent._orchestrator
    print("Registered layers:   ", list(orch._layers.keys()))
    print("Enabled in config:   ", orch.config.enabled_layers)
    print("Embedding provider:  ", type(getattr(orch, "_embedder", None)).__name__)
    missing = set(orch.config.enabled_layers) - set(orch._layers.keys())
    if missing:
        print("⚠️  ENABLED-BUT-NOT-REGISTERED:", missing)
        print("    These layers will silently no-op on auto_turn writes.")

If enabled_layers includes EPISODIC but it's NOT in the registered list, your auto-turn writes will skip silently (fixed in v7.15.3: WARNING-level log surfaces this).

Symptom: "I ran consolidation and see no edges in the graph"

Most common cause: every consolidation knob is at default + stub embedder

The framework defaults to zero-cost install: no embedder is bundled, entity linker is off, the LLM extractor for procedural promotion is off, and the synthetic-link gate requires 2 co-occurrences. On a fresh scaffold with a few turns of conversation, this produces exactly what you'd expect: 1 procedural row (regex extractor caught one rule), 1 CO_OCCURRED edge, zero entity nodes, zero MENTIONS edges.

The _StubEmbedder is the dominant problem. It produces hash-based embeddings (shake_256), which makes every embedding essentially random — semantic similarity hovers near 0.5 for everything, so any algorithm that depends on "find related entries" returns noise.

Fix

Set these in .env:

# Real embeddings -- without this, nothing semantic clusters
EMBEDDING_PROVIDER=sentence-transformers
EMBEDDING_MODEL=all-MiniLM-L6-v2

# Entity linker is the dominant edge generator
ENABLE_ENTITY_LINKER=true
ENTITY_LINKER_EXTRACTOR=regex    # or "llm" for richer extraction
ENTITY_LINKER_MIN_MENTION_COUNT=1   # default 2; for small datasets drop to 1
ENTITY_LINKER_CONFIDENCE_THRESHOLD=0.3   # default 0.5; loosen for small data

sentence-transformers is a separate optional extra — fold it into your requirements.txt:

symfonic-core[agent-api,postgres,prompts,openai,ask-user,embedding-sentence-transformers]>=7.1,<8.0

It pulls torch (1-2 GB) which is why the framework default doesn't bundle it. Rebuild your container after editing requirements.

Verification

After re-running consolidation, query Postgres:

SELECT layer, COUNT(*) FROM memory_nodes GROUP BY layer;
SELECT relationship, COUNT(*) FROM memory_edges GROUP BY relationship;

You should now see Entity:other:<word> nodes in the semantic layer and MENTIONS edges connecting them. A handful of conversation turns typically produces 10-100 entities + 50-1000 MENTIONS edges depending on content density.

Symptom: "InsightExtractor LLM calls fire (analytics shows the cost) but the semantic layer stays empty"

This was a production 2-row-stale incident (2026-06-02). The cross-cutting telemetry (analytics_llm_token_usage showing node_name='insight_extractor' rows) confirms the LLM round- trip is happening — but downstream commit_batch never reaches the semantic layer.

Most common cause: OrchestratorConfig.enabled_layers omits SEMANTIC

The engine's if get_layer(SEMANTIC) is None gate silently no-ops the write. Pre-v7.15.3 this skip was at INFO level (invisible at default WARNING). v7.15.3+ promotes it to WARNING with tenant_id + a remediation hint.

Diagnostic

print("LAYERS:  ", list(orchestrator._layers.keys()))
print("ENABLED: ", orchestrator.config.enabled_layers)

Expected output if root cause confirmed: MemoryLayer.SEMANTIC missing from both lists.

Fix

from symfonic.memory.types import MemoryLayer
from symfonic.memory.orchestrator.config import OrchestratorConfig

OrchestratorConfig(
    enabled_layers={
        MemoryLayer.WORKING,
        MemoryLayer.EPISODIC,
        MemoryLayer.SEMANTIC,
        MemoryLayer.PROCEDURAL,
        MemoryLayer.PROSPECTIVE,
    },
    # ... other config
)

Or set FrameworkConfig.enabled_layers if constructing via FrameworkConfig. Re-run a session and the semantic count should grow immediately.

Secondary: parser rejection

v7.15.3 added an INFO log distinguishing "LLM said no extractable facts" (expected on trivial prompts) from silent parser failure (bug). Watch your INFO logs for:

Self-reflection: extractor returned no insights for tenant=<id>
  (empty list -- LLM said no extractable facts, OR the parser
  rejected all candidates)

If this fires on conversations that should have insights (rich multi-turn dialogues), your min_confidence threshold may be too high or the parser is rejecting valid candidates — check the prompt template output format vs InsightExtractor._parse_insights.

Symptom: "Embedding API returns 401"

Most adopters with LLM_PROVIDER=openai pointed at an OpenAI-compatible base (DashScope / Azure / Together AI / DeepSeek's compat endpoint / OpenRouter / etc.) hit this when they also set EMBEDDING_PROVIDER=openai.

Why

The OpenAI SDK's embedding endpoint does not pick up OPENAI_API_BASE the way the chat completion endpoint does. Setting EMBEDDING_PROVIDER=openai constructs an embeddings client that targets api.openai.com directly with whatever key is in OPENAI_API_KEY — which on your stack is your DashScope key, not an OpenAI key. Hence the 401.

Fix

Use sentence-transformers for embeddings instead (runs locally in your worker process, no API key, no base URL pitfall):

EMBEDDING_PROVIDER=sentence-transformers
EMBEDDING_MODEL=all-MiniLM-L6-v2

And add the extra to requirements.txt (see the "no edges" section above for the line).

If you genuinely need cloud-hosted embeddings that hit a non-OpenAI endpoint, you need to write a custom EmbeddingProvider implementation that constructs its own httpx client with the right base URL. The framework's built-in openai provider is hardcoded to api.openai.com by design (the OpenAI SDK doesn't expose the base URL knob for the embeddings client).

Verification

list_events() on the episodic layer calls embed(" ") to get a query vector. If your worker logs show:

openai.AuthenticationError: Error code: 401 - {'error': {...}}

...inside the entity_linker phase or the consolidation pipeline, this is your issue.

Symptom: "consolidation runs successfully but the brain view shows no edges"

You've fixed the embedder and entity linker, consolidation reports entity_nodes_created: N, mentions_edges_created: M with N > 0 and M > 0, but the brain view still shows disconnected dots.

Common cause: UI field-name mismatch

The scaffolder's brain.html was written against an older edge-data shape (source_id/src_id/from). The framework's /api/v1/edges endpoint returns {id, source, target, relationship, weight}. The filter in buildGraph() looks for the old field names and silently drops every edge as "endpoint unknown."

Fix

In webapp/templates/brain.html, update the edge field accessors:

function _edgeSrc(e) { return e.source || e.source_id || e.src_id || e.from; }
function _edgeDst(e) { return e.target || e.target_id || e.dst_id || e.to; }
var visEdges = edges
  .filter(function (e) {
    return nodeIds.has(_edgeSrc(e)) && nodeIds.has(_edgeDst(e));
  })
  .map(function (e, i) {
    return {
      id: e.id || ("e" + i),
      from: _edgeSrc(e),
      to:   _edgeDst(e),
      label: e.relationship || e.type || e.rel || "",
      // ... rest of edge config
    };
  });

Rebuild your app container (templates are baked into the image, no volume mount by default) and hard-refresh the brain view (Cmd+Shift+R to bust the JS cache).

Symptom: "consolidation creates edges only for one tenant; my chat tenant has none"

The scaffolder's make consolidate defaults to tenant_id="default". If your actual chat data is under a per-user tenant (e.g. a UUID from the auth flow), consolidation never sees it.

Diagnostic

SELECT tenant_id, layer, COUNT(*)
FROM memory_nodes
GROUP BY tenant_id, layer
ORDER BY tenant_id;

Look for tenants other than default — those are your real users.

Fix

The nurse-navigator scaffold updated its worker task to accept a special "__all__" sentinel that iterates every distinct tenant in the DB:

@app.task(name="...consolidate_memory_task")
def consolidate_memory_task(tenant_id: str = "__all__") -> dict:
    if tenant_id == "__all__":
        tenants = await _discover_tenants()  # SELECT DISTINCT FROM memory_nodes
    else:
        tenants = [tenant_id]
    # ... iterate per tenant

For a single-shot run against one tenant:

make consolidate TENANT=<tenant-uuid>

For your nightly Celery beat schedule, set the args to ("__all__",) so every tenant gets consolidated, not just default.

Symptom: "$N/month silent overpay on multi-tool React turns"

This was a v7.15.0 telemetry-surfaced cache regression (2026-06-01). v7.15.0's iteration_index + per-iteration cache_creation_tokens telemetry made this visible for the first time.

What's happening

Pre-v7.15.1: the default LastStableTurnBoundaryStrategy was stateless. On every React iteration within a single turn, it walked back to the most-recent closed AIMessage and picked that as the cache anchor. When iteration 5 completed, its AIMessage became iteration 6's anchor — newer message → newer prefix → cache invalidated → cache_creation rebilled.

Fix

Upgrade to symfonic-core >=7.15.1. v7.15.1 ships the sticky-anchor fix transparently: the strategy now memoizes the anchor's AIMessage.id across calls on the same instance and only re-anchors when the anchor is dropped from history or becomes unclosed under C2. No API change, no config change, no migration required.

Verification

Watch your per-iteration cache_creation_tokens post-upgrade. Iter 0 should still incur cache_creation (cold write, anchor set). Iter 1+ should drop to near-zero cache_creation with growing cache_read_tokens (sticky hit).

Symptom: "Path E messages-region cache_control breakpoint refused to annotate"

The framework logs:

Path E messages-region cache_control breakpoint refused to annotate:
model_id='claude-sonnet-4-20250514' is in the Anthropic family but
provider_family='unknown'.

Cause

Your ModelProvider implementation isn't being classified as "anthropic" by _detect_provider_family. Common causes:

  1. Direct Protocol implementation (no subclass). v7.13.4+ introduces a _symfonic_provider_family: ClassVar[str] opt-in seam:
class MyModelProvider:
    _symfonic_provider_family: ClassVar[str] = "anthropic"
    # ... your existing implementation

  1. Subclassing a renamed/forked Anthropic adapter — ensure isinstance(provider, AnthropicProvider) holds, or use the ClassVar.

  2. Routing wrapper without _default/default — the recursion stops at the wrapper. Expose one, or set the ClassVar on the wrapper class.

The error message itself enumerates these fixes — read the full log line for the one-line fix matching your case.

Filing new issues

If you hit a failure mode not covered here, the most-actionable bug report includes:

  1. The framework version (pip show symfonic-core)
  2. The relevant log lines at WARNING level (v7.15.3+ surfaces most silent-skip antipatterns at WARN)
  3. A SQL-level snapshot: memory_nodes counts by tenant + layer, memory_edges counts by relationship, and a pg_stat_activity peek if you suspect a hang
  4. The output of the quick-audit snippet at the top of this guide
  5. Your FrameworkConfig + OrchestratorConfig construction code (the actual values, not just the type)

This is the exact evidence shape that made that production incident triage same-day-shippable. Pre-formatted diagnostic snippets in this guide produce that shape automatically.

Production adopter pattern (v7.20.0 sketch)

Three failure modes adopters hit repeatedly under multi-tenant production load. Each is a router-style entry — pick your symptom, follow the cross-ref to the dedicated guide for the recipe.

Symptom 1 — wrong tool first

The model emits a tool call but it's the wrong tool — e.g. it calls get_brand_quota before satisfying the bare-id precondition get_integration_status. The procedural rule documented the precondition; the model ignored it.

Diagnosis path: this is the L1 forcing lever's territory. Set procedural_force_tool_choice="soft" first to clean-abstain when the provider refuses; flip to "hard" for compliance / audit boundaries that cannot accept silent abstain. If you're on Anthropic + thinking, the per-turn refusal WARN tells you the structural reason — read docs/guides/10-authored-skills.md for the bare-id precondition contract and docs/concepts/hard-lever-tools.md for the lever composition rules.

Symptom 2 — right tool, wrong turn

The model calls the correct tool but on the wrong turn — e.g. it satisfies get_integration_status correctly, then calls create_scheduled_task on turn 2 instead of get_brand_quota (the next bare-id precondition in the chain).

Diagnosis path: this is the L2 routing lever's territory. The intent classifier's per-turn narrowing IS the compliance signal — flip tool_routing_mode="observe" first to validate the routing decisions, then "enforce" to constrain the bound palette. See docs/guides/07-intent-and-tool-routing.md for the classifier recipe and docs/guides/12-state-predicate-preconditions.md for the reactive complement (state predicates that gate by external state, not just message history).

Symptom 3 — right tool, right turn, wrong args

The model calls the right tool with bad arguments — e.g. it routes to the wrong tenant, queries the wrong brand, or uses a stale page_context from earlier in the conversation. The L1+L2 levers constrained the choice set correctly; the model's argument authorship missed the tenant context.

Diagnosis path: this is the on_tool_call_dispatch rewriter's territory (v7.19.0). Subscribe a handler that inspects the ToolCallDispatchEvent payload (call name, args, surrounding state including channel/tenant/user/page_context) and returns a dict {"tool_name": str, "args": dict} to rewrite the call before LangGraph dispatches. The rewriter is the right seam for tenant- aware arg correction — see docs/guides/13-tool-call-dispatch-rewriter.md for the rewriter recipe and the architectural-asymmetry rationale (the rewriter fires inside the React node, NOT post-engine, because LangGraph owns the dispatch loop).