Production Runbook¶

Lessons learned from deploying SymfonicAgent in a real production app (AbogaAI -- legal tech, FastAPI + React, Postgres/pgvector, S3, Celery).

This document covers the pitfalls that the integration guide does not.

1. The Empty Orchestrator Trap¶

The most common deployment failure: the agent runs, responds, but never writes memories.

Symptom¶

memory_nodes table stays at 0 rows
No consolidation logs
Agent responds normally (memory is optional, not required)

Root Cause¶

SymfonicAgent.__init__() has a fallback chain (engine.py:260-279):

1. orchestrator param passed?        -> use it
2. embedding_provider + both backends? -> HMSFactory.build() with Postgres
3. embedding_provider only?           -> HMSFactory.build_in_memory()
4. nothing?                           -> empty MemoryOrchestrator (NO WRITES)

If you pass model_provider and tools but forget the backends, you land in case 4. The agent works fine -- it just has amnesia.

Fix¶

Always pass all three backend params:

# `config` is a FrameworkConfig with enable_hms_prompt=True  — REQUIRED for extraction
agent = SymfonicAgent(
    model_provider=provider,
    config=config,
    tools=tools,
    graph_backend=graph_backend,  # REQUIRED for persistence
    vector_backend=vector_backend,
    embedding_provider=embedding,
)

Verification¶

After sending a message, check:

SELECT count(*) FROM memory_nodes;
-- Should be > 0 within seconds of the first conversation

2. stream_typed() vs run() Consolidation¶

Symptom¶

run() consolidates correctly but stream_typed() produces empty memory nodes or skips consolidation entirely.

Root Cause (fixed in 4.9.1)¶

Before 4.9.1, stream_typed() yielded typed StreamEvent dataclasses but never accumulated TextDeltaEvent.text. After streaming finished, _consolidate(result={}) ran with empty data. Working memory, self-reflection, and episodic writes all produced nothing.

Verification¶

After a streaming conversation:

SELECT id, layer, label, importance, created_at
FROM memory_nodes
WHERE tenant_id = 'your-org-id'
ORDER BY created_at DESC
LIMIT 5;

Expect at least 1 working-memory node (current reasoning) and potentially semantic nodes (from MEMORY_EXTRACT ops).

3. Scope Design¶

FrameworkTenantScope maps your app's tenancy model to HMS isolation.

Fields¶

Field	Maps to	Purpose
`tenant_id`	org_id / company_id	Top-level data boundary. All memory queries filter by this.
`sub_tenant_id`	user_id	Per-user isolation within the org.
`namespace`	user_id (optional)	Memory namespace. Usually same as sub_tenant_id.

Example¶

scope = FrameworkTenantScope(
    tenant_id="org-uuid",         # organization boundary
    sub_tenant_id="user-uuid",    # per-user isolation
    namespace="user-uuid",        # memory namespace
)

Case-Level Scoping¶

HMS has no built-in case/project scoping. The memory_nodes table has no case_id column. Two approaches:

A. Anchor nodes (recommended): Create a CASE_ANCHOR node per case and LINKED_TO edges from memory nodes to it. Filter by graph traversal.

B. Separate tenants: Use tenant_id = f"{org_id}:{case_id}". Simple but prevents cross-case memory sharing (e.g., global legal knowledge).

4. enable_hms_prompt is Required¶

Setting enable_hms_prompt=True does two things:

HMSSystemPromptSection -- Injects retrieved memory context into the system prompt before each LLM call.
MemoryExtractionSection -- Instructs the LLM to output <MEMORY_EXTRACT> blocks containing structured ops.

Without it, the LLM never outputs extraction blocks, and consolidation has nothing to process. Memory reads (hydration) also won't happen.

5. EmbeddingProvider Protocol¶

The EmbeddingProvider protocol expects:

class EmbeddingProvider(Protocol):
    @property
    def dimensions(self) -> int: ...

    async def embed(self, text: str) -> list[float]: ...
    #                     ^^^^^^^^ SINGLE STRING, not list

Pitfall¶

The episodic layer (before 4.9.1) called embed([query]) passing a list. If your adapter doesn't handle this, you get silent failures.

Defensive Implementation¶

class MyEmbeddingAdapter:
    @property
    def dimensions(self) -> int:
        return 1024

    async def embed(self, text: str | list[str]) -> list[float]:
        if isinstance(text, list):
            text = " ".join(text)
        # ... call your embedding service

6. Streaming + Persistence Race Condition¶

Symptom¶

Agent generates a document/artifact mid-stream. User clicks the download link immediately. Gets 404. After page refresh, it works.

Root Cause¶

The tool creates the DB record using the request-scoped session. FastAPI's get_db dependency commits only AFTER StreamingResponse finishes. The download link is sent mid-stream, before the commit.

Fix¶

Tools that create artifacts the user will access immediately must use a dedicated session with explicit commit:

async def _execute(self, ...):
    # Upload file to S3
    await storage.put(key, file_bytes)

    # Commit DB record BEFORE returning the download URL
    async with async_session_factory() as session:
        record = ExportRecord(file_path=key, ...)
        session.add(record)
        await session.commit()

    return f"Download: /api/v1/export/download/{record.id}"

7. MEMORY_EXTRACT Tag Leakage¶

Symptom¶

Raw <MEMORY_EXTRACT>{"ops": [...]} JSON appears in chat messages.

Root Cause¶

The LLM outputs extraction blocks as part of its response text. The engine strips them AFTER streaming completes (during consolidation), but individual TextDeltaEvent chunks carry the raw text.

Fix¶

Add a stateful filter in your SSE streaming layer. Per-chunk regex won't work because tags span multiple chunks.

class MemoryTagFilter:
    def __init__(self):
        self._inside = False

    def filter(self, text: str) -> str:
        # Track open/close tags across chunks
        # Suppress all content between <MEMORY_EXTRACT> and </MEMORY_EXTRACT>
        ...

Also add strip_memory_tags() as a safety net in your message persistence layer (for both streaming and non-streaming paths).

8. Background Persistence for Streaming¶

Symptom¶

Messages lost on page refresh. User sends message, gets response, refreshes -- the response is gone.

Root Cause¶

The on_complete callback runs inside the SSE generator's post-loop code. If the client disconnects mid-stream, Starlette cancels the generator via GeneratorExit. The callback never fires.

Fix¶

Use Starlette's BackgroundTask on the StreamingResponse:

state = StreamState()  # shared between generator and background task

async def persist():
    full_text = "".join(state.accumulated_text)
    await save_message(full_text, state.telemetry)

return StreamingResponse(
    sse_generator(events, state=state),
    background=BackgroundTask(persist),  # runs even on disconnect
)

9. Celery Beat Schedule File¶

If using Celery Beat for nightly consolidation:

celery.conf.update(
    beat_schedule_filename="/tmp/celerybeat-schedule",  # NOT None
)

Setting beat_schedule_filename=None crashes PersistentScheduler which tries to open a shelve at None path.

10. Consolidation Triggers¶

Trigger	When	Layers Written
Auto-consolidation	After every `run()` / `stream_typed()`	All 5 (from MEMORY_EXTRACT ops)
Self-reflection	After consolidation (if enabled)	Semantic (insights)
Working memory	During consolidation	Working (24h TTL, auto-expires)
Manual trigger	API endpoint / Celery task	All (SleepConsolidator)
Nightly	Celery Beat cron	All (SleepConsolidator per tenant)

All triggers require: - auto_consolidate=True in config - scope is not None passed to run() / stream_typed() - Backends wired (not empty orchestrator)

11. Checklist¶

Before going live, verify:

[ ] memory_nodes count increases after a conversation
[ ] memory_vectors count increases (embeddings stored)
[ ] No <MEMORY_EXTRACT> visible in chat UI
[ ] Messages persist after page refresh
[ ] Document downloads work immediately (no 404)
[ ] Celery Beat running without errors
[ ] Embedding service reachable from the API container
[ ] Consolidation logs visible at INFO level