Skip to content

Guide 09 — Consolidation & Deep Sleep

Deep Sleep is the offline maintenance engine that keeps the knowledge graph healthy. SleepConsolidator runs 11 distinct phases that strengthen recurring knowledge, prune stale nodes, generate meta-nodes, correct SOUL schemas, decay importance, summarise old episodes, build synthetic links, and promote episodic patterns into draft procedural skills.

This guide covers the three entrypoints (quick_nap, run, nightly_nap), every phase, the FrameworkConfig knobs that tune each phase, observability via ConsolidationReport, and operator scheduling for nightly_nap.

Prerequisites

Three entrypoints

Entrypoint Scope Intended cadence
quick_nap(scope) Phase 1 + optional Phase 5 + optional Phase 10 Every N turns via quick_nap_interval
run(scope) Full 11-phase roster Manual / on-demand / test fixtures
nightly_nap(scope) Full roster (v7.0 T12 alias for run) Cron / celery-beat / APScheduler once per quiet window per tenant

quick_nap — per-turn lightweight consolidation

cfg = FrameworkConfig(quick_nap_interval=10)
# Every 10th turn, _maybe_schedule_quick_nap fires the subset.

The engine auto-forwards three kwargs to quick_nap from FrameworkConfig:

  • soul_schema — from config.domain.soul_schema. Gates Phase 5.
  • max_entries — from episodic_summarization_max_entries. Phase 10 threshold override.
  • summarize_batch — from episodic_summarization_batch_size. Phase 10 batch size override.

run — the full roster

Returns a ConsolidationReport. The canonical per-tenant batch. Can be invoked manually or via the /memory/consolidate HTTP endpoint.

nightly_nap — operator-scheduled full roster

cfg = FrameworkConfig(
    nightly_nap_enabled=True,
    nightly_nap_cron="0 3 * * *",  # 03:00 UTC daily — recommendation only
)

The engine does not parse the cron string or schedule anything. The knob exists as a documented recommendation so every deployment converges on the same cadence. Operators wire their own scheduler (APScheduler / celery-beat / external cron) and invoke agent._sleep_consolidator.nightly_nap(scope) per tenant.

The 11 phases

Source: src/symfonic/core/learning/consolidation.py + src/symfonic/core/learning/phases.py + phases_episodic.py / phases_maintenance.py / phases_synthetic.py / phases_procedural_promotion.py.

Phase Name in phases_run Effect Tunable knob
1 strengthen Boost importance for recurring nodes. Signal is access_count + phase1_spreading_weight * spreading_access_count. phase1_spreading_weight (v6.2 T02)
1.5 cooccurrence Create CO_OCCURRED edges between co-temporal nodes. —
2 tag_risk Mark nodes with negative-sentiment context. —
2.5 pending_edges Write inferred pending_connections as real edges. Skipped when no pending list is passed. —
3 prune_orphans Delete stale low-confidence orphan nodes. TTL-expired nodes pruned unconditionally regardless of importance. —
4 meta_nodes Cluster raw domain nodes by label prefix into META:<prefix> summaries. Excludes existing META: nodes from clustering input to prevent META:META cascade. —
5 soul_corrections Apply SOUL schema updates from SOUL_CORRECTION nodes. Skipped when no soul_schema= arg passed. —
8 working_ttl Prune WORKING-layer nodes with expired TTL. —
9 decay_importance Reduce importance of stale nodes. Skips importance >= 8.0 (CRITICAL_IMPORTANCE_THRESHOLD) and labels starting with SOUL: or AGENT_IDENTITY: — protects user profile and agent persona from age-based attrition. —
10 episodic_summary Compress old episodes into semantic meta-nodes. episodic_summarization_max_entries, episodic_summarization_batch_size
11 synthetic_links Emit SYNTHETIC_LINK edges between semantic nodes that co-occur in episodic content (via [semantic:<label>] markers). synthetic_link_min_co_count (v6.2 T03)
12.5 entity_links Autonomously extract entity mentions from natural-language episodic content, mint Entity:<kind>:<surface> semantic nodes, and emit MENTIONS edges so spreading activation can cross months/years. Off by default. enable_entity_linker, entity_linker_extractor, entity_linker_min_mention_count, entity_linker_max_episodics_per_run, entity_linker_confidence_threshold (v7.2 Item 12)
12 procedural_promotion Promote recurring episodic patterns to draft procedural skills (status="draft"). promotion_min_pattern_count, promotion_recency_days, promotion_max_drafts_per_run

Why 11 phases in a 12-slot roster? Phases 6 and 7 are reserved slots that do not currently have bodies — the numbering was preserved from an earlier design to keep test files and downstream references stable.

ConsolidationReport observability

report = await agent._sleep_consolidator.run(scope)

print(f"Phases that fired: {report.phases_run}")
print(f"Nodes strengthened: {report.nodes_strengthened}")
print(f"Nodes pruned: {report.nodes_pruned}")
print(f"Meta-nodes created: {report.meta_nodes_created}")
print(f"SOUL updates: {report.soul_updates}")
print(f"Working TTL pruned: {report.working_ttl_pruned}")
print(f"Nodes decayed: {report.nodes_decayed}")
print(f"Episodes summarized: {report.episodes_summarized}")
print(f"Synthetic edges: {report.edges_created}")
print(f"Draft skills promoted: {report.draft_skills_promoted}")
print(f"Errors: {report.errors}")

phases_run is the observability-only list populated by each phase implementation when the phase actually executes (even if its mutation count is zero). It lets operators distinguish "phase ran and found nothing" from "phase was short-circuited and never executed".

Example distinction:

  • "soul_corrections" in report.phases_run AND report.soul_updates == 0 — Phase 5 ran but no corrections were found in the lookback window.
  • "soul_corrections" not in report.phases_run — Phase 5 was skipped because no soul_schema= was passed.

Decay exemptions (Phase 9)

decay_importance() skips two classes of nodes to protect high-signal persistent state:

  • High-importance nodes — importance >= 8.0 (CRITICAL_IMPORTANCE_THRESHOLD in phases_maintenance.py). A node deliberately set to importance 8+ by the agent is asserting persistent significance; decay should not quietly degrade it.
  • Identity labels — labels starting with SOUL: / SOUL / AGENT_IDENTITY: / AGENT_IDENTITY. These carry the user profile and agent persona. Decaying them would reset what the agent "knows" about the tenant every night.

Contract pinned by tests/core/learning/test_phase_9_decay_exemptions.py. Do not remove the exemptions — a v6.0.x regression here is why they exist.

META:META idempotence (Phase 4)

generate_meta_nodes excludes nodes whose label starts with "META:" from the clustering input. Phase 4 clusters raw domain nodes; META: nodes are outputs of this phase, not inputs to it. Without the exclusion, every existing META:foo node shares the prefix "META" (first colon-split token) and on a second consolidation run those outputs become inputs, synthesising a META:META cascade.

Contract pinned by test_idempotence_second_run_is_stable: report_2.meta_nodes_created == 0 on an immediate second run against an unchanged graph.

Phase 12 promotion (autonomous learning)

Phase 12 scans recent episodic entries, extracts repeated action patterns, and synthesises candidate procedural skills. Two extractors are available: the default regex extractor (v1, lightweight) and the LLM-backed extractor (v7.6 opt-in, structured drafts with conditional rules).

Regex extractor (default)

Heuristics applied in order:

  1. Explicit metadata['action_type'] annotation.
  2. Action-verb prefix regex on the entry content (e.g. "I ran deploy", "Then I called X").
  3. (v7.4.7 opt-in) When the above miss, fall back to metadata['tool_calls'][0]['name'] so tool-call-heavy agents whose auto-turn content starts with "User: ..." still produce signal.
Knob Default Effect
promotion_min_pattern_count 3 Minimum occurrences before pattern becomes a draft skill
promotion_recency_days 30 Lookback window
promotion_max_drafts_per_run 5 Per-run cap to prevent review-queue overflow
phase_12_action_type_from_tool_calls (v7.4.7) False Opt into the tool-call fallback heuristic

LLM extractor (v7.6 opt-in)

Set FrameworkConfig.phase_12_use_llm_extractor=True to run the LLM-backed extractor INSTEAD of the regex one. It produces structured ProceduralDraft records distinguishing four kinds:

  • action — repeated tool usage that should become a learned habit.
  • preflight — pre-flight check ("before X, call Y"). The most load-bearing kind for v7.6 since the regex extractor structurally cannot represent it.
  • recovery — error-recovery rule ("after E, call R").
  • guardrail — negative rule ("never X — use Y instead").
Knob Default Effect
phase_12_use_llm_extractor False Flip to opt in. Either-or with the regex path.
phase_12_llm_model claude-haiku-4-5 Model identifier surfaced to calculate_cost and the telemetry log line. Haiku's sweet spot is structured extraction.
phase_12_llm_max_episodes_per_run 100 Hard cap on episodes fed to the prompt (older entries dropped).
phase_12_llm_max_drafts_per_run 5 Per-run cap, mirrors the regex extractor's setting.

The LLM extractor depends on the v7.4.7 tool-call producer — without metadata['tool_calls'] on episodic entries, the LLM has no grounding for preflight or recovery rules. Run the producer for at least 4–6 weeks before defaulting the LLM extractor on, so a real corpus exists to validate the prompt against.

LLM drafts land with metadata['source']='phase_12_llm_extractor', distinct from the regex extractor's 'phase_12_promotion', so log parsers can split adoption.

Telemetry

Both extractors emit a single summary log line per run:

Phase 12: scanned=N eligible_patterns=N drafts_created=N \
  drafts_rejected_by_dedup=N min_count=N recency_days=N         (regex)

Phase 12 LLM: tenant=<id> drafts_emitted=K drafts_kept=K' \
  drafts_rejected_by_dedup=K'' max_drafts=N model=<name>        (LLM)

The drafts_rejected_by_dedup field (v7.4.8) distinguishes "no patterns above threshold" from "patterns reached threshold but every draft was merged by SemanticMerge into a pre-existing skill".

Quality gate

Phase 12 always stores status="draft" regardless of skill_auto_approve. No learned skill becomes active without explicit human approval via POST /procedures/{id}/approve. See Autonomous Learning for the full review pipeline.

SemanticMerge inside store_skill dedups against pre-existing procedural nodes so repeated consolidation runs do not flood the review queue. v7.5.0 added an authored-tier guard that exempts skills whose properties['source'] starts with authored: — see Guide 10 — Authored procedural skills for the seeding API.

Episodic summarisation (Phase 10)

Phase 10 only runs when the episodic layer holds more than episodic_summarization_max_entries entries for the tenant (default 100). When it runs, it takes the oldest episodic_summarization_batch_size entries (default 50), summarises them into a single semantic meta-node, and deletes the originals.

quick_nap accepts per-call overrides so operators can tune each tenant independently without rebuilding the consolidator:

await consolidator.quick_nap(
    scope,
    soul_schema=my_schema,
    max_entries=50,           # override default 100
    summarize_batch=25,       # override default 50
)

Synthetic linking (Phase 11)

Phase 11 cross-pollinates semantic nodes that co-occur in episodic content via [semantic:<label>] markers. When a pair of labels co-occurs in synthetic_link_min_co_count (default 2) or more episodic entries, a SYNTHETIC_LINK edge is emitted.

cfg = FrameworkConfig(synthetic_link_min_co_count=3)
# Emits fewer but stronger synthetic edges.

The floor is 2 — allowing 1 would reduce Phase 11 to the existing Phase 1.5 CO_OCCURRED edge builder.

v7.2 / Roadmap Item 12. Phase 12.5 autonomously builds the episodic -> semantic entity graph that long-range spreading activation requires. Phase 11 only links semantic nodes that share an explicit [semantic:<label>] marker; Phase 12.5 instead scans the natural language of episodic entries, extracts entity mentions, mints canonical Entity:<kind>:<surface> semantic nodes, and emits MENTIONS edges (provenance=entity_linker) with the originating episodic ids on MemoryEdge.source_episodic_ids. Spreading activation can then reach months-old or years-old episodics from a present-tense cue with no manual semantic.link() plumbing.

Source: src/symfonic/core/learning/phases_entity.py. Phase order: runs between Phase 11 (synthetic_links) and Phase 12 (procedural_promotion) inside SleepConsolidator.run. Default off.

When to enable

Turn the phase on when you want the agent to recall things like "I saw a server's shirt at the mall — that reminds me of the restaurant where they wore the same uniform" without manually annotating every episodic with [semantic:<label>] markers. Without this phase the agent can still recall what was said across time (episodic similarity), but it cannot autonomously cross-link entities into the semantic graph.

The end-to-end demo lives at examples/full_showcase/steps/memory_scenario.py (Turn 22) — three episodics about the same restaurant planted at simulated -90d, -30d, and now get linked into a single Entity:place:bistro node after a single consolidation pass, and a present-tense cue ("mall sighting") activates the older episodics through the new edges.

How to enable

from symfonic.agent import FrameworkConfig

config = FrameworkConfig(
    enable_entity_linker=True,
    entity_linker_extractor="spacy",      # or "regex" / "llm"
    entity_linker_min_mention_count=2,    # framework default
    entity_linker_confidence_threshold=0.5,
)

In a scaffolded project, the same knobs are reachable via .env:

ENABLE_ENTITY_LINKER=true
ENTITY_LINKER_EXTRACTOR=spacy

Extractor trade-offs

The extractor is pluggable via entity_linker_extractor. Pick one:

Strategy Extra Recall Latency Notes
regex (default) none low ~30 ms / run Catches capitalised proper-noun tokens at confidence=0.4. The default threshold 0.5 filters every regex candidate out by design — using regex meaningfully requires lowering entity_linker_confidence_threshold to 0.4 (or below). Ships zero-dep so a fresh install boots without spaCy or LLM cost.
spacy [entity-linker-spacy] high ~1 s / run Production-grade NER. Requires pip install symfonic-core[entity-linker-spacy] and python -m spacy download en_core_web_sm (the extra pulls the package but not the language model — air-gapped deployments must bundle the model separately). ~50 MB on disk. Best quality / cost ratio.
llm [entity-linker-llm] (marker only) highest ~30 s / run uncached Uses the agent's bound chat model. No new dependencies, but every consolidation cycle incurs token cost on whatever provider is wired. Idempotency is best-effort — results may drift across model versions; the content-hash cache mitigates within a single deployment but not across upgrades.

The default regex keeps the install graph small and CI deterministic; production deployments typically switch to spacy once they have an embedding-model budget.

Tuning knobs

The five FrameworkConfig fields, with the production-sensible values for each extractor:

Knob Default Effect Production tuning
enable_entity_linker False Master gate. The phase is a no-op even when SleepConsolidator is wired with the knob. Flip to True once the extractor is chosen.
entity_linker_extractor "regex" One of "regex" / "spacy" / "llm". Ignored when the master gate is off. "spacy" is the recommended starting point.
entity_linker_min_mention_count 2 Minimum number of distinct episodics in which an entity surface form must appear before a semantic node is minted. Mirrors synthetic_link_min_co_count for Phase 11. Leave at 2; raise to 3 on noisy corpora to suppress one-off mentions.
entity_linker_max_episodics_per_run 200 Max episodic entries scanned per consolidation cycle. Bounds per-run extractor cost. Raise for tenants with high episodic write volume; the regex path scales linearly, spaCy super-linearly.
entity_linker_confidence_threshold 0.5 Per-candidate minimum confidence from the extractor. Below-threshold candidates are dropped before the mention-count gate. Lower to 0.4 if running the regex extractor; keep at 0.5+ for spaCy / LLM.

What the phase emits

  • Entity:<kind>:<surface> semantic nodes — kind is the extractor's verdict (person / place / event / object / other); surface is the canonical form (lower-cased for object/other, case-preserved for person/place/event).
  • MENTIONS edges between entity nodes that co-occurred in the same episodic, with provenance="entity_linker" and source_episodic_ids listing the episodics that justify each edge (capped at 20 per edge to bound row size).
  • MENTIONS weight initialised at 0.5, reinforced by +0.1 per re-confirming run, capped at 2.0.

Idempotency contract

Re-running consolidation against an unchanged graph produces the same edge set — no duplicates, no new edges. Each (src, dst, provenance="entity_linker") triple is upserted, not appended, and source_episodic_ids is a set-union of previously-recorded ids. Pinned by integration tests under tests/core/learning/test_phase_entity_links_*.py.

Performance budget

Extractor Added latency per consolidation cycle
regex ~30 ms
spacy ~1 s
llm ~30 s without cache; near-zero on a warm content-hash cache

The phase respects entity_linker_max_episodics_per_run as a hard ceiling — the LLM path will not scan beyond the cap regardless of how many new episodics accumulated since the last run.

Scheduling nightly_nap

The engine does not ship a scheduler. Wire your own:

Option 1 — APScheduler

from apscheduler.schedulers.asyncio import AsyncIOScheduler

async def run_nightly(agent, tenants: list[str]):
    for tenant_id in tenants:
        scope = FrameworkTenantScope(tenant_id=tenant_id).to_memory_scope()
        report = await agent._sleep_consolidator.nightly_nap(scope)
        logger.info("nightly_nap %s phases=%s", tenant_id, report.phases_run)

scheduler = AsyncIOScheduler()
scheduler.add_job(run_nightly, "cron", hour=3, args=[agent, tenant_ids])
scheduler.start()

Option 2 — Celery beat

from celery import Celery
from celery.schedules import crontab

celery_app = Celery("symfonic", broker="redis://localhost:6379/0")

@celery_app.task(name="nightly_nap")
def nightly_nap_task(tenant_id: str):
    import asyncio
    scope = FrameworkTenantScope(tenant_id=tenant_id).to_memory_scope()
    report = asyncio.run(agent._sleep_consolidator.nightly_nap(scope))
    return report.to_dict()

celery_app.conf.beat_schedule = {
    "nightly-nap": {
        "task": "nightly_nap",
        "schedule": crontab(hour=3, minute=0),
        "args": ("acme-corp",),
    },
}

Option 3 — External cron

# /etc/cron.d/symfonic-nightly-nap
0 3 * * * www-data curl -X POST http://localhost:8000/api/v1/memory/consolidate \
    -H "X-Tenant-ID: acme-corp"

Auto-turn consolidation

When auto_consolidate=True (default), the engine parses MEMORY_EXTRACT blocks from LLM responses and commits them to the appropriate memory layers in a background task. This is distinct from Deep Sleep — it runs every turn, not offline.

  • Auto-turn _consolidate — synchronous within the request, parses structured ops, writes to Working (24h TTL) + Episodic + Semantic.
  • Deep Sleep SleepConsolidator — offline batch, runs maintenance across all stored data.

Both paths credential-scrub before persisting (v6.1.9). Both paths emit audit entries. The two are complementary — auto-turn captures every turn; Deep Sleep maintains long-term health.

Retracting a false-positive memory (retract_node)

The extraction protocol is normally additive — upsert_node, set_trigger, create_event, register_skill. That leaves one gap: a memory the agent stored wrongly. Because reinforcement is frequency-based (see Limitations below), a wrong-but-repeated fact or skill would otherwise get stronger, not corrected.

The retract_node op closes the reactive half of that gap. When the user explicitly corrects or invalidates a stored memory ("no, the DB host is X", "stop doing Y that way", "forget that"), the model emits:

{
  "type": "retract_node",
  "layer": "semantic",
  "payload": {"id": "production-database", "reason": "user corrected the host"}
}

layer is semantic for facts and procedural for skills. The engine (SymfonicAgent._retract_memory) resolves the target node — by id, else by a fuzzy label match — and soft-retracts it:

  • Sets properties["retracted"] = True plus retracted_at and retraction_reason (an audit trail).
  • The node is not deleted. GraphMemoryStore.query_nodes excludes retracted nodes from every read path (hydration, routing, dedup) by default, so the wrong memory stops influencing the agent immediately — reversibly and auditably.
  • Deep Sleep's prune_retracted phase hard-deletes nodes that have been retracted longer than the grace window (default 7 days), reclaiming the row. The ConsolidationReport.retracted_nodes_pruned counter reports how many.

This is deliberately conservative: retraction fires only on an explicit correction, never on mere absence of mention, and the soft-delete keeps a window to reverse a wrong retraction. Only graph-backed layers (semantic, procedural) are supported — episodic/vector retraction is not yet wired.

Limitations — what consolidation can not catch

Be honest with yourself about the failure mode consolidation does not solve on its own:

  • Reinforcement is frequency-based, not correctness-based. Phase 1 strengthens whatever recurs. There is no outcome/reward signal in the pipeline — consolidation never learns whether an action succeeded. So a behavior the agent learned wrong and keeps repeating gets reinforced, not flagged.
  • There is no autonomous truth oracle. Nothing lets the engine spontaneously decide a recurring memory was a mistake. The risk_node tagging is keyword sentiment on the node text, not a judgment about outcomes.
  • The real backstops are external signals, not self-detection:
  • Human review gate on learned skills — Phase 12 writes every promoted skill as status="draft"; nothing auto-activates without POST /procedures/{id}/approve.
  • Explicit correction — the retract_node path above, and SOUL corrections from user_manual_edit nodes.
  • MetacognitiveMiddleware gating live skill usage.

Future work. Closing the proactive half — demoting a memory because the actions it drove led to bad outcomes — needs an outcome/feedback signal attached to episodic traces (a success/failure marker on the turn, or a downstream error correlated back to the procedure). That signal does not exist yet; an outcome-aware demotion phase would consume it.

See also