Skip to content

symfonic.memory.dedup

dedup

Fuzzy-label deduplication for graph-backed memory writes.

Single implementation of the write-time near-duplicate collapse shared by every fuzzy-collapse write path: the engine's MEMORY_EXTRACT op path (SymfonicAgent._resolve_dedup_id delegates here) and the self-reflection insight path (:class:symfonic.tools.learning.CommitToMemoryTool).

Match contract (v8.16 -- adopter-reported boundary: restating a fact WITH EXTRA DETAIL escaped the pure-ratio match):

  1. Candidates are fetched by a 20-char label prefix (:data:DEDUP_PREFIX_LEN). This is a query optimisation, not the correctness criterion.
  2. A candidate matches when EITHER a. full-label SequenceMatcher ratio >= :data:DEDUP_SIMILARITY_THRESHOLD (tail rewording), OR b. one normalised label CONTAINS the other and the shorter side is at least :data:DEDUP_SUBSUME_MIN_CHARS chars (extra-detail / terser restatement). SequenceMatcher structurally punishes length asymmetry (ratio = 2M/(len_a+len_b)), so "Uses pytest and ruff" restated as "Uses pytest and ruff for Python projects with strict linting" scores ~0.5 -- containment is the correct test for that class.

The full collapse/miss matrix per restatement class is pinned by tests/memory/test_restatement_corpus.py -- change the thresholds or rules above and that matrix shows exactly which classes flip. Restatements that reword the fact's OPENING (first 20 chars) still do not collapse here; that class needs semantic-equivalence judgment and is handled offline by the consolidation-time semantic-merge phase (symfonic.core.learning.phases_semantic_merge).

Ownership guard: query_nodes is ancestor-visible (a child scope sees a parent scope's rows), but collapsing a write into a node the write scope does not OWN would re-stamp that node's scope_path on upsert and silently move an ancestor's memory into the child scope -- the same visibility != ownership class the retraction path guards against. Only nodes materialised at exactly the write scope are eligible canonicals.

find_canonical_node async 

find_canonical_node(
    graph: Any,
    scope: TenantScope,
    *,
    layer: MemoryLayer,
    label: str,
    threshold: float = DEDUP_SIMILARITY_THRESHOLD,
    requested_id: str | None = None,
) -> MemoryNode | None

Return the existing node the write should collapse into, if any.

Queries graph for same-layer nodes sharing the label prefix and returns the first candidate owned by exactly scope that matches per :func:labels_match. When requested_id names a fetched candidate, that node is returned as-is (the upsert will update the row the caller already targets). Soft-retracted nodes never match (query_nodes filters them by default), so a fact retracted as wrong is re-learned as a fresh node instead of resurrecting the retracted row.

All errors are swallowed and logged at DEBUG: a dedup lookup failure must never break the write it guards. Returns None when graph is missing or does not expose query_nodes (stub/spy backends).

Source code in src/symfonic/memory/dedup.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
async def find_canonical_node(
    graph: Any,
    scope: TenantScope,
    *,
    layer: MemoryLayer,
    label: str,
    threshold: float = DEDUP_SIMILARITY_THRESHOLD,
    requested_id: str | None = None,
) -> MemoryNode | None:
    """Return the existing node the write should collapse into, if any.

    Queries ``graph`` for same-layer nodes sharing the label prefix and
    returns the first candidate owned by exactly ``scope`` that matches
    per :func:`labels_match`.  When ``requested_id`` names a fetched
    candidate, that node is returned as-is (the upsert will update the
    row the caller already targets).  Soft-retracted nodes never match
    (``query_nodes`` filters them by default), so a fact retracted as
    wrong is re-learned as a fresh node instead of resurrecting the
    retracted row.

    All errors are swallowed and logged at DEBUG: a dedup lookup failure
    must never break the write it guards.  Returns ``None`` when
    ``graph`` is missing or does not expose ``query_nodes`` (stub/spy
    backends).
    """
    label = (label or "").strip()
    if not label or graph is None or not hasattr(graph, "query_nodes"):
        return None

    # The whole lookup sits under one guard: mock/stub backends return
    # non-list candidates or partial node shapes, and none of that may
    # break the write this dedup merely optimises.
    try:
        candidates = list(
            await graph.query_nodes(
                scope,
                layer=layer,
                label_prefix=label[:DEDUP_PREFIX_LEN],
            )
        )[:DEDUP_QUERY_LIMIT]

        if requested_id:
            for node in candidates:
                if str(getattr(node, "id", "")) == str(requested_id):
                    return node

        write_path = materialise_scope_path(scope)
        for node in candidates:
            node_label = getattr(node, "label", "") or ""
            if not labels_match(node_label, label, threshold=threshold):
                continue
            stored = stored_scope_path(
                getattr(node, "properties", None),
                getattr(node, "tenant_id", ""),
            )
            if stored != write_path:
                # Visible but owned by an ancestor scope: not ours to
                # collapse (visibility != ownership).
                continue
            return node
    except Exception:
        logger.debug("dedup: lookup failed, skipping dedup", exc_info=True)
    return None

labels_match 

labels_match(
    stored: str, incoming: str, *, threshold: float
) -> bool

The shared same-fact criterion: fuzzy ratio OR subsumption.

Source code in src/symfonic/memory/dedup.py 
85
86
87
88
89
90
91
92
93
def labels_match(stored: str, incoming: str, *, threshold: float) -> bool:
    """The shared same-fact criterion: fuzzy ratio OR subsumption."""
    a, b = normalise_label(stored), normalise_label(incoming)
    if not a or not b:
        return False
    if SequenceMatcher(None, a, b).ratio() >= threshold:
        return True
    shorter, longer = (a, b) if len(a) <= len(b) else (b, a)
    return len(shorter) >= DEDUP_SUBSUME_MIN_CHARS and shorter in longer

normalise_label 

normalise_label(label: str) -> str

Whitespace-collapsed, lowercased, terminal-punctuation-stripped form.

v8.17: terminal punctuation is stripped because extractor LLMs vary on emitting a trailing period, and "uses pytest and ruff." is not a substring of "uses pytest and ruff for python projects" -- one stray dot silently defeated the containment rule (adopter-verified; their workaround was prompt-side "no terminal punctuation"). Internal punctuation is preserved -- it carries meaning.

Source code in src/symfonic/memory/dedup.py 
71
72
73
74
75
76
77
78
79
80
81
82
def normalise_label(label: str) -> str:
    """Whitespace-collapsed, lowercased, terminal-punctuation-stripped form.

    v8.17: terminal punctuation is stripped because extractor LLMs vary
    on emitting a trailing period, and "uses pytest and ruff." is not a
    substring of "uses pytest and ruff for python projects" -- one
    stray dot silently defeated the containment rule (adopter-verified;
    their workaround was prompt-side "no terminal punctuation").
    Internal punctuation is preserved -- it carries meaning.
    """
    collapsed = " ".join(str(label or "").lower().split())
    return collapsed.rstrip(".!?;:,")