Skip to content

symfonic.capabilities.memory.overlay

overlay

The scope a cycle works on, held in memory until the cycle commits.

A consolidation cycle has to be able to read what it has already written -- prune_orphans runs after meta_nodes and must see the meta nodes, semantic_merge retracts what earlier phases strengthened -- while nothing it writes may be visible to anybody else until the whole cycle commits. That is a read-your-writes overlay, and the temptation is to hand-roll one: a dict of pending nodes, a set of deleted ids, and a filter matcher written from memory to make query_nodes behave.

That matcher is the trap. The filter vocabulary is real -- layer, label, label_prefix, arbitrary property equality, prefix-visibility by scope path, subtree containment, the limit rules -- and a second implementation of it that disagrees anywhere is a phase that silently reads a different graph inside a cycle than outside one. So this overlay does not reimplement any of it: it is :class:~symfonic.memory.backends.in_memory.InMemoryGraphBackend, the reference implementation of that vocabulary, pre-loaded with the scope. Reads then behave in the overlay exactly as they behave in the durable store, because the code answering them is the code the store's own tests pin.

One honest caveat: reads inside a cycle are answered by the in-memory matcher, so if a durable backend ever disagreed with it about filter semantics -- a property comparison, a limit edge case -- a phase would read slightly differently inside a cycle than outside one. The vocabulary is documented and shared, and this is the implementation its tests pin, which is the strongest available position; it is not a proof that every backend agrees.

Subclassed rather than composed for one reason: :meth:seed has to place rows into the backend's tables as they were stored, and every public write path stamps the writing scope's path into the property bag. An ancestor node loaded through add_node would come back stamped with the cycle's scope and become visible to scopes that must not see it. Seeding is the one operation that must bypass the write path, and a subclass is where that is legitimate.

IncompleteScopeProjection

Bases: RuntimeError

A bounded snapshot cannot prove that it contains the complete scope.

ScopeOverlay 

ScopeOverlay()

Bases: InMemoryGraphBackend

An in-memory graph pre-loaded with one scope's durable rows.

Source code in src/symfonic/capabilities/memory/overlay.py 
63
64
65
def __init__(self) -> None:
    super().__init__()
    self._seeded: set[str] = set()

seed async 

seed(durable: Any, scope: TenantScope) -> None

Load everything a cycle over scope can read. Once per scope.

Three reads, because the two node directions answer different questions and neither substitutes for the other: query_nodes is prefix-visibility-scoped and returns what this scope may see (its ancestors), query_subtree returns what this scope and its children produced. A cycle does both, so the overlay holds both.

A full page cannot establish completeness with these unpaged ports. Refuse before populating the overlay; an incomplete adjacency list would otherwise turn a connected node into a deletable orphan.

Source code in src/symfonic/capabilities/memory/overlay.py 
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
async def seed(self, durable: Any, scope: TenantScope) -> None:
    """Load everything a cycle over ``scope`` can read. Once per scope.

    Three reads, because the two node directions answer different
    questions and neither substitutes for the other: ``query_nodes`` is
    prefix-visibility-scoped and returns what this scope may *see* (its
    ancestors), ``query_subtree`` returns what this scope and its children
    *produced*. A cycle does both, so the overlay holds both.

    A full page cannot establish completeness with these unpaged ports.
    Refuse before populating the overlay; an incomplete adjacency list
    would otherwise turn a connected node into a deletable orphan.
    """
    key = f"{scope.tenant_id}:{getattr(scope, 'path', '')}"
    if key in self._seeded:
        return
    visible = list(await durable.query_nodes(scope, {}, limit=CONSOLIDATION_CANDIDATE_LIMIT))
    descendants = list(
        await durable.query_subtree(scope, {}, limit=CONSOLIDATION_CANDIDATE_LIMIT)
    )
    nodes = {str(node.id): node for node in (*visible, *descendants)}
    page = list(await durable.query_edges(scope, None, _PAGE, 0))
    if (
        len(visible) >= CONSOLIDATION_CANDIDATE_LIMIT
        or len(descendants) >= CONSOLIDATION_CANDIDATE_LIMIT
        or len(nodes) > CONSOLIDATION_CANDIDATE_LIMIT
        or len(page) >= _PAGE
    ):
        raise IncompleteScopeProjection(
            "consolidation snapshot completeness exceeds the bounded projection"
        )
    for node in nodes.values():
        # Straight into the table, keeping the stored scope path: see the
        # module docstring on why this cannot go through ``add_node``.
        self._nodes[node.tenant_id][str(node.id)] = node
    for edge in page:
        self._edges[edge.tenant_id][str(edge.id)] = edge
    self._seeded.add(key)