symfonic.capabilities.memory.fencing

fencing

A cycle that lost its scope stops writing, at the write rather than the end.

Checking the lease once before publishing is enough only if a cycle's whole output is staged and nothing touches the store until then. On this roster it is not: ten of the phase modules call add_node, update_node, delete_node, add_edge or delete_edge on the graph directly -- strengthening importances, pruning orphans, decaying, expiring, retracting a merged duplicate, minting entities and their mention edges. Nineteen call sites that never go near the write coordinator.

So the window is real and it is wide. Deep Sleep can run for minutes -- five of its phases may call a model, and a quick nap has been measured at 28 seconds -- and if the lease expires halfway, worker A carries on mutating while worker B takes the scope and does the same. Two cycles interleaving direct writes is the outcome the lease exists to prevent, and a fence at the end prevents none of it.

Two mechanisms, and both are needed.

Renewal keeps a slow-but-living worker in possession: a heartbeat re-stamps the deadline while the cycle runs, so a model call outlasting the TTL costs nothing. Without it every long phase is a handover waiting to happen, and expiry stops meaning "this worker died" -- which is the only thing it should mean.

Fencing is what happens when renewal was not enough -- the process was suspended, the database was unreachable, the worker really did stall. Every durable mutation asks first, and once the answer is no the cycle raises :class:LeaseLost rather than writing. Structural rather than a rule each phase has to remember: a phase written next year gets it by construction, and one that forgot would be exactly the phase nobody thought to check.

Reads pass through untouched. A cycle that has lost its scope may still finish reading -- it is about to stop -- and gating reads would turn a handover into an exception storm in code that was only looking.

Fence

Fence(held: Callable[[], Awaitable[bool]], locked: Callable[[], Awaitable[bool]] | None = None)

One cycle's authority over its scope, and whether it still has it.

Per cycle rather than per graph wrapper, because the wrapper is built once by the phase factory and reused by every cycle that runs after it: a latch living on the wrapper would let one lost lease disable consolidation for the rest of the process.

Parameters:

Name	Type	Description	Default
held	Callable[[], Awaitable[bool]]	the cheap check, made before every durable mutation while the cycle runs. A plain read: it must not take locks, because it is asked dozens of times across a cycle that can last minutes.	required
locked	Callable[[], Awaitable[bool]] | None	the commit-time check, which takes the lease row's lock and keeps it until the transaction ends. Defaults to held, which is correct only where nothing can interleave between the answer and the write -- one process, one loop. Every deployment with a database passes both.	None

Source code in src/symfonic/capabilities/memory/fencing.py

def __init__(
    self,
    held: Callable[[], Awaitable[bool]],
    locked: Callable[[], Awaitable[bool]] | None = None,
) -> None:
    """
    Args:
        held: the cheap check, made before every durable mutation while the
            cycle runs. A plain read: it must not take locks, because it is
            asked dozens of times across a cycle that can last minutes.
        locked: the commit-time check, which takes the lease row's lock and
            keeps it until the transaction ends. Defaults to ``held``, which
            is correct only where nothing can interleave between the answer
            and the write -- one process, one loop. Every deployment with a
            database passes both.
    """
    self._held = held
    self._locked = locked or held
    self._lost = False

lost property

lost: bool

Whether this cycle has already been found to have lost its scope.

check async

check(doing: str) -> None

Raise :class:LeaseLost unless this cycle still holds its scope.

Source code in src/symfonic/capabilities/memory/fencing.py

async def check(self, doing: str) -> None:
    """Raise :class:`LeaseLost` unless this cycle still holds its scope."""
    await self._ask(self._held, doing)

check_locked async

check_locked(doing: str) -> None

Like :meth:check, and the answer stays true until the commit ends.

The distinction is the whole of the check-to-write race: check reports the past tense the moment it returns, and a batch applied afterwards can land under a lease that lapsed in between. This one holds the row, so a rival acquisition waits for this transaction instead of overlapping it.

Source code in src/symfonic/capabilities/memory/fencing.py

async def check_locked(self, doing: str) -> None:
    """Like :meth:`check`, and the answer stays true until the commit ends.

    The distinction is the whole of the check-to-write race: ``check``
    reports the past tense the moment it returns, and a batch applied
    afterwards can land under a lease that lapsed in between. This one
    holds the row, so a rival acquisition waits for this transaction
    instead of overlapping it.
    """
    await self._ask(self._locked, doing)

FencedGraph

FencedGraph(graph: Any)

The graph a cycle writes through while it still holds the scope.

Wraps rather than subclasses: the phases take "a graph" duck-typed, the real one has a surface this module has no business restating, and anything not named in :data:FENCED_MUTATIONS is forwarded exactly as it was.

Parameters:

Name	Type	Description	Default
graph	Any	the store the phases would otherwise have been handed.	required

Source code in src/symfonic/capabilities/memory/fencing.py

def __init__(self, graph: Any) -> None:
    """
    Args:
        graph: the store the phases would otherwise have been handed.
    """
    self._graph = graph

unfenced property

unfenced: Any

The wrapped store, for the runtime's own reads and for equality.

Named rather than private: a caller that legitimately needs the real object -- a test asserting on rows, a phase factory rewrapping -- should say so, instead of reaching for _graph and coupling to the field.

LeaseLost

Bases: MemoryContractError

This cycle no longer holds its scope, and stopped before writing.

Its own type because the three answers a cycle can end with are three different facts. clean says the work happened. A phase error says the work was attempted and broke. This says the work was abandoned -- another holder has the scope, is doing it properly, and everything this cycle already wrote is that holder's problem rather than a partial result anybody should read.

current_fence

current_fence() -> Fence | None

The fence this cycle is running under, or None outside one.

Source code in src/symfonic/capabilities/memory/fencing.py

def current_fence() -> Fence | None:
    """The fence this cycle is running under, or ``None`` outside one."""
    return _FENCE.get()

held async

held(fence: Fence | None) -> AsyncIterator[Fence | None]

Run a cycle under fence, and put back whatever was in force.

Source code in src/symfonic/capabilities/memory/fencing.py

@asynccontextmanager
async def held(fence: Fence | None) -> AsyncIterator[Fence | None]:
    """Run a cycle under ``fence``, and put back whatever was in force."""
    token = _FENCE.set(fence)
    try:
        yield fence
    finally:
        _FENCE.reset(token)