symfonic.capabilities.memory.heartbeat

heartbeat

Keeping a lease alive while the work it authorises is still going.

Without renewal a TTL has to answer two incompatible questions at once: how long a crashed worker may hold a scope hostage, and how long the slowest legitimate cycle takes. Deep Sleep runs seventeen phases and five of them can call a model; a quick nap has been measured at 28 seconds. Sizing the TTL for the second question makes a real crash cost minutes of blocked consolidation. Sizing it for the first makes a living worker lose its scope to its own model call -- and once that happens, expiry no longer means "this worker died", which is the only thing anybody wants it to mean.

A heartbeat separates them. The TTL stays short, so a crash is noticed quickly; a worker that is still running says so, repeatedly, and keeps what it holds.

It renews at a third of the TTL, so two consecutive renewals can be lost -- to a slow database, a scheduling hiccup, a paused event loop -- before the lease actually lapses. Renewing at half would leave one failure fatal, and the whole point is to distinguish trouble from death.

When a renewal comes back refused, the heartbeat stops. Refused means the scope is somebody else's now, and a heartbeat that kept trying would eventually succeed against a lease it does not own. Stopping is not a way of losing -- :class:~.fencing.FencedGraph is what makes the loss safe. The heartbeat's job is only to make it rare, and honest when it happens.

heartbeat async

heartbeat(leases: Any, lease: Any) -> AsyncIterator[None]

Renew lease for as long as the block runs.

Cancelled on the way out on every path -- success, failure, cancellation -- and awaited afterwards, so no renewal is still in flight when the caller releases the lease. A stray renew landing after the release would resurrect a lease nobody holds and lock the scope out for a full TTL.

A port without renew is accepted and simply not renewed. That keeps the contract additive for an adopter's own implementation; what such a deployment gets is the previous behaviour, where a cycle longer than the TTL loses its scope and is stopped by the fence.

Source code in src/symfonic/capabilities/memory/heartbeat.py

@asynccontextmanager
async def heartbeat(leases: Any, lease: Any) -> AsyncIterator[None]:
    """Renew ``lease`` for as long as the block runs.

    Cancelled on the way out on every path -- success, failure, cancellation --
    and awaited afterwards, so no renewal is still in flight when the caller
    releases the lease. A stray renew landing after the release would resurrect
    a lease nobody holds and lock the scope out for a full TTL.

    A port without ``renew`` is accepted and simply not renewed. That keeps the
    contract additive for an adopter's own implementation; what such a
    deployment gets is the previous behaviour, where a cycle longer than the
    TTL loses its scope and is stopped by the fence.
    """
    renew = getattr(leases, "renew", None)
    if renew is None:
        yield
        return
    task = asyncio.create_task(_beat(renew, lease))
    try:
        yield
    finally:
        task.cancel()
        # Suppressed rather than raised: the block's own outcome is the
        # caller's news, and a cancelled heartbeat is not an error at all.
        await asyncio.gather(task, return_exceptions=True)