symfonic.services.observability.otel

otel

The OpenTelemetry adapter — built only when asked, imported only when built.

Everything about this module is arranged around one property: an adopter who never enables OTEL must not pay for it. Not an SDK import, not a bridge object, not a per-event branch. The shipped OTelExporter.build() already defers its opentelemetry imports; this module keeps that promise one level up by deferring the exporter module itself until otel_enabled is true.

The adapter is thin, but not a pass-through, because forwarding is not instrumentation. The shipped CallbackBridge is a set of thin forwarders into OTelTracer.recorder, and three of its methods are deliberately incapable of opening a span:

• on_agent_start / on_agent_end are explicit no-ops — the root span belongs to OTelTracer.start_run_span, a context manager the engine used to enter around a run.
• on_llm_end pops (run_id, node_name) from the recorder's open-span table and returns silently when nothing is there; only on_llm_start writes that table.
• on_node_error closes a node span the same tolerant way.

So an adapter that forwarded only the closing events would report success and export nothing at all. This module therefore drives the span lifecycle: it opens the root span on run start and closes it on run finish, and it emits the opening callback event that pairs with each closing one. The tests assert against a real tracer and an in-memory exporter, because a fake bridge that records method names cannot tell the two situations apart.

The second job is identity: the service's trace scope is published into the OTEL carrier so both halves agree on one run rather than minting two.

The third job is lifetime. The documented wiring recipe is per-run (compose_event_sink(scope, observers_from_config(config, ...), config=config)), so both of an observer's possible lifetimes have to be safe, and the obvious implementation gets each one wrong in a different way. Two rules answer that:

• The exporter is not run-scoped. Building one allocates a TracerProvider, a background BatchSpanProcessor worker and a gRPC channel, so doing it per run leaks a thread and a socket per run. It is cached and released by :mod:.otel_handles, which owns that question alone.
• Observer state is keyed by run. The root span and the carrier token live in :mod:.otel_spans, keyed on run_id, so one observer may be shared by any number of overlapping runs without run B's start overwriting run A's open span. Keying is not enough on its own: a reused key is displaced explicitly rather than dropped, and release is per-run (:meth:OTelObserver.release_run) so one abandoned run cannot end its neighbours' spans. aclose stays wholesale and means process shutdown.

Together those make both lifetimes safe: per-run construction leaks nothing, and a shared observer keeps each run's span identity and lifetime its own.

The bound on sharing. Span identity is keyed here; span nesting is not, and cannot be. OTelTracer.start_run_span nests children by attaching the OTEL current-span Context, which is a contextvars stack — the thing that makes symfonic.llm.call land under symfonic.run at all. A shared observer is therefore correct exactly as far as contextvars isolation reaches: each asyncio task gets its own copy of the context, so concurrent runs delivered from their own tasks — the topology the callback_fanout rendezvous policy produces, since it fans out inline in the run's own task — keep separate, correctly-parented trees. Two runs whose events are interleaved within a single task share one context stack, and their enters and exits unwind out of order: run B's root nests under run A's, and A's LLM span lands under B's root. Keying by run_id cannot fix that, because the corruption is in a stack this module does not own. test_otel.py pins both halves — the parentage that concurrent tasks must preserve, and the identity/lifetime that survives even the interleaved case.

OTelObserver

OTelObserver(handles: Any)

Drives the shipped OTEL span lifecycle from the kernel event stream.

Safe to build per run, and shareable across runs that are driven from their own asyncio task. What one run holds — its root span and its carrier token — belongs to :class:RunSpanTable, which owns the resource lifetime so this class can own the event lifetime: which kernel observation opens or closes which span.

Sharing is bounded by contextvars, not by that table: span nesting comes from the OTEL current-span context stack that start_run_span attaches, and only per-task context copies keep two runs' stacks apart. Runs interleaved inside one task get correct span identities and durations but cross-parented trees. See the module docstring.

Source code in src/symfonic/services/observability/otel.py

def __init__(self, handles: Any) -> None:
    self.handles = handles
    self._spans = RunSpanTable(handles)
    self._manager = CallbackManager([handles.callback_bridge])
    self._delegate = CallbackHandlerObserver(self._manager)

aclose async

aclose() -> None

Release every run this observer still holds. Never raises.

Process shutdown, not run cleanup: only runs whose terminal never arrived are left here — the stream guarantees one terminal per run, but a killed process does not. A caller that owns a single run wants :meth:release_run; calling this instead ends the spans of every other run in flight. It deliberately does not shut the exporter down: the handles are process-scoped and shared (see :func:build_otel_observer), so releasing them is :func:shutdown_otel's job, not one observer's.

Source code in src/symfonic/services/observability/otel.py

async def aclose(self) -> None:
    """Release *every* run this observer still holds. Never raises.

    Process shutdown, not run cleanup: only runs whose terminal never
    arrived are left here — the stream guarantees one terminal per run, but
    a killed process does not. A caller that owns a single run wants
    :meth:`release_run`; calling this instead ends the spans of every other
    run in flight. It deliberately does **not** shut the exporter down: the
    handles are process-scoped and shared (see :func:`build_otel_observer`),
    so releasing them is :func:`shutdown_otel`'s job, not one observer's.
    """
    self._spans.release_all()

on_cost async

on_cost(observation: CostObservation) -> None

Open the LLM span, then let the delegate's LLMEndEvent close it.

CallbackBridge.on_llm_end is a close: it pops the span keyed on (run_id, node_name) and returns when the key is absent. Without the paired start there is no span to attach the usage to, and the cost of the run reaches no exporter.

Source code in src/symfonic/services/observability/otel.py

async def on_cost(self, observation: CostObservation) -> None:
    """Open the LLM span, then let the delegate's ``LLMEndEvent`` close it.

    ``CallbackBridge.on_llm_end`` is a *close*: it pops the span keyed on
    ``(run_id, node_name)`` and returns when the key is absent. Without the
    paired start there is no span to attach the usage to, and the cost of
    the run reaches no exporter.
    """
    scope = observation.scope
    await self._manager.on_llm_start(
        LLMStartEvent(
            model=observation.model,
            messages=[],
            run_id=scope.run_id,
            system_prompt="",
            node_name=INVOCATION_NODE,
        )
    )
    await self._delegate.on_cost(observation)

on_run_failed async

on_run_failed(observation: RunFailed) -> None

Open the node span the delegate's NodeErrorEvent will close.

Same asymmetry as the LLM span: on_node_error calls end_node, which is tolerant of a missing start and therefore silent about it.

Source code in src/symfonic/services/observability/otel.py

async def on_run_failed(self, observation: RunFailed) -> None:
    """Open the node span the delegate's ``NodeErrorEvent`` will close.

    Same asymmetry as the LLM span: ``on_node_error`` calls ``end_node``,
    which is tolerant of a missing start and therefore silent about it.
    """
    await self._manager.on_node_start(
        NodeStartEvent(
            node_name=INVOCATION_NODE, run_id=observation.scope.run_id
        )
    )
    await self._delegate.on_run_failed(observation)

release_run async

release_run(run_id: str) -> None

Release one run's span and carrier. Never raises.

The call a run-scoped owner makes when its run ends without a terminal — see ObservabilityBridge.aclose. Scoped rather than wholesale because this observer may be shared by any number of overlapping runs, and one cancelled run must not end the others' spans.

Source code in src/symfonic/services/observability/otel.py

async def release_run(self, run_id: str) -> None:
    """Release one run's span and carrier. Never raises.

    The call a run-scoped owner makes when its run ends without a terminal
    — see ``ObservabilityBridge.aclose``. Scoped rather than wholesale
    because this observer may be shared by any number of overlapping runs,
    and one cancelled run must not end the others' spans.
    """
    self._spans.release(run_id)

build_otel_observer

build_otel_observer(config: Any, *, builder: Callable[[Any], Any] | None = None) -> OTelObserver | None

Return the OTEL observer, or None when OTEL is off or unavailable.

None is a first-class answer here, not a failure: compose_event_sink drops absent observers, so a disabled or un-installed OTEL simply removes itself from the wiring without any call site learning it exists.

Lifetime. The returned observer is cheap and may be built per run; the exporter behind it is not, and is therefore cached per configuration for the life of the process and released by :func:shutdown_otel. An explicit builder bypasses that cache entirely — the caller supplying the exporter owns its lifetime, which is what test doubles want.

Source code in src/symfonic/services/observability/otel.py

def build_otel_observer(
    config: Any,
    *,
    builder: Callable[[Any], Any] | None = None,
) -> OTelObserver | None:
    """Return the OTEL observer, or ``None`` when OTEL is off or unavailable.

    ``None`` is a first-class answer here, not a failure: ``compose_event_sink``
    drops absent observers, so a disabled or un-installed OTEL simply removes
    itself from the wiring without any call site learning it exists.

    **Lifetime.** The returned observer is cheap and may be built per run; the
    exporter behind it is not, and is therefore cached per configuration for
    the life of the process and released by :func:`shutdown_otel`. An explicit
    ``builder`` bypasses that cache entirely — the caller supplying the
    exporter owns its lifetime, which is what test doubles want.
    """
    if not getattr(config, "otel_enabled", False):
        return None
    handles = builder(config) if builder is not None else resolve_handles(config)
    if handles is None:
        logger.warning(
            "otel_enabled is set but the exporter could not be built; "
            "continuing without OpenTelemetry. Install the extra: "
            "pip install symfonic-core[otel]",
        )
        return None
    return OTelObserver(handles)