symfonic.observability.tool_span_seam

tool_span_seam

v7.23.3 — Tool-span emit-site wire-up.

External-adopter bug report (2026-06-10, Jarvio brain team via CORTEX observability platform): no symfonic.tool.<name> spans were appearing during agent execution despite observability/otel/_span_recorder.py shipping fully-built start_tool() / end_tool() methods.

Cross-file trace (architect verdict 2026-06-10) confirmed the gap:

• Recorder surface present (start_tool at _span_recorder.py:188, end_tool at _span_recorder.py:210) -- zero production callers.
• Engine dispatch via raw langgraph.prebuilt.ToolNode at core/presets.py:98-101 -- no observability wrap.
• ObservabilityHook.on_tool_start / on_tool_end at core/observability/protocol.py:53-62 -- also zero callers (second dead surface; closing both with the same wire-up).

This module is the v7.23.3 fix. It introduces:

1. A module-level ContextVar carrying the active SpanRecorder. The contextvar is OTel-agnostic (typed as Any) so this module can be imported from core/presets.py without pulling the optional opentelemetry dependency at framework-without-OTel install time.
2. set_active_recorder / reset_active_recorder helpers that the OTel tracer's start_run_span context manager calls to install / remove the recorder for the duration of a run.
3. InstrumentedToolNode(ToolNode) subclass that, on every ainvoke / invoke, inspects the contextvar. When a recorder is active, it iterates the inbound AIMessage's tool_calls, opens a span per call (correlated by tool_call_id), runs the wrapped tool, then closes each span with the tool result or an error attribution. When the contextvar is empty (OTel disabled / not configured), the wrapper falls through to super().ainvoke() with one ContextVar.get(None) check -- effectively zero overhead.
4. make_tool_node(tools) factory returning InstrumentedToolNode. The zero-cost invariant lives in the runtime contextvar check rather than a factory-time branch on otel_enabled (architect's verdict accepts both shapes; the runtime check is simpler -- one place to reason about, zero coupling to FrameworkConfig).

Sequencing constraint (architect verdict): the v7.19.0 on_tool_call_dispatch hook in core/nodes/react.py:1035-1116 mutates ai_message.tool_calls INSIDE the react node, BEFORE LangGraph hands off to ToolNode. So InstrumentedToolNode.ainvoke sees the already-rewritten calls. This is the right ordering -- tool spans reflect the executed name/args, not the original LLM output. Architecture composes correctly; no re-ordering needed.

Multi-call invariant: LangGraph's ToolNode handles multiple tool_calls per AIMessage as parallel branches. Each tool_call_id keys a distinct entry in the recorder's _tool_spans dict. Wrapper iterates and opens a span per call before dispatch, then closes each as results arrive (correlated by tool_call_id on the returned ToolMessages).

InstrumentedToolNode

InstrumentedToolNode(
    *args: Any,
    handle_tool_errors: Any = True,
    **kwargs: Any,
)

Bases: ToolNode

ToolNode subclass that opens a symfonic.tool.<name> span per invoked tool call, correlated by tool_call_id.

See module docstring for the architectural rationale.

Source code in src/symfonic/observability/tool_span_seam.py

def __init__(self, *args: Any, handle_tool_errors: Any = True, **kwargs: Any) -> None:
    # v9.0.0 (langgraph 1.x migration): langgraph flipped the ToolNode
    # ``handle_tool_errors`` default from ``True`` (catch EVERY tool
    # exception and feed it back to the model as an error ToolMessage) to
    # a callable that only handles ``ToolInvocationError`` and RE-RAISES
    # anything else -- which would crash the whole graph run on the first
    # tool that raises.  symfonic's react loop depends on a raised tool
    # error becoming a steering ToolMessage (the model sees the failure and
    # recovers), and the span/hook error path keys off that ToolMessage.
    # Restore the langgraph-0.x catch-all default; callers may still
    # override by passing ``handle_tool_errors=`` explicitly.
    super().__init__(*args, handle_tool_errors=handle_tool_errors, **kwargs)

get_active_recorder

get_active_recorder() -> Any | None

Return the active SpanRecorder if one is installed; else None.

Source code in src/symfonic/observability/tool_span_seam.py

def get_active_recorder() -> Any | None:
    """Return the active SpanRecorder if one is installed; else ``None``."""
    return _active_recorder.get(None)

make_tool_node

make_tool_node(tools: list[Any]) -> InstrumentedToolNode

Return a ToolNode that auto-instruments per-call spans when an OTel recorder is active for the current run; falls through to plain ToolNode semantics when no recorder is set.

Always returns InstrumentedToolNode. The zero-cost invariant lives in :meth:InstrumentedToolNode.ainvoke's contextvar check, not in this factory.

Source code in src/symfonic/observability/tool_span_seam.py

def make_tool_node(tools: list[Any]) -> InstrumentedToolNode:
    """Return a ToolNode that auto-instruments per-call spans when an
    OTel recorder is active for the current run; falls through to plain
    ToolNode semantics when no recorder is set.

    Always returns ``InstrumentedToolNode``.  The zero-cost invariant
    lives in :meth:`InstrumentedToolNode.ainvoke`'s contextvar check, not
    in this factory.
    """
    return InstrumentedToolNode(tools)

reset_active_recorder

reset_active_recorder(token: Token) -> None

Reset the contextvar to whatever value it held before the matching :func:set_active_recorder call.

Source code in src/symfonic/observability/tool_span_seam.py

def reset_active_recorder(token: Token) -> None:
    """Reset the contextvar to whatever value it held before the matching
    :func:`set_active_recorder` call."""
    _active_recorder.reset(token)

set_active_recorder

set_active_recorder(recorder: Any) -> Token

Install recorder as the active SpanRecorder for the current task.

Returns the ContextVar token; pass it to :func:reset_active_recorder when the run scope ends. The OTel tracer's start_run_span context manager owns the set/reset pair.

Source code in src/symfonic/observability/tool_span_seam.py

def set_active_recorder(recorder: Any) -> Token:
    """Install ``recorder`` as the active SpanRecorder for the current task.

    Returns the ContextVar token; pass it to :func:`reset_active_recorder`
    when the run scope ends.  The OTel tracer's ``start_run_span`` context
    manager owns the set/reset pair.
    """
    return _active_recorder.set(recorder)