Skip to content

symfonic.services.observability.otel_handles

otel_handles

Who owns the OpenTelemetry exporter, and for how long.

This module exists because the exporter's lifetime and the observer's lifetime are not the same lifetime, and conflating them is a resource leak.

OTelExporter.build is expensive and stateful: it creates a TracerProvider, a BatchSpanProcessor with its own background worker thread, and an OTLP gRPC channel. OTelTracer.shutdown is the only thing that flushes and releases them. The documented wiring recipe, meanwhile, is per-run — compose_event_sink(scope, observers_from_config(config, ...), config=config) is called once per invocation. Building the exporter on that path means one abandoned thread and one abandoned channel per run in any long-lived service, plus every span still buffered in the discarded processor dropped in silence.

So the exporter is cached per configuration for the life of the process, and :func:shutdown_otel (also registered with atexit) is what releases it. :mod:symfonic.services.observability.otel keeps the per-run half — the observer — and stays out of the lifetime question entirely.

resolve_handles 

resolve_handles(config: Any) -> Any

Return this configuration's exporter handles, building them at most once.

None propagates unchanged and is not cached: an un-installed SDK is a condition that a later call may legitimately find resolved, and caching the absence would pin the process to a decision made before the extra existed.

Source code in src/symfonic/services/observability/otel_handles.py 
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
def resolve_handles(config: Any) -> Any:
    """Return this configuration's exporter handles, building them at most once.

    ``None`` propagates unchanged and is not cached: an un-installed SDK is a
    condition that a later call may legitimately find resolved, and caching the
    absence would pin the process to a decision made before the extra existed.
    """
    global _ATEXIT_REGISTERED
    key = _config_key(config)
    if key is None:
        return _default_builder(config)
    with _LOCK:
        cached = _HANDLES.get(key)
        if cached is not None:
            return cached
        handles = _default_builder(config)
        if handles is None:
            return None
        _HANDLES[key] = handles
        if not _ATEXIT_REGISTERED:
            atexit.register(shutdown_otel)
            _ATEXIT_REGISTERED = True
        return handles

shutdown_otel 

shutdown_otel() -> None

Flush and release every cached exporter. Safe to call more than once.

Registered with atexit the first time handles are cached, because a BatchSpanProcessor holds spans that only shutdown flushes: a process that exits without it exports nothing from its final batch. Exposed publicly so a host that tears an application down deterministically — a test, a worker that reconfigures — need not wait for interpreter exit.

Source code in src/symfonic/services/observability/otel_handles.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def shutdown_otel() -> None:
    """Flush and release every cached exporter. Safe to call more than once.

    Registered with ``atexit`` the first time handles are cached, because a
    ``BatchSpanProcessor`` holds spans that only ``shutdown`` flushes: a process
    that exits without it exports nothing from its final batch. Exposed
    publicly so a host that tears an application down deterministically — a
    test, a worker that reconfigures — need not wait for interpreter exit.
    """
    with _LOCK:
        handles = tuple(_HANDLES.values())
        _HANDLES.clear()
    for handle in handles:
        shutdown = getattr(getattr(handle, "tracer", None), "shutdown", None)
        if shutdown is None:
            continue
        try:
            shutdown()
        except Exception:  # noqa: BLE001 - telemetry never breaks a shutdown
            logger.warning("OTel tracer shutdown raised; suppressing", exc_info=True)