State-Scope Mismatch — A Recurring Bug Pattern¶
Status: internal engineering note Audience: PR reviewers, contributors touching long-lived stateful surfaces Origin: two hotfixes shipped 2026-05-25 (v7.3.1, v7.3.2) traced to the same root pattern
TL;DR¶
A state-scope mismatch bug occurs when state is initialized at one scope (process / module / instance) but the consuming code makes a decision at a finer scope (per-call / per-connection / per-tenant). Defaults look right, local tests pass, and production blows up — usually only after a particular race or initialization order is hit.
If you find yourself writing _GLOBAL_FLAG = True or if some_module._FLAG:, stop and ask whether the consumer actually wants per-X state. The cost of being wrong is paid in degraded retrieval quality, silent crashes on customer boot, or both.
Case study 1 — v7.3.1 python-jose boot crash¶
The bug¶
# src/symfonic/agent/middleware/pause_token.py (pre-fix)
from jose import JWTError, jwt # module-level eager import
SymfonicAgent.__init__ at engine.py:1150 imported this module unconditionally. The ask_user_enabled=False gate three lines below was bypassed. Result: every consumer not on [ask-user] or [testing] crashed on boot.
The scope mismatch¶
| Layer | Actual scope | Code-as-written scope |
|---|---|---|
python-jose dependency |
Per-feature (only ask_user_enabled=True needs it) |
Module load (always required) |
The fix shape¶
Soft probe at module load (jwt = None on missing) + _require_jose() helper called inside each crypto method:
try:
from jose import JWTError, jwt
except ImportError:
JWTError = None
jwt = None
def _require_jose() -> None:
if jwt is None:
raise ImportError("python-jose is required — install [ask-user]")
How it slipped past CI¶
pyproject.toml [testing] extra installs python-jose. Every CI install had jose available. No base-install smoke test existed.
The regression test that now catches this class of bug:
# tests/release/test_no_jose_at_boot.py
def block_jose(monkeypatch):
for name in list(sys.modules):
if name == "jose" or name.startswith("jose."):
del sys.modules[name]
real_import = builtins.__import__
def _blocked(name, *args, **kw):
if name == "jose" or name.startswith("jose."):
raise ImportError(f"jose blocked for test")
return real_import(name, *args, **kw)
monkeypatch.setattr(builtins, "__import__", _blocked)
Case study 2 — v7.3.2 pgvector codec latch¶
The bug¶
# src/symfonic/memory/backends/pool.py (pre-fix)
_PGVECTOR_REGISTRATION_FAILED: bool = False # process-wide
async def _register_pgvector_on_connection(conn):
try:
await _pgvector_register_vector(conn) # per-connection
except Exception:
_PGVECTOR_REGISTRATION_FAILED = True # process-wide
# src/symfonic/memory/backends/postgres_vector.py (pre-fix)
def _encode_vector_param(emb):
if _pool_mod._PGVECTOR_REGISTRATION_FAILED:
return str(flat) # text form
return flat # list form
The scope mismatch¶
| Layer | Actual scope | Code-as-written scope |
|---|---|---|
pgvector codec registration |
Per physical connection | Process-wide via latch |
The CI repro that local-dev missed¶
- Fresh database has no
vectortype untilensure_schema()runsCREATE EXTENSION - Pool with
min_size>=1opens connection #1 beforeensure_schema()—register_vectorraisesunknown type: public.vector, latch flips True ensure_schema()creates the extension- Pool needs connection #2 —
register_vector(conn2)now succeeds, conn2 has the codec _encode_vector_paramreads the (stale) latch → returns text form → sent to a codec-registered conn2 → asyncpg'sVector._to_db_binary('[1.0,...]')→np.asarray(string, dtype=>f4)→ crash
Local dev pre-creates the extension (often as a sidecar setup step), so connection #1 succeeds and the latch never flips. CI provisions a fresh DB per run.
The fix shape¶
Per-connection registry, NOT process-wide latch:
import weakref
_CODEC_REGISTRY: weakref.WeakKeyDictionary[Connection, bool] = weakref.WeakKeyDictionary()
async def _register_pgvector_on_connection(conn):
try:
await _pgvector_register_vector(conn)
_CODEC_REGISTRY[conn] = True
except Exception:
_CODEC_REGISTRY[conn] = False
def connection_has_pgvector_codec(conn) -> bool:
raw = getattr(conn, "_con", conn) # PoolConnectionProxy._con
return _CODEC_REGISTRY.get(raw, False)
Encoders take an optional conn parameter. The process-wide latch survives ONLY to dedupe warning logs.
The contract-pinning test¶
TestPerConnectionEncoderContract (4 tests) asserts:
1. Encoder follows per-connection codec when conn is passed
2. Encoder falls back to latch when conn=None (legacy callers)
3. PoolConnectionProxy._con is resolved before registry lookup
4. Init hook marks codec state per connection
Case study 3 — Defensive .get() masking a state-shape contract miss¶
The bug¶
An adopter's plugin had been reading the wrong state surface for the entire lifetime of the plugin:
# The adopter's plugin (paraphrased)
def inject_system_prompt(self, state):
tenant_id = state.get("tenant_id", "unknown") # lowercase
# ... use tenant_id ...
state.get("tenant_id") returned None on every invocation. The plugin still ran because .get(default="unknown") silently substituted the default. The adopter's downstream code worked off the default value and nobody noticed for the lifetime of the plugin.
Then we shipped the QUERY-contract pin (case study 0 / commit f173bb8), advertised TENANT_ID (uppercase) as the stable contract, and the adopter asked: "wait, will lowercase still be populated in 7.3.5?" The audit revealed it had never been populated — they were reading from a different state surface than they thought.
The scope mismatch¶
| Layer | Actual scope | Code-as-written scope |
|---|---|---|
| Runtime LangGraph state | lowercase keys (tenant_id, query), built by core/runtime.py:_build_input_state |
Plugin author assumed this is what inject_system_prompt(state) receives |
| Plugin-render state | UPPER_SNAKE keys (TENANT_ID, QUERY), built by agent/engine.py:_build_*_prompt_for_strategy |
The actual surface plugins see — fully isolated from runtime state |
Two state surfaces with overlapping conceptual content, completely different casing, zero normalisation between them. Defensive .get(default) patterns mean the wrong choice is silent.
The fix shape¶
There wasn't one. The adopter's plugin needed a one-line change on their side (state["tenant_id"] → state.get("TENANT_ID")). The framework's contract was correct; the plugin's read was wrong; defensive .get() hid it.
The deeper fix is a pattern shift in how plugins consume state: stop using .get(key, default) for keys that the contract guarantees. Use direct indexing state[key] for stable keys (KeyError is the friend that catches contract violations early) and reserve .get(key, default) for explicitly optional keys.
How it slipped past everything¶
- The plugin guide documents uppercase, but the lowercase typo wasn't a
KeyError— it was a silentNonemasked by the default. - The regression test pinned uppercase (now
tests/agent/test_plugin_state_query_contract.py), but the adopter's plugin wasn't in our test suite. - The reviewer (the adopter's own engineer) never logged the keys mid-development to confirm what the state actually contained — they wrote what they expected to see.
- A
logger.warning("plugin state keys: %s", sorted(state.keys()))insideinject_system_promptwould have shown UPPER_SNAKE on day one.
Generalisation — .get() is a code-review smell¶
When you see state.get(KEY, default) in a plugin/handler/middleware, ask:
- Is
KEYpart of a documented stable contract? If yes, the.get()is wrong — it masks contract violations. Use direct indexing. - Is
defaulta "reasonable fallback" or a "this should never fire" sentinel? If the latter, anassert key in stateis more honest. - Is the author SURE they're reading the right state surface? If two surfaces exist with overlapping names, dual-key normalisation OR direct indexing forces a crash on the wrong surface — both are better than silent wrong-default.
The contract-pinning test that prevents this¶
tests/agent/test_plugin_state_query_contract.py (shipped with case study 0) pins the stable-key set via direct-indexing assertions:
_STABLE_KEYS = ("TENANT_ID", "NAMESPACE", "DOMAIN_NAME", "QUERY")
def test_jit_path_stable_key_set(captured_state):
missing = [k for k in _STABLE_KEYS if k not in captured_state]
assert not missing, f"JIT state missing stable keys: {missing}"
This test fires red if a refactor silently drops any of the four. Note it asserts presence via in not via .get(...) is not None — same reason.
Case study 4 — Type contracts that aren't enforced at the boundary¶
The bug¶
Two adopter findings on v7.4.0, same root pattern at different layers:
4a — thinking TypeError (engine.py:687, latent since at least 7.3.6):
# core/state.py:28 — type declaration
final_response: str | None
# core/nodes/react.py:327 — assignment
state["final_response"] = ai_message.content # str when content is text;
# list[ContentBlock] when thinking enabled
# engine.py:1985-1986 — consumer
raw_response = result.get("final_response") or "" # `or ""` does NOT catch truthy list
final_response, extracted_ops = _parse_and_strip_extraction(raw_response)
# -> _EXTRACT_RE.search(text)
# -> TypeError: expected string or bytes-like object, got 'list'
4b — silent-drop sibling (engine.py:619, same root):
def _find_final_response(data):
final = data.get("final_response")
if isinstance(final, str): # list content silently rejected
return final
return "" # streaming consumers receive empty string instead of the LLM's reply
Streaming-enabled tenants with extended thinking on were writing "" to working memory for every turn since the day the streaming path was wired. No crash, no log line, just empty content.
The scope mismatch¶
| Layer | Declared scope | Actual scope |
|---|---|---|
final_response: str \| None |
Type annotation says string-or-none | Assigned list[ContentBlock] when thinking is enabled |
_find_final_response returns |
Documented as the LLM reply | Returns "" when the assignment violates the type contract |
_parse_and_strip_extraction accepts |
Expects str per the function signature implicit in re.search(text) |
Receives list from a caller that trusted the type annotation |
The type annotation IS the contract. It's also a lie. Python's gradual typing means no boundary check fires when the assignment violates it — the violation propagates downstream and either crashes (4a) or silently drops content (4b) depending on the consumer's defensive style.
The fix shape¶
_flatten_content_blocks helper at the consumer boundary:
def _flatten_content_blocks(value):
"""Idempotent on str; concatenates text from text-type blocks; drops thinking."""
if isinstance(value, str):
return value
return "".join(b.get("text", "") for b in value if b.get("type") == "text")
def _parse_and_strip_extraction(text):
text = _flatten_content_blocks(text) # normalize at entry
match = _EXTRACT_RE.search(text)
...
Two-line guard at the consumer. Auto-fixes every downstream call site because the function entry now enforces the type the rest of the function assumed.
The deeper smell¶
The fix at the consumer is the minimal fix, not the root fix. The root fix would be one of:
- Enforce the type at assignment —
core/nodes/react.py:327doesstate["final_response"] = ai_message.content if isinstance(ai_message.content, str) else _flatten_content_blocks(ai_message.content). The producer pays the cost; consumers stay simple. - Widen the declared type —
final_response: str | list[ContentBlock] | None. Honest about reality; every consumer now KNOWS it must handle both shapes. Worse ergonomics but no hidden bug. - Use Pydantic at the boundary —
AgentState(BaseModel)with a validator that flattens lists on the way in. Pydantic enforces the type contract Python's annotations don't.
Each is "more correct" than the consumer-side guard but ships later because they ripple through every assignment site. The consumer-side guard is the right hotfix; the producer-side or schema-side enforcement is the right 7.5 redesign.
How it slipped past every test¶
- Provider mocks in the test suite return
final_responseas astrbecause that's how the type annotation reads. The real Anthropic provider returns alist[ContentBlock]when thinking is enabled. The fidelity gap is the bug. - Tests for
_find_final_responsedon't exercise the list path — there was no list-path test to write because nobody believed lists could arrive (the type said so). mypy --strictdoesn't catch it either: the assignment inreact.py:327is from alangchain_core.AIMessage.contentfield that's typed asstr | list[dict]. The Python-side narrowing happens via runtime behaviour, not at the assignment site.
Generalisation — distrust the declared type¶
When you read a type annotation field: T, ask:
- Where is the producer? Does the producer KNOW the annotation? Are there branches where the producer could write a different shape?
- Where is the consumer? Does the consumer do
isinstance(value, T)or trust the annotation? Defensive consumers crash loudly on violations; trusting consumers crash subtly (4a) or silently lose content (4b). - Is there a boundary enforcement layer? Pydantic? A validator? An assertion? Or just the implicit promise of the annotation?
Type annotations without boundary enforcement are documentation, not contracts. If you can't run assert isinstance(value, T) at the boundary and have it pass, the annotation is aspirational.
The contract-pinning test¶
tests/agent/test_thinking_typeerror_regression.py pins the new boundary:
def test_parser_handles_list_input():
"""_parse_and_strip_extraction must accept list-shaped final_response."""
blocks = [{"type": "thinking", "thinking": "reasoning"}, {"type": "text", "text": "Hello"}]
response, ops = _parse_and_strip_extraction(blocks)
assert response == "Hello"
def test_find_final_response_flattens_list():
"""_find_final_response must extract text from list-shaped content."""
state = {"final_response": [{"type": "thinking", "thinking": "x"}, {"type": "text", "text": "Hi"}]}
assert _find_final_response(state) == "Hi"
Note these tests assert what the consumer does with the list, not the type-system shape. The consumer is the boundary; the boundary is what's enforced. Type-checker compliance is necessary but not sufficient.
Case study 5 — Unvalidated handoff: silent attribute typo through direct mutation¶
The bug¶
An adopter's consumer needed to late-bind episodic + procedural layers onto an already-constructed SleepConsolidator. The supported path looks like:
agent = SymfonicAgent(model_provider=..., config=...)
consolidator = SleepConsolidator(...) # built earlier
consolidator.bind_layers( # post-v7.4.6
episodic_layer=agent._orchestrator.get_layer("episodic"),
procedural_layer=agent._orchestrator.get_layer("procedural"),
)
But pre-v7.4.6 there was no bind_layers method. The adopter reached for direct attribute assignment, and made one wrong guess:
# The adopter's code (pre-fix consumer)
consolidator._episodic_layer = agent._orchestrator.get_layer("episodic")
consolidator._procedural_layer = agent._orchestrator.get_layer("procedural")
The actual storage attributes are _episodic and _procedural (no _layer suffix — verified at consolidation.py:140-141). The constructor's KWARG is episodic_layer=..., but it writes to self._episodic. The asymmetry between kwarg and storage attribute is the trap.
Python silently created phantom attributes _episodic_layer and _procedural_layer on the consolidator instance. The consolidator's internal _episodic and _procedural stayed None. Every Phase 12 promotion saw "no episodic data" and skipped silently. The adopter's agent's procedural memory never grew.
The scope mismatch¶
| Layer | Declared/documented scope | Actual scope |
|---|---|---|
| Constructor kwarg | episodic_layer |
Writes to self._episodic |
| Storage attribute | (private, undocumented) | _episodic (no _layer suffix) |
| Consumer's mental model | kwarg name == attribute name |
Asymmetric — Python convention isn't enforced |
The class is a plain Python class with no __slots__, no @dataclass, no Pydantic. Setting an arbitrary attribute on an instance succeeds silently — that's Python's data model, and it's intentional to support monkey-patching. The asymmetry is symfonic-core's design choice; Python's permissiveness is what makes the bug invisible.
The fix shape¶
SleepConsolidator.bind_layers(*, episodic_layer=None, procedural_layer=None, overwrite=False) -> None shipped in v7.4.6 (commit bf55f34). Three properties make it typo-safe:
- Keyword-only signature (
*, episodic_layer=None, ...). Python raisesTypeError: bind_layers() got an unexpected keyword argument '_episodic_layer'at the call site when an unknown kwarg is passed. The exact typo the adopter hit is now caught at the API surface. - Single source of truth. The method writes to
self._episodicandself._procedural. Callers don't need to know the asymmetric storage names — they pass the kwarg, the method handles the mapping. overwrite=Falsedefault. Rebinding an already-bound layer raisesRuntimeErrorinstead of silently replacing. Accidental double-wire is caught.
What was NOT the fix¶
We did NOT add __slots__ to the consolidator. Three reasons:
- The class's attribute set grows organically (each new phase adds fields).
__slots__would force a coordinated update on every additive change. - External subclasses (test fixtures, downstream extensions) would break if they add attributes.
- The bind_layers method is sufficient for the documented late-bind pattern;
__slots__solves the same problem at a higher cost.
We also did NOT have SymfonicAgent own SleepConsolidator construction internally (the adopter's Option B). The router's request-scoped consolidator pattern + 4 example call sites depend on the existing kwarg path — internalizing construction would have been a breaking change for a problem that doesn't exist (the kwarg path works; 5 in-repo examples prove it).
Honest framing¶
The adopter diagnosed the bug as "chicken-and-egg forces post-hoc patching." The audit found that framing self-justifying:
- Exhaustive grep across
src/,tests/,examples/returned ZERO sites where any consumer post-hoc patchesconsolidator._episodic = .... The 5 in-repo examples all use the documented kwarg path correctly. - The kwarg path itself works: build agent → read layers via
agent._orchestrator.get_layer(...)→ build consolidator with kwargs → attach viaagent._sleep_consolidator = ...(patching the AGENT, which IS a documented hook). - The adopter's bug was their consumer bypassing the supported API, not being forced into a bad one.
bind_layers is a typo-safety upgrade for the late-bind pattern (when the consolidator is built BEFORE the agent's layers are available — a valid use case), NOT a redesign of a broken constructor surface.
The contract-pinning test¶
def test_unknown_kwarg_with_layer_suffix_raises_typeerror(self) -> None:
"""The exact typo the adopter hit (_episodic_layer vs _episodic) is caught."""
consolidator = _make_consolidator()
with pytest.raises(TypeError, match="unexpected keyword argument"):
consolidator.bind_layers(_episodic_layer=MagicMock())
The load-bearing test. Asserts that the bad attribute name fires TypeError at the call site instead of silently creating a phantom attribute. Python's keyword-only enforcement is the boundary check; the test pins that the boundary IS there.
Generalisation — direct attribute mutation across a class boundary is suspect¶
Whenever you see instance.private_attr = value from outside the class, ask:
- What's the supported path? Constructor kwarg? Setter method? Public property?
- Does the supported path exist? If yes, why did the consumer bypass it?
- If the supported path doesn't exist, why not? Is the late-bind pattern legitimate (chicken-and-egg) or is the consumer doing something the class wasn't designed for?
- What happens on typo? Plain class → silently creates phantom attribute.
__slots__→ AttributeError.@dataclass(frozen=True)→ FrozenInstanceError. Pydanticextra="forbid"→ ValidationError.
If the boundary check doesn't exist AND the supported path doesn't exist, add a validated setter (Option A) before reaching for __slots__ (Option C). The setter is additive, low-cost, and documents the intent — __slots__ is a strong defensive line but creates ripple effects whenever the attribute set grows.
Pattern recognition checklist¶
When reviewing a PR that touches long-lived state, ask:
- [ ] What scope is this state actually about? Process, module, class, instance, request, connection, transaction, turn, tenant?
- [ ] Does the code initialize state at the matching scope? Module-level flag for per-connection state is a smell.
- [ ] What's the worst case if the state is wrong at one scope but right at another? Silent degradation? Crash? Data corruption?
- [ ] Does the test exercise the scope you care about? A unit test mocking a module-level flag passes regardless of whether the production scope is module-level.
- [ ] What initialization order does the production path actually use? Pool init runs before extension creation? Module import runs before feature flag is read? Connect-first vs schema-first?
Things to grep for during review¶
# Module-level mutable flags
grep -rE "^_[A-Z_]+: (bool|int) = " src/
# Module-level eager imports of optional dependencies
grep -rE "^from (jose|pgvector|spacy|psycopg|asyncpg)" src/
# Process-wide state that's mutated in init callbacks
grep -rE "global _[A-Z_]+" src/
# Encoder/decoder/codec patterns that consume external state
grep -rE "def _(encode|decode|format)_.*if .*_FLAG" src/
Each match is a candidate for the scope-mismatch question.
Refactor recipes¶
Recipe 1 — Module-level eager import → soft probe + lazy guard¶
# Before:
from optional_dep import Something # crashes if not installed
# After:
try:
from optional_dep import Something
except ImportError:
Something = None
def _require_optional_dep() -> None:
if Something is None:
raise ImportError("optional_dep is required — install [extras]")
# Then call _require_optional_dep() at the top of each method that uses Something.
Recipe 2 — Process-wide latch → per-X registry¶
# Before:
_GLOBAL_FAILED: bool = False
def _consume() -> Any:
if _GLOBAL_FAILED:
return fallback()
return optimal()
# After:
import weakref
_REGISTRY: weakref.WeakKeyDictionary[ObjectType, bool] = weakref.WeakKeyDictionary()
def _record(obj, ok: bool) -> None:
_REGISTRY[obj] = ok
def _consume(obj=None) -> Any:
if obj is None:
return fallback() # legacy / mock callers
if not _REGISTRY.get(obj, False):
return fallback()
return optimal()
WeakKeyDictionary is critical when obj is something asyncpg-style with __slots__ (no setattr allowed) AND when the key is expected to be reclaimable by GC.
Recipe 3 — Test that the FINER scope works¶
# Bad: tests pass with both scopes wrong because the test only exercises one
def test_consume_returns_fallback_when_flag_true():
_GLOBAL_FAILED = True
assert _consume() == fallback_value
# Good: tests pin the per-X contract
def test_consume_follows_per_object_state_when_obj_passed():
obj_a, obj_b = make_obj(), make_obj()
_record(obj_a, True); _record(obj_b, False)
assert _consume(obj_a) == optimal_value
assert _consume(obj_b) == fallback_value
def test_consume_falls_back_to_global_when_obj_is_None():
# legacy contract
...
Why this pattern keeps recurring¶
- Module-level flags feel cheap. Adding
_GLOBAL_FLAG: bool = Falseis one line. Threading a parameter through a 5-call-deep stack is annoying. - Per-X state is invisible in shallow stack traces. A latch flip in
__init__propagates to a call 50 functions away with no audit trail. - Local-dev environments mask race conditions. Pre-created extensions, pre-installed extras, warm caches — all hide the cold-start race that CI hits first.
- Unit tests of degraded mode are easy. Tests of the transition (right scope, wrong moment) are hard and usually skipped.
Mandatory mitigations going forward¶
- Every optional dependency in
pyproject.tomlmust have a base-install smoke test intests/release/that imports the module undersys.moduleseviction. - Every cross-scope state read (encoder consults a flag mutated by an init callback) must have a test that exercises the transition where the scopes diverge.
- Code review must ask the scope question for any mutable module-level state.
Cross-references¶
tests/release/test_no_jose_at_boot.py— case study 1 regression testtests/memory/unit/test_pool_pgvector_codec.py::TestPerConnectionEncoderContract— case study 2 regression testtests/agent/test_plugin_state_query_contract.py— case study 3 regression testtests/agent/test_thinking_typeerror_regression.py— case study 4 regression testtests/unit/learning/test_sleep_consolidator_bind_layers.py— case study 5 regression test.claude/docs/2026-05-25-jose-import-regression-eval.md— v7.3.1 investigation- Commit
96d7d51— v7.3.1 jose hotfix - Commit
4b1b77f— v7.3.2 pgvector hotfix - Commit
f173bb8— v7.3.5 QUERY-in-plugin-state contract pin - Commit
bf55f34— v7.4.6 SleepConsolidator.bind_layers setter (case study 5) - v7.4.1 — thinking TypeError hotfix (case study 4)
Last updated: 2026-05-27