symfonic.agent.middleware.fabrication

fabrication

Fabrication-verdict executor (v7.0 T10, patched v7.0.3, v7.0.6).

v6.1.10 shipped TOOL DISCIPLINE at the prompt tier — telling the LLM not to invent task IDs, PIDs, or tool outputs. T10 adds the executor-side counterpart so a non-compliant LLM is still caught.

The detector scans the response draft for tool-call-shaped prose: phrases like "I ran", "I called", "task_id=", "PID=", UUID patterns, and any key listed in DomainTemplate.identifier_keys. For each candidate identifier it cross-references:

1. Tool results captured for this turn (LLMPreCallEvent / on_llm_end fan-outs).
2. Hydrated semantic node properties injected in the system prompt.
3. The user message verbatim.

Unresolved identifiers produce a FabricationFinding with a confidence score. Callers decide whether to downgrade, revise, or refuse based on FrameworkConfig.fabrication_check_mode.

Zero-cost contract: when the draft is empty, tool_calls_this_turn is empty, and no identifier_keys are configured, the detector returns an empty list after a single regex pre-filter.

v7.0.3 patches (F1, F2):

• F1 short-hex detector. The canonical-UUID-only pattern let 8-16-hex-char fakes like 9f2b1e47 slip through when the LLM decided a "prettier" identifier looked more plausible. A new _SHORT_HEX_PATTERN fires only in anchored contexts -- within ~60 chars of a claim verb match, or immediately after a known identifier key (task_id=, pid=, ...). Short-hex findings carry confidence=0.6 so the refuse-mode threshold can gate them while observe-mode still logs telemetry. Full-UUID findings keep confidence=0.95 -- well above the refuse threshold.
• F2 per-verb tool-call grounding. Previously a single tool call anywhere in the turn silenced every verb-claim finding. Now each verb-claim finding consults _VERB_TO_TOOL_HINTS and is dropped only when a tool whose name matches the verb's expected pattern actually fired. This catches the real-world case where hydrate_context ran but the response still claims "I spawned subagent X" without spawn_subagent having executed.

v7.0.6 patch — language-agnostic intent_action_no_tool finding:

• _CLAIM_VERB_PATTERN is intentionally English-only ("I ran", "I sent", ...). Spanish passive ("Gráfica enviada"), French ("J'ai envoyé"), German separable verbs, and Portuguese third-person all bypass it. Adding per-locale verb lexicons compounds technical debt — combinatorial growth, grammar coupling, drift from the actual concern.

v7.0.6 reuses the already-computed IntentVerdict (from v7.0 T07 + v7.0.3 F3 Unicode tokenizer + DomainTemplate. tool_trigger_keywords) instead. When the verdict says label="action" AND tool_calls_this_turn is empty, the LLM promised a side-effect and produced none, regardless of language. The classifier already does the i18n work — the detector just consumes the verdict. Zero extra latency: the same verdict that drove intent_filter / tool_routing this turn is threaded into the detector via apply_fabrication_check. When neither knob is enabled, intent_verdict is None and the new finding never fires (back-compat).

Confidence is 0.7 — below the default fabrication_refuse_min_confidence=0.8 so v7.0.6 ships in observe / revise friendly form. Operators dial the threshold to 0.65 to opt into refuse-mode draft replacement on the new finding kind.

FabricationFinding dataclass

FabricationFinding(
    identifier: str,
    kind: FabricationKind,
    severity: FabricationSeverity,
    reason: str,
    key: str | None = None,
    pattern_kind: str = "",
    confidence: float = 0.8,
    span: tuple[int, int] = (0, 0),
    grounded_sources: list[str] = list(),
)

A single flagged identifier in the response draft.

Attributes:

Name	Type	Description
identifier	str	The literal token the LLM claimed was real.
kind	FabricationKind	uuid / kv / verb_claim / short_hex_near_verb / short_hex_near_key — which detector fired.
severity	FabricationSeverity	Heuristic severity; callers gate on this via fabrication_threshold if desired.
reason	str	Short human-readable explanation.
key	str | None	When kind == "kv" or "short_hex_near_key", the normalised key portion.
pattern_kind	str	Stable lowercase string mirror of kind for telemetry payload consumers who want to pattern-match without being coupled to the Literal union.
confidence	float	0.0-1.0 detection confidence. v7.0.3 uses 0.95 for canonical UUIDs, 0.6 for context-anchored short-hex, and 0.8 for kv / verb_claim (the v7.0 priors). Callers may gate refusal on fabrication_refuse_min_confidence.
span	tuple[int, int]	(start, end) character offsets in the draft.
grounded_sources	list[str]	Sources consulted during cross-reference. A finding is only emitted when none of these grounded the identifier.

to_event_payload

to_event_payload() -> dict[str, Any]

Return a JSON-serialisable payload for telemetry events.

Source code in src/symfonic/agent/middleware/fabrication.py

def to_event_payload(self) -> dict[str, Any]:
    """Return a JSON-serialisable payload for telemetry events."""
    return {
        "identifier": self.identifier,
        "kind": self.kind,
        "severity": self.severity,
        "reason": self.reason,
        "key": self.key,
        "pattern_kind": self.pattern_kind or self.kind,
        "confidence": round(self.confidence, 3),
        "span": list(self.span),
    }

detect_fabrication

detect_fabrication(
    response_text: str,
    tool_calls_this_turn: list[dict[str, Any]]
    | None = None,
    *,
    hydrated_context: str = "",
    user_message: str = "",
    identifier_keys: list[str] | None = None,
    intent_verdict: IntentVerdict | None = None,
) -> list[FabricationFinding]

Scan response_text for unresolved identifiers.

Parameters:

Name	Type	Description	Default
response_text	str	The LLM draft to scrutinise.	required
tool_calls_this_turn	list[dict[str, Any]] | None	Serialised tool-call records; each dict is expected to expose a result / output / args field whose string form we scan for grounding. Tool name is read from the name / tool key when present so per-verb grounding (F2) can match against it.	None
hydrated_context	str	The assembled memory context string used in the system prompt this turn.	''
user_message	str	The user's current-turn query.	''
identifier_keys	list[str] | None	Domain-configured extra key lexicon. Empty list preserves behaviour — callers can opt in per-domain.	None
intent_verdict	IntentVerdict | None	v7.0.6 — the IntentClassifier verdict for the user query this turn (the same verdict consumed by the hydration-side intent filter and by tool routing). When label="action" with confidence ≥ 0.55 and tool_calls_this_turn is empty, the detector emits an intent_action_no_tool finding regardless of response language. None (the default) preserves v7.0.5 behaviour byte-for-byte for callers who have not enabled intent_filter_mode / tool_routing_mode.	None

Returns:

Type	Description
list[FabricationFinding]	List of FabricationFinding — one per unresolved identifier.
list[FabricationFinding]	Empty list for empty drafts. Never raises.

Source code in src/symfonic/agent/middleware/fabrication.py

def detect_fabrication(
    response_text: str,
    tool_calls_this_turn: list[dict[str, Any]] | None = None,
    *,
    hydrated_context: str = "",
    user_message: str = "",
    identifier_keys: list[str] | None = None,
    intent_verdict: IntentVerdict | None = None,
) -> list[FabricationFinding]:
    """Scan ``response_text`` for unresolved identifiers.

    Args:
        response_text: The LLM draft to scrutinise.
        tool_calls_this_turn: Serialised tool-call records; each dict
            is expected to expose a ``result`` / ``output`` / ``args``
            field whose string form we scan for grounding. Tool name
            is read from the ``name`` / ``tool`` key when present so
            per-verb grounding (F2) can match against it.
        hydrated_context: The assembled memory context string used in
            the system prompt this turn.
        user_message: The user's current-turn query.
        identifier_keys: Domain-configured extra key lexicon.  Empty
            list preserves behaviour — callers can opt in per-domain.
        intent_verdict: v7.0.6 — the IntentClassifier verdict for the
            *user query* this turn (the same verdict consumed by the
            hydration-side intent filter and by tool routing). When
            ``label="action"`` with confidence ≥ 0.55 and
            ``tool_calls_this_turn`` is empty, the detector emits an
            ``intent_action_no_tool`` finding regardless of response
            language. ``None`` (the default) preserves v7.0.5
            behaviour byte-for-byte for callers who have not enabled
            ``intent_filter_mode`` / ``tool_routing_mode``.

    Returns:
        List of ``FabricationFinding`` — one per unresolved identifier.
        Empty list for empty drafts.  Never raises.
    """
    text = (response_text or "").strip()
    if not text:
        return []

    tool_calls = tool_calls_this_turn or []

    grounding = _build_grounding_corpus(
        tool_calls,
        hydrated_context,
        user_message,
    )
    fired_tool_names = _collect_tool_names(tool_calls)

    all_keys = _DEFAULT_IDENTIFIER_KEYS | {
        k.lower() for k in (identifier_keys or [])
    }

    findings: list[FabricationFinding] = []

    # --- Verb claims (v7.0.3 F2: per-verb grounding) ------------------------
    verb_spans: list[tuple[int, int]] = []
    for verb_match in _CLAIM_VERB_PATTERN.finditer(text):
        action_word = (verb_match.group("action") or "").lower()
        verb_phrase = verb_match.group("verb").strip()
        verb_spans.append(verb_match.span())
        if _verb_claim_is_grounded(action_word, fired_tool_names):
            continue
        findings.append(
            FabricationFinding(
                identifier=verb_phrase,
                kind="verb_claim",
                severity="medium",
                reason=(
                    "Draft claims an action was taken but no matching "
                    "tool call fired this turn."
                ),
                pattern_kind="verb_claim",
                confidence=0.8,
                span=verb_match.span(),
                grounded_sources=[],
            )
        )

    # --- UUIDs (v7.0.3: now carry confidence=0.95) --------------------------
    uuid_spans: list[tuple[int, int]] = []
    for match in _UUID_PATTERN.finditer(text):
        uuid_spans.append(match.span())
        ident = match.group(0)
        if _is_grounded(ident, grounding):
            continue
        findings.append(
            FabricationFinding(
                identifier=ident,
                kind="uuid",
                severity="high",
                reason="UUID not found in tool results, hydration, or user query.",
                pattern_kind="uuid",
                confidence=0.95,
                span=match.span(),
                grounded_sources=[],
            )
        )

    # --- Short-hex (v7.0.3 F1: context-anchored only, confidence=0.6) -------
    for match in _SHORT_HEX_PATTERN.finditer(text):
        ident = match.group(0)
        start, end = match.span()

        # Skip hex spans already inside a full UUID match.
        if _span_overlaps(start, end, uuid_spans):
            continue

        # Skip pure-digit runs. Line numbers, counts, timestamps are
        # not identifier-shaped.  A valid fake-id needs at least one
        # a-f character to look "hashy"; without it we'd false-positive
        # on every number above 6 digits.
        if ident.isdigit():
            continue

        # Grounded in tool results / hydration / user message -> skip.
        if _is_grounded(ident, grounding):
            continue

        pattern_kind, key = _classify_short_hex_context(text, start, end, all_keys, verb_spans)
        if pattern_kind is None:
            continue

        findings.append(
            FabricationFinding(
                identifier=ident,
                kind=pattern_kind,  # "short_hex_near_verb" | "short_hex_near_key"
                severity="high",
                reason=(
                    "Short-hex identifier near a claim verb with no grounding"
                    if pattern_kind == "short_hex_near_verb"
                    else f"Short-hex identifier after '{key}' with no grounding"
                ),
                key=key,
                pattern_kind=pattern_kind,
                confidence=0.6,
                span=match.span(),
                grounded_sources=[],
            )
        )

    # --- Key=value shapes (v7.0.3: confidence=0.8) --------------------------
    for match in _KV_PATTERN.finditer(text):
        raw_key, value = match.group(1), match.group(2)
        normalised = raw_key.lower()
        if normalised not in all_keys:
            continue
        if _is_grounded(value, grounding):
            continue
        findings.append(
            FabricationFinding(
                identifier=value,
                kind="kv",
                severity="high" if normalised in _DEFAULT_IDENTIFIER_KEYS else "medium",
                reason=(
                    f"Identifier '{raw_key}={value}' not found in tool "
                    "results, hydration, or user query."
                ),
                key=normalised,
                pattern_kind="kv",
                confidence=0.8,
                span=match.span(),
                grounded_sources=[],
            )
        )

    # --- Intent-action-without-tool (v7.0.6, language-agnostic) -------------
    # Reuse the already-computed IntentVerdict so we can detect "action
    # claimed, zero tools fired" in any language the classifier
    # understands. The classifier consumes ``DomainTemplate.
    # tool_trigger_keywords`` + the v7.0.3 F3 Unicode tokenizer, so
    # Spanish / French / German / Portuguese queries that overlap the
    # domain's keyword set still classify as ``action`` even though
    # ``_CLAIM_VERB_PATTERN`` (above) is English-only by design.
    #
    # Zero-cost guarantee: when the caller does not pass an
    # ``intent_verdict`` (i.e. ``intent_filter_mode`` and
    # ``tool_routing_mode`` are both ``"off"``), this branch is a
    # single ``is None`` check and exits.
    if (
        intent_verdict is not None
        and intent_verdict.label == "action"
        and intent_verdict.confidence >= _INTENT_ACTION_MIN_CONFIDENCE
        and not tool_calls
    ):
        matched_kw = list(getattr(intent_verdict, "matched_tool_keywords", []) or [])
        confidence_str = f"{intent_verdict.confidence:.2f}"
        findings.append(
            FabricationFinding(
                identifier="",
                kind="intent_action_no_tool",
                severity="medium",
                reason=(
                    f"Intent verdict=action (confidence={confidence_str}) "
                    "with zero tool calls this turn. "
                    f"matched_tool_keywords={matched_kw}"
                ),
                pattern_kind="intent_no_grounding",
                confidence=0.7,
                span=(0, 0),
                grounded_sources=[],
            )
        )

    # De-dup: same identifier twice under different kinds collapses on the
    # (identifier, kind) tuple.
    unique: dict[tuple[str, str], FabricationFinding] = {}
    for f in findings:
        unique.setdefault((f.identifier, f.kind), f)
    return list(unique.values())