Skip to content

symfonic.agent.triage.lexicon

lexicon

Verb lexicons for the deterministic intent classifier (v7.0 T06).

Kept small, conservative, and English-biased — see risk register in v7.0-plan.md § 5. DomainTemplate subclasses can extend by registering domain-specific verbs through tool_trigger_keywords (the overlap path). An intent_verbs dict on DomainTemplate is filed for v7.1 i18n.

v7.0.3 F3: TOKEN_RE upgraded to Unicode-aware so Spanish / German / French / Portuguese deployments no longer collapse to zero tokens. tokenize now NFC-normalises the input so precomposed ("á") and decomposed ("a" + U+0301) forms match consistently. Accents are preserved -- "gráfica" is deliberately not folded to "grafica". Accent-stripping changes meaning for some domains (Spanish sí / si) and is a UX decision, not a tokenizer decision; file for v7.1 behind an explicit opt-in.

tokenize 

tokenize(text: str) -> list[str]

Lowercase, word-like token split. Preserves ordering.

v7.0.3 F3: input is NFC-normalised before tokenisation so precomposed and decomposed variants ("á" vs "a" + U+0301) match consistently. Accents are preserved -- stripping is a UX decision, not a tokenizer decision (v7.1 if requested).

Source code in src/symfonic/agent/triage/lexicon.py 
65
66
67
68
69
70
71
72
73
74
def tokenize(text: str) -> list[str]:
    """Lowercase, word-like token split.  Preserves ordering.

    v7.0.3 F3: input is NFC-normalised before tokenisation so
    precomposed and decomposed variants (``"á"`` vs ``"a" + U+0301``)
    match consistently.  Accents are preserved -- stripping is a UX
    decision, not a tokenizer decision (v7.1 if requested).
    """
    normalised = unicodedata.normalize("NFC", text or "")
    return [m.group(0).lower() for m in TOKEN_RE.finditer(normalised)]