Skip to content

symfonic.capabilities.triage.lexicon

lexicon

Verb lexicons for the deterministic intent classifier.

Ported verbatim from :mod:symfonic.agent.triage.lexicon: a capability may import the kernel's contracts and its own package, so the lexicon lives here rather than being imported from the agent layer. Kept small, conservative, and English-biased; deployments extend coverage through the tool-keyword map (the overlap path).

TOKEN_RE is Unicode-aware so non-ASCII scripts yield tokens instead of being silently stripped. tokenize NFC-normalises its input so precomposed ("á") and decomposed ("a" + U+0301) forms match consistently. Accents are preserved -- "gráfica" deliberately does not match "grafica".

tokenize 

tokenize(text: str) -> list[str]

Lowercase, word-like token split. Preserves ordering.

Source code in src/symfonic/capabilities/triage/lexicon.py 
67
68
69
70
def tokenize(text: str) -> list[str]:
    """Lowercase, word-like token split. Preserves ordering."""
    normalised = unicodedata.normalize("NFC", text or "")
    return [m.group(0).lower() for m in TOKEN_RE.finditer(normalised)]