symfonic.evals.memory_assertions

memory_assertions

Deterministic assertions over opt-in memory evidence.

Targets decide whether to expose the content-bearing attributes consumed here. The assertions never copy that content into their representation or verdicts.

MemoryCategorySeparated dataclass

MemoryCategorySeparated(expected: Mapping[str, str], *, attribute: str = 'memory_classifications', minimum_per_category: int = 1, allow_extra: bool = False)

Require memory records to retain their declared category and subject.

Evidence is a sequence of records with record_id, category and subject fields. Expected values are category-to-subject declarations. A record identity appearing in multiple categories is always a failure.

Source code in src/symfonic/evals/memory_assertions.py

def __init__(
    self,
    expected: Mapping[str, str],
    *,
    attribute: str = "memory_classifications",
    minimum_per_category: int = 1,
    allow_extra: bool = False,
) -> None:
    if not expected or any(not key or not value for key, value in expected.items()):
        raise ValueError("MemoryCategorySeparated requires category/subject pairs")
    if not attribute:
        raise ValueError("MemoryCategorySeparated requires an attribute name")
    if minimum_per_category < 1:
        raise ValueError("minimum_per_category must be positive")
    object.__setattr__(self, "expected", tuple(sorted(expected.items())))
    object.__setattr__(self, "attribute", attribute)
    object.__setattr__(self, "minimum_per_category", minimum_per_category)
    object.__setattr__(self, "allow_extra", allow_extra)
    object.__setattr__(self, "name", "memory_category_separated")

PromptRecallContains dataclass

PromptRecallContains(*fragments: str, attribute: str = 'prompt_recall', case_sensitive: bool = False)

Require fragments in the exact recall text supplied to the model.

The target opts in by placing the delimited recall contribution in attributes[attribute]. This deliberately checks model input rather than store contents or the answer, either of which can produce a false positive while prompt injection is broken.

Source code in src/symfonic/evals/memory_assertions.py

def __init__(
    self,
    *fragments: str,
    attribute: str = "prompt_recall",
    case_sensitive: bool = False,
) -> None:
    if not fragments or any(not fragment for fragment in fragments):
        raise ValueError("PromptRecallContains requires non-empty fragments")
    if not attribute:
        raise ValueError("PromptRecallContains requires an attribute name")
    object.__setattr__(self, "fragments", tuple(fragments))
    object.__setattr__(self, "attribute", attribute)
    object.__setattr__(self, "case_sensitive", case_sensitive)
    object.__setattr__(self, "name", "prompt_recall_contains")

PromptRecallExcludes dataclass

PromptRecallExcludes(*fragments: str, attribute: str = 'prompt_recall', case_sensitive: bool = False)

Require content not to reach the model, reporting only its digest.

Source code in src/symfonic/evals/memory_assertions.py

def __init__(
    self,
    *fragments: str,
    attribute: str = "prompt_recall",
    case_sensitive: bool = False,
) -> None:
    if not fragments or any(not fragment for fragment in fragments):
        raise ValueError("PromptRecallExcludes requires non-empty fragments")
    object.__setattr__(self, "fragments", tuple(fragments))
    object.__setattr__(self, "attribute", attribute)
    object.__setattr__(self, "case_sensitive", case_sensitive)
    object.__setattr__(self, "name", "prompt_recall_excludes")

ResponseDoesNotContradictRecall dataclass

ResponseDoesNotContradictRecall(recall_attribute: str = 'recall_claims', response_attribute: str = 'response_claims', minimum: int = 1, name: str = 'response_does_not_contradict_recall')

Compare target-normalized response claims with recalled claims.

Both attributes are mappings from a stable claim id to a normalized value. Only claim ids present in both mappings are compared. minimum prevents an empty response-claim mapping from passing vacuously.