symfonic.evals.fixtures.model

model

Public values for a versioned, checksum-pinned text fixture.

A fixture owns both its corpus and the truth used to grade answers about it.

BookClaim dataclass

BookClaim(id: str, kind: ClaimKind, question: str, answer: str, sources: tuple[str, ...], must_contain: tuple[str, ...] = (), expected_current_assertions: Mapping[str, str] = dict(), forbidden_current_assertions: tuple[ForbiddenCurrentAssertion, ...] = (), derivation_inputs: tuple[str, ...] = ())

One answerable question and the sections that entail its answer.

current_assertion_violations

current_assertion_violations(assertions: Mapping[str, object]) -> tuple[ForbiddenCurrentAssertion, ...]

Return obsolete/wrong propositions asserted as current.

Text is not searched: naming a withdrawn value does not make it current.

Source code in src/symfonic/evals/fixtures/model.py

def current_assertion_violations(
    self, assertions: Mapping[str, object]
) -> tuple[ForbiddenCurrentAssertion, ...]:
    """Return obsolete/wrong propositions asserted as current.

    Text is not searched: naming a withdrawn value does not make it current.
    """
    normalized = {
        str(fact): str(value).strip().casefold() for fact, value in assertions.items()
    }
    return tuple(
        rule
        for rule in self.forbidden_current_assertions
        if normalized.get(rule.fact) == rule.value.strip().casefold()
    )

BookFixture dataclass

BookFixture(id: str, title: str, version: str, sections: tuple[BookSection, ...], claims: tuple[BookClaim, ...] = (), relations: tuple[BookRelation, ...] = (), negative_facts: tuple[NegativeFact, ...] = ())

A synthetic corpus plus the truth manifest that grades answers about it.

chapters property

chapters: tuple[BookSection, ...]

Only the narrative chapters, in order.

source_ids property

source_ids: tuple[str, ...]

Every section identifier, in manifest order.

claims_of_kind

claims_of_kind(kind: ClaimKind) -> tuple[BookClaim, ...]

Every expected claim exercising one retrieval competence.

Source code in src/symfonic/evals/fixtures/model.py

def claims_of_kind(self, kind: ClaimKind) -> tuple[BookClaim, ...]:
    """Every expected claim exercising one retrieval competence."""
    return tuple(row for row in self.claims if row.kind is kind)

mentions

mentions(term: str) -> tuple[str, ...]

Identifiers of every section stating term as a whole word.

Source code in src/symfonic/evals/fixtures/model.py

def mentions(self, term: str) -> tuple[str, ...]:
    """Identifiers of every section stating ``term`` as a whole word."""
    return tuple(row.id for row in self.sections if row.mentions(term))

section

section(section_id: str) -> BookSection

Look one section up by its stable identifier.

Source code in src/symfonic/evals/fixtures/model.py

def section(self, section_id: str) -> BookSection:
    """Look one section up by its stable identifier."""
    for row in self.sections:
        if row.id == section_id:
            return row
    raise KeyError(section_id)

BookRelation dataclass

BookRelation(id: str, subject: str, predicate: str, target: str, sources: tuple[str, ...])

A subject-predicate-target edge the corpus supports, with provenance.

BookSection dataclass

BookSection(id: str, ordinal: int, kind: SectionKind, title: str, filename: str, sha256: str, text: str, status: SectionStatus = SectionStatus.CURRENT, supersedes: tuple[str, ...] = (), superseded_by: str | None = None, signed_by: str | None = None)

One addressable source section and the checksum that pins its text.

mentions

mentions(term: str) -> bool

Whether this section states term as a whole word or phrase.

Source code in src/symfonic/evals/fixtures/model.py

def mentions(self, term: str) -> bool:
    """Whether this section states ``term`` as a whole word or phrase."""
    return _word_pattern(term).search(self.text) is not None

ClaimKind

Bases: StrEnum

The retrieval competence an expected claim is designed to exercise.

ForbiddenCurrentAssertion dataclass

ForbiddenCurrentAssertion(fact: str, value: str)

One normalized proposition that an answer must not assert as current.

This is not a text fragment: a correction may name the withdrawn value without recommending it.

SectionKind

Bases: StrEnum

What kind of source section a fixture file holds.

SectionStatus

Bases: StrEnum

Whether a section is still the operating authority for its subject.