Skip to content

symfonic.evals.fixtures.loader

loader

Load a committed book fixture and verify it against its truth manifest.

Loading always verifies: a fixture whose text has drifted from the checksum the manifest pins is not a weaker fixture, it is a different corpus, and every expected answer built on it is silently wrong.

Two properties make that verification mean what it says:

Bytes, not text. Checksums are taken over the bytes on disk and compared before anything is decoded. Reading text first would apply universal-newline translation, so rewriting a section from LF to CRLF would keep the recorded digest while changing every committed byte.

Containment, not trust. A manifest is data, and data does not get to name arbitrary files. Every path a manifest reaches for must be a single ordinary filename that still resolves inside the fixture directory once symlinks are followed.

FixtureChecksumError

Bases: FixtureError

A fixture file's bytes no longer match the checksum its manifest pins.

FixtureError

Bases: ValueError

A committed fixture is missing, malformed or no longer matches itself.

FixturePathError

Bases: FixtureError

A fixture identifier or manifest filename reaches outside its root.

checksum 

checksum(data: bytes) -> str

The sha256 a manifest must record for the exact bytes data.

Takes bytes rather than text on purpose: b"a\n" and b"a\r\n" are different corpora and must have different digests, but they decode to the same string under universal newlines.

Source code in src/symfonic/evals/fixtures/loader.py 
61
62
63
64
65
66
67
68
def checksum(data: bytes) -> str:
    """The sha256 a manifest must record for the exact bytes ``data``.

    Takes bytes rather than text on purpose: ``b"a\\n"`` and ``b"a\\r\\n"`` are
    different corpora and must have different digests, but they decode to the
    same string under universal newlines.
    """
    return hashlib.sha256(data).hexdigest()

load_book_fixture cached 

load_book_fixture(fixture_id: str = GLASS_HARBOR) -> BookFixture

Load a fixture shipped inside this package by its identifier.

The identifier selects a directory, so it is checked as a filename before it is used as one: an absolute or traversing id names something that is not a packaged fixture, whether or not it exists.

Source code in src/symfonic/evals/fixtures/loader.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
@cache
def load_book_fixture(fixture_id: str = GLASS_HARBOR) -> BookFixture:
    """Load a fixture shipped inside this package by its identifier.

    The identifier selects a directory, so it is checked as a filename before
    it is used as one: an absolute or traversing id names something that is not
    a packaged fixture, whether or not it exists.
    """
    _safe_component(fixture_id, "fixture id")
    directory = _contained_child(_FIXTURE_ROOT, fixture_id.replace("-", "_"), "fixture id")
    if not directory.is_dir():
        raise FixtureError(f"no packaged fixture named {fixture_id!r}")
    fixture = load_fixture_from(directory)
    if fixture.id != fixture_id:
        raise FixtureError(f"fixture directory {fixture_id!r} declares id {fixture.id!r}")
    return fixture

load_fixture_from 

load_fixture_from(directory: Path) -> BookFixture

Read and verify the fixture stored in directory.

directory is the trust boundary: nothing the manifest names may be read from outside it, and no failure reports the contents of a file it declined to accept.

Source code in src/symfonic/evals/fixtures/loader.py 
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
def load_fixture_from(directory: Path) -> BookFixture:
    """Read and verify the fixture stored in ``directory``.

    ``directory`` is the trust boundary: nothing the manifest names may be read
    from outside it, and no failure reports the contents of a file it declined
    to accept.
    """
    root = Path(directory).resolve()
    manifest_path = _contained_child(root, MANIFEST_FILENAME, "manifest filename")
    try:
        manifest = json.loads(manifest_path.read_bytes().decode("utf-8"))
    except FileNotFoundError as exc:
        raise FixtureError(f"no truth manifest at {MANIFEST_FILENAME}") from exc
    except IsADirectoryError as exc:
        raise FixtureError(f"{MANIFEST_FILENAME} is a directory") from exc
    except UnicodeDecodeError as exc:
        raise FixtureError(f"{MANIFEST_FILENAME} is not valid UTF-8") from exc
    except json.JSONDecodeError as exc:
        raise FixtureError(f"{MANIFEST_FILENAME} is not valid JSON") from exc
    if not isinstance(manifest, dict):
        raise FixtureError(f"{MANIFEST_FILENAME} must hold a JSON object")
    try:
        return BookFixture(
            id=str(manifest["id"]),
            title=str(manifest["title"]),
            version=str(manifest["version"]),
            sections=tuple(_section(row, root) for row in manifest.get("sections", ())),
            claims=tuple(_claim(row) for row in manifest.get("claims", ())),
            relations=tuple(_relation(row) for row in manifest.get("relations", ())),
            negative_facts=tuple(_negative(row) for row in manifest.get("negative_facts", ())),
        )
    except KeyError as exc:
        raise FixtureError(f"{MANIFEST_FILENAME} is missing key {exc}") from exc
    except ValueError as exc:
        if isinstance(exc, FixtureError):
            raise
        raise FixtureError(str(exc)) from exc