Agent regression evaluations

An evaluation (eval) is a repeatable conversation with checks attached. For example: tell the agent your preferred bread, open a different conversation, and ask what bread you prefer. The test checks both the answer and whether the agent actually retrieved the saved memory. A correct answer alone could be a lucky guess.

This guide is for developers who own a generated Symfonic application. Start with the offline walkthrough; you do not need to understand the kernel or write an evidence adapter. The reference sections later explain those extension points.

Unreleased preview

Use the integration-candidate wheel supplied for testing this documentation. Installing the released 9.12.0 package from PyPI is not equivalent, even though the candidate currently has the same package version. Keep the wheel or commit identifier with your results. Existing generated applications do not acquire new eval files just by upgrading their library dependency.

1. Run your first eval without a model

You need Python 3.11 or newer, a terminal, and the candidate wheel. These commands use a POSIX shell (macOS/Linux). Dependency installation needs internet access; the fast eval itself needs no model credentials, database or running web app.

Create a disposable application

Run this in an empty working directory. Replace the wheel path with the actual file you downloaded or built:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "/absolute/path/to/symfonic_core-9.12.0-py3-none-any.whl[init,openai]"
symfonic init eval-demo --llm-provider openai
cd eval-demo
python -m pip install -r requirements.txt

The provider selection prepares the app for a later live run. It does not make the offline profiles call OpenAI. Keep the same virtual environment active. For an existing candidate-generated project, activate its environment, install its requirements, and change into its root instead. That directory must contain both app/ and evals/.

Run and save a report

PYTHONPATH=. symfonic eval evals.suite:SUITE --profile fast \
  --json eval-fast.json --junit eval-fast.xml

evals.suite:SUITE means “load the object named SUITE from evals/suite.py.” PYTHONPATH=. makes the generated application's modules importable by the CLI; run the command from the project root, not inside evals/.

The terminal prints a summary in this format (counts depend on the candidate):

passed: N/N scenarios passed; M packs not applicable

This is a format example, not a claimed test result. N counts executed scenarios; M counts optional groups that this configuration cannot test. The two report files are written into the current directory:

• eval-fast.json: individual scenarios, trials and assertion outcomes.
• eval-fast.xml: JUnit format for a CI test-results viewer.

A passing fast run proves that the generated composition handles the tested turns and tool behavior with a scripted model. It does not prove that your real model remembers facts, understands a book or survives a database restart.

2. Understand the result

A scenario is a conversation sequence. A step is one interaction in it. An assertion checks an expected property, such as “the tool succeeded.” A trial repeats a scenario against a fresh target. A pack groups related scenarios, such as memory or delegation. Evidence is what the runtime observed—not what the assistant claimed it did.

Result	Meaning	What to do next
passed / exit 0	Selected scenarios met their policies	Check which packs were not applicable before claiming coverage
failed / exit 1	Behavior did not meet an assertion or trial threshold	Locate the failing assertion; inspect the corresponding runtime behavior
error / exit 2	Evaluation execution failed; exit 2 also covers an invalid suite	Check imports, configuration and protected diagnostic logs
unavailable / exit 3	Required infrastructure was reported unavailable	Restore the dependency and rerun; do not lower the acceptance threshold
Pack not applicable	This target lacks a required feature or evidence channel	Enable/implement it if you need that coverage; this is not a pass

JUnit represents unavailable scenarios and non-applicable packs as skipped. Inspect their reason and the CLI exit code rather than treating every skip as optional success. Not every connection error is automatically classified as unavailable; an unhandled failure can be an execution error.

Read a JSON report without any extra tools:

python -c 'import json; r=json.load(open("eval-fast.json")); print(r["status"]); print(json.dumps(r["scenarios"], indent=2)); print(json.dumps(r.get("applicability", {}), indent=2))'

Within a scenario, inspect trials, then steps, then assertions: each assertion has a name and a passed boolean. Reports deliberately omit the chat text and raw exceptions. Keep local execution diagnostics when investigating; the report is not a transcript viewer.

Symptom	First check
No module named 'evals'	Project root, generated evals/ directory and PYTHONPATH=.
Answer check fails, retrieval check passes	Whether the retrieved facts are relevant, then model/prompt behavior
Answer passes, retrieval check fails	The model may have guessed or used conversation history; do not count this as recall
memory.consolidation has no-change	Read its reason and counts: it can mean no records, waiting for the final model round, or degraded extraction/storage
A stage is absent from the trace	Check composition, execution path and whether trace capture completed; absence alone does not establish the cause
Most packs are not applicable	Check whether you chose an offline profile and whether the target supplies the required features/evidence

3. Choose offline or live testing

Candidate-generated scaffolds contain evals/suite.py and evals/scenarios.py. Run these from that project's root:

PYTHONPATH=. symfonic eval evals.suite:SUITE --profile fast
PYTHONPATH=. symfonic eval evals.suite:SUITE --profile integration
PYTHONPATH=. symfonic eval evals.suite:SUITE --profile live \
  --json eval-report.json --junit eval-report.xml

Profile	Resources	What it tells you
fast	Scripted model and in-memory stores	Basic composition and deterministic behavior work
integration	Deterministic model and in-memory adapters; no network services	The wider generated composition works together offline
live	Configured real model, embeddings and durable storage	Applicable behaviors work against your deployed dependencies

Before a live run

Use a disposable test database, not a production account or real customer conversations. Live scenarios create records, invoke tools and run maintenance; unique test tenants provide isolation, not automatic cleanup or zero cost.

1. Copy the generated .env.example to .env and configure it locally.
2. Set LLM_PROVIDER, LLM_MODEL, LLM_COMPACTION_MODEL and the matching provider credentials. For an OpenAI-compatible service, set OPENAI_API_BASE to its API base. Chat and extraction may use different models.
3. Configure POSTGRES_DSN for the test Postgres/pgvector database. Start the services and run make migrate using the generated project's README. Addresses must be reachable from where the eval process runs (host versus container).
4. Configure the embeddings settings documented in the generated .env.example (EMBEDDINGS_URL or EMBEDDING_PROVIDER and its model settings). Graph-only recall does not establish that vector retrieval works.
5. Check the maintenance settings required by your selected scenarios. The reusable QUICK journey expects the five-turn cadence; don't silently change that contract to match a different deployment configuration.

Then run the live command above. It can make many model calls: book scenarios ingest multiple sections and repeat trials. No fixed duration or price applies to all providers. Start with a tagged project scenario, shown below, before running the complete catalog. Keep model settings and the candidate artifact fixed when comparing results. Remove disposable resources only after saving the reports you need; do not use a broad tenant-deletion command for cleanup.

4. Add a test for your own application

The generated evals/scenarios.py belongs to you. evals/suite.py already includes its PROJECT_SCENARIOS; you do not need to edit the runner.

The example below writes a preference in conversation writer, then asks for it in a different conversation, reader, in the same test scope. It requires both a response containing “rye” and a retrieval event admitting at least one memory. This is a starting smoke test, not proof that the specific preference was retrieved; the advanced assertions below support that stronger check.

Replace the initially empty file with this example, or append the scenario to your existing tuple without deleting your other tests.

from symfonic.evals import EvalStep, ResponseContains, Scenario, StageObserved

PROJECT_SCENARIOS = (
    Scenario(
        "customer-preference-recall",
        (
            EvalStep("Remember that I prefer rye bread.", conversation="writer"),
            EvalStep(
                "What bread do I prefer?",
                (
                    ResponseContains("rye"),
                    StageObserved(
                        "memory.retrieval", minimum_counts={"admitted": 1}
                    ),
                ),
                conversation="reader",
            ),
        ),
        tags=frozenset({"live", "memory", "my-project"}),
    ),
)

Run just your tagged example:

PYTHONPATH=. symfonic eval evals.suite:SUITE --profile live --tag my-project \
  --json eval-preference.json --junit eval-preference.xml

The live and my-project tags must both match. If you use different tags, change the command too. This is a real-model test, not part of the offline walkthrough. Do not expect every model to pass it on its first attempt.

5. Test reading a book

Use a book with answers you already know, not arbitrary text with no answer key. Symfonic includes Glass Harbor, a fictional manual with chapters, an erratum and a fixed truth manifest (the expected facts and their source identifiers). It lets you distinguish understanding from a plausible invented answer.

Question type	What the test asks the agent to do	What must be checked
Direct recall	Find a fact explicitly stated in one section	Correct answer and the relevant section actually admitted to the prompt
Paraphrase	Find the same information using different words	Relevant source retrieved, including the claimed vector route
Synthesis	Combine information from multiple chapters	All required sources and the resulting claim
Correction	Answer using the erratum rather than the old value	Current authority, supersession and no obsolete value asserted as current
Unknown answer	Answer a question the manual does not cover	Explicit lack of support instead of invention

The sequence is fresh test scope → ingest sections → ask questions → check answer and sources. An ingestion run alone is not a comprehension test. The generated live catalog wires the book journeys; inspect the applicability report to see whether your target can supply their required evidence. The source IDs in an assertion are book-section IDs, not conversation IDs.

Test your own Markdown or database records

You do not need a book. Your test data can be a policy folder, saved Markdown memories, exported database rows or another knowledge source. The important separation is records for the agent; questions and expected answers for the evaluator.

Download the runnable custom-knowledge example. It contains input records, a complete manifest, two data adapters and a suite that uses public Symfonic APIs. This is application-owned example code you can copy and adapt, not a new symfonic eval book command or a universal data importer.

Run the supplied example first

Use the candidate environment from the first walkthrough, with the cli, testing and openai extras installed from that same candidate wheel. Download the ZIP, extract it into a new directory, and open a terminal in the directory containing custom_knowledge/:

PYTHONPATH=. symfonic eval custom_knowledge.suite:SUITE --profile fast \
  --json knowledge-fast.json --junit knowledge-fast.xml

This runs three checks without a network connection:

Input record	Question	Accepted answer
data/returns.md	When are returns accepted?	Returns are accepted within 30 days of delivery.
data/warranty.md	When are warranty claims accepted?	Warranty claims are accepted within 90 days of delivery.
No matching record	Do you accept cryptocurrency?	NOT FOUND

Expected summary: passed: 3/3 scenarios passed. The offline provider echoes the selected record from the actual prompt; it never reads the manifest. This tests retrieval, prompt delivery and the assertion wiring—not model intelligence.

Replace the records and write your answer key

1. Put your UTF-8 .md files in a separate folder. One file is one record; returns.md has ID returns. Use stable, opaque lowercase IDs containing letters, digits, hyphens or underscores. IDs may appear in reports; do not put personal information in them.

2. Point the example at that folder and inspect its revisions:

   export KNOWLEDGE_DIR=/absolute/path/to/my-records
   PYTHONPATH=. python -m custom_knowledge.inventory
   

3. Create your own manifest.json using this structure. This example's hash is for the downloaded returns.md; replace it with your record's inventory hash:

   {
     "schema_version": 1,
     "cases": [
       {
         "id": "returns-window",
         "question": "When are returns accepted?",
         "answers": ["Returns are accepted within 30 days of delivery."],
         "sources": {
           "returns": "24df736f06350c95e26ecf81fa8b6426daad128aacaac5e39056e33a04da5e7a"
         }
       }
     ]
   }
   

4. Select the manifest and rerun:

   export KNOWLEDGE_MANIFEST=/absolute/path/to/my-manifest.json
   PYTHONPATH=. symfonic eval custom_knowledge.suite:SUITE --profile fast \
     --json my-knowledge.json
   

answers contains complete accepted answers, not keywords. Matching ignores case and repeated whitespace after removing [source:ID] citations; extra contradictory prose fails. Add reviewed aliases for legitimate alternative wordings. The offline echo model can only satisfy answers that match a retrieved record's text; inference and synthesis require a real model or another test provider.

sources identifies the records and exact text revisions that must reach the model. Hashes use the complete UTF-8 text, including final newlines. A changed record fails even if its ID stays the same. Review a change before updating its expected hash; automatically accepting every new hash would defeat the check. Do not feed this manifest to the agent.

For an unknown-answer case, use "answers": ["NOT FOUND"] and "sources": {}. This example deliberately requires zero admitted records as well as the abstention. It does not grade the more general case where irrelevant records are retrieved and the model correctly rejects them.

Use rows in a database instead of files

The example has a read-only SQLite adapter for an existing table:

CREATE TABLE memories (
    scope TEXT,
    id TEXT,
    content TEXT,
    PRIMARY KEY (scope, id)
);

To try it safely, create a new disposable database from the supplied records:

PYTHONPATH=. python -m custom_knowledge.seed_sqlite ./demo-knowledge.db
export KNOWLEDGE_DB="$PWD/demo-knowledge.db"
export KNOWLEDGE_SCOPE=demo
unset KNOWLEDGE_MANIFEST
PYTHONPATH=. symfonic eval custom_knowledge.suite:SUITE --profile fast \
  --json knowledge-db.json

The seed command refuses to overwrite an existing file. It also inserts a contradictory record under a different scope, so the demo does not assume that an empty neighboring scope proves filtering. The eval itself opens SQLite in read-only mode, selects the explicit scope with a bound SQL parameter, and makes no writes. KNOWLEDGE_DB takes precedence over KNOWLEDGE_DIR; unset it to return to Markdown. The supplied Markdown and database records use the same IDs/text, so the same manifest works for both.

For your existing SQLite database, set its path and authorized scope and use your own manifest. Different schemas require adapting the query. For existing pgvector or Atlas indexes, use the separate walkthrough below—do not export your whole index into SQLite just to run these tests.

Existing pgvector or Atlas Vector Search

The same download includes vector_search.py, connections.py and existing_suite.py. They query existing records in place: no ingestion, table creation, re-embedding stored records, index creation or production writes. Use read-only test credentials and an explicit authorized scope. Query embeddings still call your configured embedding service and can incur cost.

There are two distinct tests:

Command profile in existing_suite	What runs	What a pass proves
integration	Real embedding + real index query, no chat model	Required source IDs and text revisions are in the returned top K
live	Same query + public Agent + real chat model	Required records reach the actual model input and the answer/citations match

Profile names belong to a suite. Unlike the generated scaffold's offline integration profile, this explicitly selected existing_suite profile uses network services. It has no fast profile and never silently falls back to Markdown or an in-memory index.

Shared setup: your manifest and your existing embedder

Keep the earlier manifest format, but use questions, record IDs and hashes from your own reviewed source records. Hash the exact UTF-8 chunk text, including newlines, from an authoritative export/read—not an expected answer and not whatever an unverified search happens to return. This example uses a text hash as its revision; it does not infer freshness from a database timestamp.

The example schema has a string id, a string scope, a content string and an embedding vector. Use the chunk's stable ID, not its parent document's ID when several chunks share that parent. Adapt the field mapping if your schema differs; Atlas _id is not automatically substituted for id.

export KNOWLEDGE_MANIFEST=/absolute/path/to/my-manifest.json
export KNOWLEDGE_SCOPE=my-authorized-test-scope
export KNOWLEDGE_DIMENSIONS=1536
export KNOWLEDGE_TOP_K=5
export KNOWLEDGE_EMBEDDING_MODEL=your-existing-embedding-model
export KNOWLEDGE_EMBEDDING_BASE=https://your-embedding-service.example/v1
# Supply KNOWLEDGE_EMBEDDING_KEY through your secret mechanism.

The model name, URL and dimension above are configuration examples: replace them with your deployment's actual values. Use the same embedding model and query preprocessing as your stored vectors. Matching dimensions alone does not make different embedding models compatible. No fake/hash embedding fallback is used. Empty, zero, non-finite or wrong-dimension vectors are rejected before querying the database. The supplied embed_query uses an OpenAI-compatible embedding API; replace it with your own embedding function when needed.

Option A: pgvector

Install the driver in the candidate environment:

python -m pip install 'psycopg[binary]'
export KNOWLEDGE_SEARCH_FACTORY=custom_knowledge.connections:pgvector
export KNOWLEDGE_SCHEMA=public
export KNOWLEDGE_TABLE=memories
# Supply KNOWLEDGE_POSTGRES_DSN through your secret mechanism.
PYTHONPATH=. symfonic eval custom_knowledge.existing_suite:SUITE \
  --profile integration --json pg-retrieval.json --junit pg-retrieval.xml

This adapter uses cosine distance (<=>), a parameterized scope predicate, bounded LIMIT and a five-second statement timeout in a read-only transaction. Table/schema names are quoted as identifiers, not interpolated as SQL. It does not alter index parameters or guarantee the query planner chooses your ANN index. If your application uses inner product, hybrid search, reranking or extra visibility filters, adapt the query or use your existing search function instead; a different query does not establish parity with production. See the pgvector query and indexing documentation.

Option B: Atlas Vector Search

python -m pip install pymongo
export KNOWLEDGE_SEARCH_FACTORY=custom_knowledge.connections:atlas
export KNOWLEDGE_DATABASE=my_database
export KNOWLEDGE_COLLECTION=memories
export KNOWLEDGE_INDEX=my_existing_vector_index
export KNOWLEDGE_VECTOR_PATH=embedding
export KNOWLEDGE_CANDIDATES=100
# Supply KNOWLEDGE_ATLAS_URI through your secret mechanism.
PYTHONPATH=. symfonic eval custom_knowledge.existing_suite:SUITE \
  --profile integration --json atlas-retrieval.json --junit atlas-retrieval.xml

The adapter places the scope predicate inside $vectorSearch.filter, not only after nearest-neighbor selection. Your existing vector-search index must support that filter field. numCandidates, top K and the embedding path are explicit settings; use those of the application being evaluated. Only IDs, scope, bounded text and search scores are projected; embeddings are not returned. The query has a five-second server time limit. Use a database role with read-only permissions; a pipeline containing only reads is not a replacement for that role. See MongoDB's vector-search query contract.

Both adapters also check every returned row's scope before exposing fragments. A cross-scope row, duplicate ID, oversized chunk, invalid vector or driver error fails the evaluation rather than masquerading as “nothing found.” A scope filter is not your complete authorization policy: retain your deployment's user/group, publication, soft-delete and other access rules too.

Add the answer check

With either factory selected, configure the chat model separately:

export KNOWLEDGE_MODEL=your-chat-model
# Supply OPENAI_API_KEY; set OPENAI_API_BASE for a compatible local/gateway API.
PYTHONPATH=. symfonic eval custom_knowledge.existing_suite:SUITE \
  --profile live --json index-agent.json --junit index-agent.xml

A retrieval pass followed by a live failure narrows the investigation: inspect prompt admission/size limits, citations and answer wording. It does not by itself prove that the LLM is at fault. A top-K nearest-neighbor query can return irrelevant records for an unanswerable question; the example's empty-source negative case will fail unless your actual retrieval policy returns no records. Do not add an arbitrary similarity cutoff just to make it green.

Prefer your own search function when one already exists

Set KNOWLEDGE_SEARCH_FACTORY=my_app.eval_search:build to reuse an existing retriever rather than recreate production's ranking in our sample SQL/pipeline. The factory is trusted executable application configuration, never part of the answer manifest. It must return an object with:

• retrieve(question, limit): calls your authorized search and returns bounded RetrievedFragment values with source labels record-id:sha256-of-text.
• top_k: the explicit search limit, and returned: the latest returned fragments, reset for each query.
• close(): releases any clients the factory owns.

The supplied adapters are complete examples of that contract, not requirements to change your storage format. Their result budget is at most 20 records and 8 KiB per chunk; prompt admission has its own independent budget. They test search completeness against required sources, not ranking metrics such as NDCG or a promise that the database performs bounded internal scan work.

If you want to validate the existing application, not the example Agent, connect its AgentTarget, HostTarget or HttpChatTarget and real evidence adapter. Reusing a database is not the same as testing the deployed application's prompt, retriever configuration, permissions and routing.

Verification boundary: driver contract tests cover both adapters. The pgvector example was also exercised on a disposable local Postgres/pgvector database with populated scopes. No live Atlas cluster or consumer production index is claimed validated by those tests.

Try your real model, then your real retriever

The example supports an OpenAI-compatible model through the public provider:

export KNOWLEDGE_MODEL=your-deployed-model-name
# Configure OPENAI_API_KEY through your local secret mechanism.
# For a local/gateway service, also set OPENAI_API_BASE to its API base URL.
PYTHONPATH=. symfonic eval custom_knowledge.suite:SUITE --profile live \
  --json knowledge-live.json --junit knowledge-live.xml

Live mode changes the model, not the retriever. records.py deliberately uses a small lexical top-one search, not embeddings. Replace Retriever.retrieve with your actual scoped search to evaluate vector, hybrid or database retrieval. Return RetrievedFragment values and keep the example's ID/revision source labels and returned list so the evidence adapter can verify their delivery. If your application formats context differently, adapt the evidence reader to that actual format rather than fabricating admission from the manifest.

The safety bounds here are intentionally small: at most 200 records and 8 KiB per record. Oversized inputs are refused; the example is not a scalable database retrieval implementation. Knowledge and prompt size limits may additionally omit a record, which must fail a required-source assertion rather than count as successful retrieval.

What a passing result proves—and does not prove

The check reads messages captured at the model boundary. A record counts only when its complete rendered fragment appears inside the actual untrusted context block. It then requires the expected revision, the required citations and an accepted answer. A correct guessed answer with the wrong record fails; merely returning a row from the database does not establish prompt delivery.

This example runs a new public Agent with a knowledge/prompting composition; it does not attach automatically to your existing scaffold or HTTP service. To validate that application, retain the scenarios/assertion but connect its real target and evidence adapter. The example does not establish automatic memory extraction, durable writes, cross-chat recall, restart, multi-tenant authorization, semantic similarity or general absence of hallucinations. Use the dedicated journeys for those behaviors.

Reports contain case IDs and assertion outcomes, not record text or expected answers. Provider-boundary content is captured temporarily in process memory for these checks; use synthetic or authorized test data and protect the process accordingly.

6. Use evals in CI

For your generated application's CI, install its dependencies, change to the project root and run the offline gate:

- name: Fast agent regression gate
  run: PYTHONPATH=. symfonic eval evals.suite:SUITE --profile fast --junit eval-fast.xml

Run integration offline too. Run live separately in a protected environment with test storage and model credentials, never on untrusted pull-request code. Archive reports and record the tested artifact/configuration. A library test suite does not replace testing a newly generated app from the release wheel.

Advanced reference: assertions and target adapters

You can stop here if you use the generated suite and add project scenarios. The remaining sections describe stricter checks and how to connect a different application. A target is the adapter that runs a scenario against an agent or application. Its evidence channels provide observed runtime facts to assertions. A trait declares a specific capability that a general stage name alone cannot prove. SC-* labels below are stable scenario identifiers, not steps you must implement in numerical order.

Book ingestion and grounding contracts

Read the actual book first

Glass Harbor is shipped as UTF-8 Markdown text, not a PDF. It contains eight chapters, an appendix and an erratum. No OCR or PDF parser is involved in these book tests; validating PDF extraction would be a separate test.

• Read or save the complete Glass Harbor text.
• Read or save the answer key and section checksums.

These downloads are generated from the same packaged fixture the tests load. The text download joins the ten sections for human reading; the test ingests them separately, preserving each section's identifier and checksum. The JSON manifest contains the questions, expected answers and supporting section IDs. It is the evaluator's answer key—do not give it to the agent as reading material.

You can also read the book from your installed candidate without downloading anything. Save this as read_book.py in your generated project and run python read_book.py:

from symfonic.evals import load_book_fixture

book = load_book_fixture()
print(book.title)
for section in book.sections:
    print(f"\n--- {section.id}: {section.title} ---\n")
    print(section.text)

This prints the book; it does not store it in the agent's memory.

Run one real book test

After completing the live configuration above, run this from the generated project root:

PYTHONPATH=. symfonic eval evals.suite:SUITE --profile live \
  --tag book --tag retrieval --trials 1 \
  --json book-retrieval.json --junit book-retrieval.xml

The tags select glass-harbor-hybrid-retrieval. --trials 1 is for your first exploratory run; the shipped scenario normally repeats three trials. This command calls your real model and uses your configured storage and embeddings. It does not read a PDF you upload to the chat UI.

Here is what that single trial does:

1. Allocates a fresh test scope and checks that it contains no fixture records.
2. Sends the text of each of the ten sections in a separate turn, asking the agent to save it through remember.
3. Requires a successful remember tool result for each section, then audits stored IDs/checksums for missing or duplicated records. “I've saved it” is not sufficient.
4. Asks the two questions below in new conversations in that same scope.
5. Checks the answers, citations and the sources actually retrieved into the model's prompt. It writes assertion outcomes to the two report files.

A concrete question and its answer

Chapter 2 says:

The north beacon uses a cobalt key, and it is used only when the tide gauge on the inner mole reads below 2.4 metres.

The hybrid-retrieval scenario asks these actual questions:

Question	Expected meaning	Additional runtime proof
Which key operates the northern light, and under what water-level condition?	Cobalt key; tide gauge below 2.4 metres	Chapter ch-02 admitted through graph retrieval; current claims supported by ch-02 and erratum-01
When the sea marker reads 2.1 metres, how should the beacon be activated?	Use the cobalt key: 2.1 is below the 2.4-metre threshold	Chapter ch-02 admitted through vector retrieval, not just some unrelated vector result

The test appends instructions to cite sources as [source:ch-02] and to emit machine-readable claim annotations. Those annotations help evaluate the answer; they are not substitutes for correct visible prose or actual retrieval.

For example, “Use the cobalt key below 2.4 metres” can still fail if the agent never retrieved the relevant section. “Use the slate key” fails the answer check. A correct answer with invented citations also fails. Conversely, a missing embedder is a configuration problem to investigate—not a reason to remove the vector assertion.

To run all five book scenarios, including synthesis, the erratum and questions the book cannot answer, use:

PYTHONPATH=. symfonic eval evals.suite:SUITE --profile live --tag book \
  --json book-suite.json --junit book-suite.xml

Each scenario repeats its own ingestion for every trial; it does not reuse the first command's memories. Expect substantially more model calls. These are the fixed Glass Harbor checks, not a universal grader for any uploaded book. For your own text, author an answer key and scenarios that cite its sources.

Adapter reference: two ingestion routes

Use a closed fixture corpus for grounding. Require citations to IDs returned by retrieval with CitationsSupported; include a distractor and an identifier that does not exist. For tools, assert ToolCalled and ToolSucceeded, not prose claiming that a tool ran. For mutations, use a business idempotency key: model tool-call IDs correlate transcript records and are not replay keys.

The shipped Glass Harbor fixture has two public ingestion routes. The default scaffold route creates one remember turn per checksum-pinned section:

from symfonic.evals import remember_fixture_scenario

BOOK_INGESTION = remember_fixture_scenario()

An adopter that composes document knowledge can select the same sections through the public knowledge bridge:

from symfonic.capabilities.knowledge import ContextRequest, knowledge_sources
from symfonic.evals import FixtureDocumentStore

documents = FixtureDocumentStore()
ids = documents.select_ids(("ch-02", "erratum-01"))
sources = knowledge_sources(store=documents, document_ids=ids)

The knowledge bridge and prompt compiler enforce separate size ceilings. Raise the public prompting policy to the exact selected block size so a valid chapter is not silently dropped by the smaller learned-content default:

from symfonic.capabilities.knowledge import ContextRequest, knowledge_sources
from symfonic.capabilities.prompting import PromptingCapability, RenderPolicy
from symfonic.evals import FixtureDocumentStore

documents = FixtureDocumentStore()
ids = ("ch-02", "erratum-01")
source = knowledge_sources(store=documents, document_ids=documents.select_ids(ids))[0]
required_chars = len(source.read(ContextRequest("glass-harbor")).text)
capability = PromptingCapability(
    sources=(source,),
    options={
        "policy": RenderPolicy(max_learned_chars=required_chars)
    },
)

Call fixture_ingestion_applicability with the actual registered tool names and the concrete document adapter. Its report always lists both routes; an absent optional document route is not_applicable, never an omitted or passing test. FixtureIngestionComplete expects payload-free evidence containing section id, source checksum, record id, and layer, and refuses missing, changed, or duplicate records without copying book or memory content into the report.

BOOK_GROUNDING_SCENARIOS is the reusable SC-05 through SC-09 acceptance pack. Each comprehension scenario first proves that its physical scope contains zero fixture rows, then performs and audits all ten remember turns. Each scenario runs three trials and therefore requires a newly allocated physical scope from the target factory per trial; it never relies on another run having populated the store. The pack then checks exact and paraphrased hybrid retrieval, a derivation, cross-chapter synthesis, signed supersession, and three unsupported premises.

Targets provide payload-free grounding evidence alongside each observation:

• retrieved_source_ids: source ids admitted to the model prompt;
• response_claims: normalized fact/value annotations parsed from the response itself, so naming an obsolete value is distinct from recommending it;
• derivation_inputs_used: stable inputs used to compute a derived answer;
• admitted_sources_by_route: source ids admitted through graph and vector retrieval independently;
• retrieved_source_records: source id, rank, status and supersession metadata;
• authority_source_ids: the source selected as current authority;
• unsupported_fact_ids and unsupported_claim_count: closed-corpus verdicts from the response grounding pass.

BookClaimGrounded requires the fixture's answer fragments, every required source both admitted and cited, no forbidden current assertion, and all declared derivation inputs. Exact token boundaries and the manifest's normalized expected values prevent 240 from satisfying 24, or a related word from satisfying an expected key name. SourceRouteAdmitted proves the relevant source, rather than an arbitrary distractor, came through the vector route. SupersessionResolved requires the reciprocal edge plus a current-first rank or explicit authority selection. ClosedCorpusGrounded requires NOT IN MANUAL, a normalized unsupported verdict, zero invented claims in annotations parsed from the actual response, and citations only for an admitted supported correction.

Use build_book_response_evidence(response) on the response returned by the deployment. The prompts request [assert:fact=value], [unsupported:fact-id] and [input:value] annotations; the builder validates and normalizes those annotations and independently counts propositions absent from the fixture truth. BookClaimGrounded checks the claim's declared visible forms separately from its normalized proposition, so a canonical paraphrase does not have to repeat the storage value verbatim. It compares expected and forbidden values with nearby polarity/currentness cues inside each visible clause, independent of word order; an expected token and annotation cannot hide prose that calls that value historical or declares a forbidden value current. ClosedCorpusGrounded independently rejects the negative fact's absent terms in annotation-stripped prose and requires supported correction values to be visible as whole terms. Its absence marker must also be visible; hiding NOT IN MANUAL in an annotation does not count. Do not synthesize the resulting counters from the expected answer.

Memory classification and consistency

Content-bearing memory evidence is opt-in. A target may expose the exact delimited recall contribution as prompt_recall, normalized fact mappings as recall_claims and response_claims, and classification records containing only record_id, category, and subject as memory_classifications. This enables PromptRecallContains, ResponseDoesNotContradictRecall, and MemoryCategorySeparated respectively. The assertion results contain only counts and hashes; raw memory and normalized values never enter reports.

Graph and maintenance targets can opt in with memory_edges, a consolidation_ledger, paired consolidation_cycles, and normalized fresh_conversation_claims. Use MeaningfulMemoryEdges to require typed edges with readable endpoints, ConsolidationLedgerComplete to account for the phase roster, ConsolidationIdempotent to prove a replay is a stable no-op, and FreshConversationConsistency to bound claim drift across new chats. These records should contain stable identifiers, labels, counters, and state digests; do not place raw memories in the evidence.

QUICK consolidation (SC-11)

SC_11_QUICK_CONSOLIDATION adds the frequent-maintenance journey. A real turn first populates the comparison tenant. The first four successful owner-scope turns require that no cycle has completed; the fifth requires one applied memory.nap event, all four QUICK phases accounted for, and a consolidation_mutation record joining the cycle counters to before/after record identifiers. The four registered phases must be unique and partitioned exactly once across run, skipped and failed; overlaps and extra phases fail. The same mutation record carries the triggering and actual scope, the nonempty set of protected identifiers that survived, the independently eligible and examined candidate identifiers, and the digest of a populated second tenant before and after the cycle. ConsolidationStoreMutation rejects a counter-only success, inflated or unrelated candidate counts, a missing protected row, scope drift, or an empty or changed comparison tenant. The final turns require the consolidated profile in the exact model input and verify that its canary is absent from the other tenant. Targets may expose identifiers and digests, but must not put memory text in this evidence.

DEEP consolidation and graph recall (SC-12)

SC_12_DEEP_GRAPH exercises the public manual DEEP entry over retained manual and profile evidence. It requires all 17 phases to be accounted for, a held scope lease to exclude a second worker without mutation, and the first real cycle to create typed MENTIONS edges whose endpoints are readable labels rather than storage identifiers. The edge evidence also joins each relation to its source episodic record identifiers and entity_linker provenance. A second unchanged cycle must report zero entity/edge mutations and preserve the complete node-and-edge identity set and digest. Finally, a fresh conversation starts from one entity seed and uses SpreadingActivation over the created relations. Every traversed relation is present in the exact model input, and each edge's cited episodic source must contain both endpoints and have produced both during extraction. Disabling traversal, substituting an unrelated valid episode, or adding an unrelated node or non-MENTIONS edge on replay must fail; a fluent answer or plausible counters cannot replace those proofs. A populated comparison tenant is checked before and after owner-scope maintenance, and a final control verifies that none of the owner's entity labels become visible there.

The reusable MEMORY_PROFILE_SCENARIOS pack covers the default memory contract without conflating a fluent answer with recall. It exercises same-chat working context, repeated semantic-profile recall in four fresh conversations, explicit memory across restart, a neighboring-code negative control, and separation of profile, saved content, past event, procedure, and future commitment categories. Include the pack only when the target exposes the opt-in evidence fields above; a missing field is a failed proof rather than a silently skipped assertion.

Security and isolation

The SECURITY_ISOLATION_SCENARIOS pack supplies the adversarial counterpart. It puts an instruction canary in stored recall, requires that exact text inside the memory.recall untrusted wrapper, checks a separate secret canary across the response, events and target-loaded diagnostics, and requires structured evidence that governance blocked the requested privileged effect. Its isolation journey first proves that the source rows exist and are visible at their origin; only then can a zero-admission read from a sibling conversation or tenant count as isolation. IsolationBoundaryObserved rejects an empty-store false green.

Observability (SC-18)

SC_18_OBSERVABILITY joins the four projections an operator relies on. For each memory, tool, and QUICK turn, ExecutionTraceJoined requires the same nonempty run and conversation identity and the same ordered kernel-owned event indices in the live stream, durable execution rows, the generated Turn Inspector payload, and OTel child spans. Text deltas must remain ordered root events rather than flooding the waterfall. Inspector memory totals and tool statuses are recomputed from the durable rows, so a fluent answer or a plausible summary cannot substitute for execution evidence. Each OTel child must carry the root trace ID and name that root span as its parent; the Turn Inspector's conversation and run IDs must identify exactly one turn. A same-run orphan span or a UI group containing two turns fails.

The journey also drives metadata and content tracing over the same turn. TracePrivacyModes requires metadata to remain the default and contain no payloads, while content capture must be an explicit opt-in, access-controlled, bounded by the declared byte ceiling, credential-redacted, and free of raw reasoning fields. At least one model artifact must carry nonempty model input, output, and admitted recall that match the actual request, response, and recall block; unrelated nonempty content is not evidence. Access is proved by real authorized and anonymous HTTP requests to the Grafana Tempo-proxy path, and the trace ID returned to the authorized caller must be the captured trace. Tempo remains internal to the generated Docker network so this proxy is the only host-facing path. A deployment target supplies these projections in the observability evidence attribute; assertion failures report only which contract was missing and never copy captured content into the eval report.

Concurrency and restart (SC-19)

SC_19_CONCURRENCY_RESTART is the live durability and isolation soak. It must run from an installed wheel against a freshly generated project and a durable Postgres store; the hermetic profiles report it as not applicable. Four processes wait behind one pipe barrier before writing: three conversations in one tenant and one conversation in another. Evidence joins each distinct worker, run, conversation, resolved scope path, kernel event run ID, stored turn ID, and published record ID. Each scope must then see exactly its own records, while every publication is accounted for exactly once.

The maintenance race uses a separate process to hold the generated app's scope lease. A contender must return already_running without phases or commits; only after an explicit pipe release may another process complete the DEEP roster. This makes the interleaving deterministic rather than relying on timing sleeps. The final step boots new application processes and compares payload-free projections of memory identity, retrieval order, graph state, and reviewed procedures. Replaying unchanged consolidation after the restart must report zero mutations and preserve the complete state digest. Changing a scope mapping, weakening the durable backend, skipping the restart, allowing two lease winners, duplicating a publication, or mutating replay state makes the scenario fail without placing prompts or memory content in the report.

Bounded memory volume (SC-20)

SC_20_BOUNDED_MEMORY_VOLUME is the ordinary scale regression gate: ten thousand durable records, one hundred topical distractors, all five memory layers, relationships, sibling conversations, and a second tenant. The corpus is yielded in scope-homogeneous batches of at most 200 records, so the fixture itself cannot hide a full-corpus allocation before the store is measured.

The five questions have protected record identities. SC-20 compares those identities before and after QUICK and DEEP; a fluent answer is not evidence and the small live model is never an oracle. Its hard default budgets are:

Work	Ceiling
retrieval p95	1,500 ms
graph + vector candidates per question	128
peak RSS growth	96 MiB
seed batch / admin page	200 rows
browser node or edge projection	400 rows
QUICK / DEEP identities examined	500 each

The live gate builds a wheel, installs it in a clean virtual environment, generates a project from that wheel, and drives PostgreSQL with pgvector. It records row counts returned by each graph read, two admin pages across a page boundary, retrieval source counts, phase ledgers, tenant/conversation-negative reads, inventory deltas, and protected identities. Reports carry none of the memory text. Removing the graph candidate limit, making an admin page unbounded, exploding phase work, loading the complete tenant graph, or losing a protected fact each has a mutation that turns the scenario red.

Ten thousand is a regression workload, not a retention promise. Larger deployment sizing should raise a separate profile with its own measured budgets; silently raising these ceilings makes a release slower without saying which operational envelope it now supports.

Optional capability packs

Optional behavior is selected from what the agent actually compiled, not from an adopter-maintained feature list. compiled_evidence() compares the supplied composition with the agent's immutable, payload-free composition manifest; same-name capabilities with different sources, tools, or preconditions are not accepted as equivalent.

For a generated application, start with optional_packs in evals/catalog.py; that is the shipped wiring, not an extra function you must write to run evals. For a custom target, use probed_traits to obtain declared traits and compiled_evidence to describe its actual composition. Pass the resolved pack results to applicability_report and applicable_scenarios. These functions need your agent, target and runtime services; they are adapter APIs, not a standalone copy-and-paste example.

The scaffold scenarios assert the capabilities actually folded into the compiled Agent. Removing a capability without updating behavior therefore fails the suite instead of leaving a stale configuration green.

The generated evals/catalog.py also wires the reusable memory-profile, fixture-book, tool, delegation, security and QUICK journeys into the live catalog. Optional packs are resolved after compiling a disposable target for the selected profile. That probe is closed before trials begin, so it cannot seed a memory that later makes continuity look healthy.

JSON reports include applicability.packs and applicability.not_applicable. JUnit writes every not-applicable pack as a named skipped case with its missing capability, trait, operation, turn input or attributed tool. This is not a pass and it is not an omission. The generated fast and integration profiles intentionally report live-only packs as not applicable while remaining fully hermetic.

The generated target also installs ScaffoldEvidenceAdapter. It receives a completed runtime turn and reads the actual provider calls, committed memory records, decision sink, and telemetry resources; it is never handed the requested prompt, state, or attachments. Its declared evidence channels are therefore capabilities of that adapter, not promises that an input was used. For example, multimodal applicability requires both the public attachments turn input and provider-boundary attachment_delivery evidence. Dropping the attachment before the provider fails the scenario even if the requested bytes were correct. Knowledge packs likewise require an exact configured source trait such as knowledge.fixture:glass-harbor-v1; a generic knowledge stage or a different source cannot make the fixture pack applicable.

The explicit durable-memory negative check is a separate memory-unsupported-claims pack. It is not applicable until the deployment provides an authoritative unsupported_claims evidence channel that inspects the response against its own closed truth. The generated adapter does not guess that count from fluent prose, so SC-04 is reported by name instead of running with evidence it cannot produce.

A probe distinguishes an absent optional feature from an operational failure. Only explicit absence produces not_applicable; database, network, and implementation failures remain errors. Publish the applicability report beside the eval report so an omitted pack never looks like a passing one.

The shipped packs cover:

Pack	Required compiled evidence
procedural skill	tools plus procedural review and precondition traits
approval/resume	human capability plus the target's real resume operation
extension tool	the exact attributed extension tool, not an empty bundle
knowledge grounding	prompting, the exact configured source trait, and provider-boundary context evidence
structured output	the public output_type turn input
multimodal attachment	the public attachments turn input plus provider-boundary delivery evidence

Approval uses the deployment's public checkpoint-backed resume seam and checks continuation plus token replay refusal. Procedural evaluation proves that a draft is inert before approval and then correlates the reviewed procedure's stable identity with the identities read by the real precondition on both the allowed and blocked turns. A static same-name gate cannot satisfy that proof.

Multimodal evidence records ordered block kind, source type, MIME type, and SHA-256 digest. This proves that the expected bytes reached the provider without placing attachment content in reports. Procedural evidence is similarly limited to identities, statuses, counts, and digests.

Thresholds

Keep deterministic scenarios at one trial and a threshold of 1.0. For a live model, state the tolerated variance explicitly:

from symfonic.evals import TrialPolicy

policy = TrialPolicy(trials=5, pass_threshold=0.8, timeout_seconds=180)

Never lower a threshold to accommodate an infrastructure outage. Missing infrastructure is unavailable; a timeout is an error; an assertion mismatch is a behavioral failure. Those outcomes demand different fixes.

Debug safely

JSON and JUnit reports contain scenario names, counters, structured verdicts, and redacted reasons. They intentionally omit prompts, memory text, tool arguments, credentials, and exception messages. Diagnose with correlation IDs against access-controlled local telemetry. Do not add raw production chats to fixtures or upload unredacted reports as CI artifacts.

When an answer is correct but an evidence assertion fails, investigate the evidence: the model may have guessed or used its current transcript. no-change is an outcome, not a diagnosis. Inspect its reason and counters; it can also describe waiting or an operational failure. A missing event calls for checking both the execution path and the completeness of trace capture.

Maintainer reference: release validation

Run the hermetic gate on every change and publish only redacted reports:

- name: Fast agent regression gate
  run: PYTHONPATH=. symfonic eval evals.suite:SUITE --profile fast --junit eval-fast.xml

Run integration without network services. Run live in a protected job with pinned model and embedding endpoints, temperature zero, explicit timeout, and secrets supplied by the CI secret store. Do not run live evals on untrusted pull-request code. A release gate should build the wheel, generate a fresh scaffold from that artifact, and execute the live profile against that project; testing the source checkout alone cannot detect a broken shipped template.

An optional semantic judge may assess style or completeness, but label it nondeterministic and keep deterministic evidence assertions authoritative.

Protected live release gate

This section describes Symfonic's source repository, not a workflow automatically installed in every adopter's project. Paths beginning scripts/ or .gitea/ here belong to that repository.

The repository uses three gates:

When	Workflow	What a failure means
Every PR and main push	scaffold-evals.yml	A deterministic contract or wheel-generated fast/integration journey failed. No live secrets are used; generated eval execution denies network access.
Nightly at 08:00 UTC or manually	scaffold-live-release.yml	The default live catalog regressed, or its configured infrastructure is unavailable. Investigate the failed Actions run.
Before public PyPI upload	publish-pypi.yml	Publication stops until the live gate passes on the exact wheel being published.

The live gate generates a clean default project from the wheel, migrates a dedicated evaluation Postgres/pgvector database, and runs the project's complete live catalog against the configured model and embedding endpoint. It never runs on pull requests. Public publication supplies --wheel rather than rebuilding the artifact; the report records its SHA-256 and distribution checksums are verified again immediately before upload. Unavailable infrastructure is a failure, not a successful skip.

Maintainers must configure the SYMFONIC_LIVE_* secrets listed in scripts/scaffold_live_gate/README.md, restrict live runs to trusted code, and provide runner access to the test services. Never point this gate at a production database. Add the offline job's reported check context to branch protection: committing a workflow alone does not make it required for merging. These are repository administration steps, not settings installed by symfonic init.

The policy in scripts/scaffold_live_gate/baseline.json pins every default pack as either applicable or explicitly not applicable. Applicable semantic scenarios run three trials and need two passes; stage, tool, storage, isolation, applicability and other deterministic evidence must pass on every trial. The process has bounded per-step and job timeouts, retries migrations at most three times, and retains reports for fourteen days.

Memory classification is persisted structured evidence: each extracted node declares memory_category and subject together, with the category compatible with its storage layer. Invalid declarations are dropped with stable classification_* codes in ExtractionResult.dropped. Neither the eval target nor consolidation infers missing identity from a fixture's prose. Validated profile/user records receive the same reserved recall slot as legacy SOUL profiles, so topical memories do not silently crowd out the user's identity. Validated past_event memories can cross conversations within the same scope; raw episodic transcript rows remain conversation-local. Sharing a storage layer does not make a remembered event a transcript.

Response-category and empty-memory-claim checks are separately named semantic assertions. SC-10 also checks recalled facts in the actual prompt and response; category headings alone cannot pass. The live target records its auxiliary extraction provider separately, preserving the deployed Qwen extraction configuration while the conversational provider retains its own settings. prompt_recall includes only system-message memory.recall blocks, never the current user prompt or conversation history. Full input remains model_input.

MemoryQuery.candidate_limit bounds candidate rows retained/returned by each route (default 64, maximum 500), separately from prompt admission. Native Postgres, MongoDB and in-process selectors filter scope, publication, layer and conversation eligibility before ranking and capping. A reserved profile slot cannot be crowded out by topical rows. This bounds materialized rows, not database scan CPU or physical pages. Administrative reads use MemoryAdminService.record_page when they need omission evidence. A bounded export marks itself incomplete if the raw scan hits its ceiling, including when filtered pending rows consumed that ceiling. It must not infer completeness from the smaller number of returned records. Custom graph backends without CandidateSelectionPort retain a conservative bounded-prefix fallback with an incomplete marker; implement the native port to avoid prefix starvation. CandidateRequest(exact_scope=True, profile_slots=0) provides a bounded same-owner lookup for maintenance, not a lifecycle cursor.

RecallBudget(max_total_bytes=8192, max_record_bytes=4096) can be passed to memory_capabilities or MemoryBundleFactory as recall_budget. It counts UTF-8 rendered memory lines, including category/subject metadata, and replaces the legacy character ceilings in both selection and hydration. Oversized facts drop whole. Compiler wrapper markup is outside this payload budget. Generated projects expose MEMORY_RECALL_LIMIT, MEMORY_RECALL_MAX_BYTES and MEMORY_RECALL_RECORD_MAX_BYTES; recall count no longer borrows the setting named WORKING_MEMORY_RECENT_TURNS. The scaffold also aligns PromptingCapability(options={"policy": RenderPolicy(max_learned_chars=...)}) with that byte ceiling. Custom composition roots must configure this independent learned-content gate too: its default 500-character ceiling otherwise drops an oversized contribution in full even when retrieval admitted it. The content remains untrusted; raising a size ceiling never changes its trust tier.

Validated memory_category and subject now travel with the recalled text inside the untrusted memory block. SC-10 requests a table and uses the public ResponseMemoryCategorized assertion to check each fact's own category column. PromptMemoryCategorized independently requires that category on the same recalled record line; finding a classified record elsewhere in the store is not enough. This is a hard input check, not part of the response pass threshold. The renderer escapes pipes in retained prose as \u007c before appending its metadata suffix, so content cannot forge a category by imitating that suffix. The scaffold's remember tool accepts the public MemoryCategory enum and a subject, and derives the storage layer from MemoryCategory.layer. Generic content-only calls remain valid as saved content, with a content-derived identifier; other categories require a subject. Retaining a future commitment records a memory, not a scheduled reminder. Procedure memories are not approved executable skills merely because this tool stored their text. Mentioning all five category names elsewhere cannot pass; a storage-layer label such as semantic does not count as a category. These are semantic response checks, separate from hard store, scope and actual-prompt evidence.

Only the projected release report and payload-free JUnit file are uploaded. The projection contains a gate identifier, scenario/assertion counts, latency percentiles and SHA-256 fingerprints for the model configuration, wheel, fixture and policy. Raw prompts, responses, memory text, tool arguments, endpoint URLs, credentials and exception messages are neither projected nor uploaded. Exit 3 distinguishes unavailable infrastructure from behavioral exit 1; an outage must never become a green baseline.