Skip to content

Guide 09 — Corpus Scrub for Phase 12 LLM-Extractor Eval

v7.8.0 shipped symfonic.tools.corpus_scrub — a public adopter-facing module for redacting PII from Phase 12 episodic corpus exports BEFORE they leave the adopter's environment. This is the mandatory pre-flight for the v7.9.0 LLM-extractor eval: raw production data must not cross the symfonic boundary.

This guide walks through the design contract, the CLI workflow, the programmatic API, and the recipe for a MongoDB-backed export (modelled on a typical production export shape).

When to use this

  • You're an adopter preparing an episodic-stream sample so the symfonic-core team can measure the v7.9.0 LLM-extractor's precision/recall on your tenant's production patterns.
  • You're scrubbing a one-time corpus snapshot for offline eval — not a runtime stream. The scrubber is single-pass file I/O, not a SymfonicAgent component.
  • Your data carries business PII (customer names, emails, addresses) but follows known structural patterns (product IDs, account IDs, tool call traces).

Design contract

Three guarantees:

  1. Scrubbing happens adopter-side. symfonic-core ships the patterns + tokeniser; you run them locally and upload only the scrubbed output. Raw PII never crosses the network.
  2. Tokens are stable per-export. The same input string always maps to the same <REDACTED-EMAIL-N> within one CorpusScrubber instance, so co-occurrence patterns inside an export are preserved. Different exports start counting from 1 again — there is no cross-export token leakage.
  3. Drop on doubt. Rows whose unresolved-match count exceeds a threshold (default 3) are dropped from the output rather than shipped with partial scrubbing.

CLI workflow

The simplest path:

python -m symfonic.tools.corpus_scrub \
    --input  dryrun_corpus.jsonl \
    --output dryrun_corpus.scrubbed.jsonl.gz \
    --manifest scrub_summary.json

The CLI accepts plain .jsonl or gzipped .jsonl.gz on input. Output compression is determined by the extension on --output. The manifest sidecar carries row counts and per-label match counts — attach it next to the scrubbed file when handing the corpus off.

CLI flags

Flag Default Purpose
--input / -i required Input .jsonl or .jsonl.gz
--output / -o required Output path. .gz extension enables gzip
--manifest none Optional scrub_summary.json sidecar
--unresolved-threshold 3 Max unresolved matches before row drop
--no-drop-on-doubt off Disable the threshold drop (analysis runs only)

Programmatic API

from pathlib import Path
from symfonic.tools.corpus_scrub import CorpusScrubber, scrub_jsonl_file

scrubber = CorpusScrubber(unresolved_threshold=3)
summary = scrub_jsonl_file(
    input_path=Path("dryrun_corpus.jsonl"),
    output_path=Path("dryrun_corpus.scrubbed.jsonl.gz"),
    scrubber=scrubber,
)
print(summary)
# {
#   "input_rows": 1000,
#   "output_rows": 997,
#   "dropped_rows": 3,
#   "match_counts": {"EMAIL": 412, "PHONE": 88, "SELLER_ID": 156, ...}
# }

Or row-by-row when you need finer control:

from symfonic.tools.corpus_scrub import CorpusScrubber

scrubber = CorpusScrubber()
for row in raw_rows:
    result = scrubber.scrub_row(row)
    if result.scrubbed is None:
        log.warning("dropped row: %s", result.dropped_reason)
        continue
    yield result.scrubbed

Pattern families

The default scrubber applies three pattern groups in order:

1. Value credentials (DEFAULT_VALUE_CREDENTIAL_PATTERNS)

Bearer tokens, JWTs, sk_live_/pk_test_ prefixes, AWS access keys (AKIA/ASIA), GitHub tokens (ghp_/gho_/etc.), Anthropic/OpenAI keys (sk-ant-…/sk-…). Redacted to <REDACTED-TOKEN-N>.

Distinct from symfonic.agent.hygiene.compile_credential_pattern which scans KEY NAMES — these patterns scan VALUES.

2. Generic PII (DEFAULT_PII_PATTERNS)

Email, phone (US + international), URL (http/https), IPv4, street address fragments. Each gets its own label (<REDACTED-EMAIL-N>, <REDACTED-PHONE-N>, etc.) so downstream co-occurrence learning can still detect "the user mentions the same email twice in a conversation."

3. Domain-specific keep/redact

Keep-patterns are checked FIRST and protect their spans from redaction:

  • PRODUCT_ID (B[0-9A-Z]{9}) — structural product identifier, not PII
  • REGION_CODE (documented public constants) — not PII

Redact-patterns run after keep-patterns are masked out:

  • ACCOUNT_ID (A[0-9A-Z]{12,15}) — account identifier, PII
  • ORDER_ID (\d{3}-\d{7}-\d{7}) — purchase reference, PII

These structural patterns are illustrative — register your own domain's identifier shapes via the keep/redact pattern lists.

Recipe: MongoDB export → scrub → ship

The shape a production dry-run pipeline produces, end to end:

# 1. Adopter-side: extract from MongoDB into per-message JSONL rows.
#    Each row carries:
#        tenant_id   : pseudonym (NEVER the real brand_tenant_id)
#        created_at  : per-MESSAGE timestamp (synthesise if needed)
#        content     : scrubbed message text
#        metadata.speaker     : "user" | "assistant" | "tool"
#        metadata.source      : "interactive" | "scheduled_run" | ...
#        metadata.tool_calls  : [{name, args, result}, ...] on assistant rows
#
# Cap result strings at 600 chars and args at 1000 chars during export
# to avoid blowing the LLM-extractor's per-episode prompt budget.

# 2. Run corpus_scrub over the raw export.
python -m symfonic.tools.corpus_scrub \
    --input  raw_extract.jsonl \
    --output scrubbed.jsonl.gz \
    --manifest manifest.json

# 3. Validate token coverage.
jq '.match_counts' manifest.json
# {
#   "EMAIL": 412,
#   "URL": 88,
#   "SELLER_ID": 156,
#   "ORDER_ID": 41,
#   "TOKEN": 0   <-- if non-zero, audit your export pipeline; OAuth
#                    tokens should never have left the credential store.
# }

# 4. Ship scrubbed.jsonl.gz + manifest.json + anchors.json
#    (a small spot-label sidecar; see Guide 10 v7.9.0 eval).

Customising the scrubber

For tenant-specific patterns (a brand's product line, a domain's internal entity names), construct the scrubber with extra patterns:

import re
from symfonic.tools.corpus_scrub import (
    CorpusScrubber,
    DEFAULT_VALUE_CREDENTIAL_PATTERNS,
    DEFAULT_PII_PATTERNS,
    AMAZON_KEEP_PATTERNS,
    AMAZON_REDACT_PATTERNS,
)

CUSTOM_PII = (
    ("CUSTOMER_ID", re.compile(r"\bCUST-\d{8}\b")),
    ("INTERNAL_SKU", re.compile(r"\bSKU-[A-Z]{3}-\d{6}\b")),
)

scrubber = CorpusScrubber(
    value_credential_patterns=DEFAULT_VALUE_CREDENTIAL_PATTERNS,
    pii_patterns=(*DEFAULT_PII_PATTERNS, *CUSTOM_PII),
    amazon_keep_patterns=AMAZON_KEEP_PATTERNS,
    amazon_redact_patterns=AMAZON_REDACT_PATTERNS,
)

What the scrubber does NOT do

  • Does not scan key names. Use symfonic.agent.hygiene.compile_credential_pattern to drop dict keys whose names look like credentials (password, api_key).
  • Does not normalise schema. Your transform owns shape; the scrubber redacts content within a fixed shape.
  • Does not enforce Anthropic prompt-cache constraints. Truncate tool_calls[].args (~1000 chars) and tool_calls[].result (~600 chars) during export, not in the scrubber.
  • Does not provide statistical PII detection. The conservative pattern set will leak truly novel PII shapes. The drop-on-doubt threshold is the backstop; raise it (--unresolved-threshold=5) if you are seeing too many drops or lower it (1) for paranoid exports.

Round-trip with the eval harness

The eval harness lives at tests/integration/learning/test_llm_procedural_extractor_corpus.py and is gated on SYMFONIC_EXTRACTOR_MODEL. Once you have the scrubbed corpus, the symfonic-side replay is:

SYMFONIC_EXTRACTOR_MODEL=claude-haiku-4-5 \
SYMFONIC_CORPUS_PATH=./scrubbed.jsonl.gz \
pytest tests/integration/learning/test_llm_procedural_extractor_corpus.py \
    --max-batches=0

--max-batches=0 runs the full corpus (one extractor invocation per 100 episodes — about 248 calls for a 24k-turn corpus, ~$12.65 at Haiku 4.5 list rates). CI runs at --max-batches=10 for the structural smoke.

References