Skip to content

symfonic.tools.corpus_scrub.jsonl_io

jsonl_io

Streaming a JSONL corpus through :class:CorpusScrubber.

The file-level half of corpus scrub: open (optionally gzipped) input, scrub row by row, write (optionally gzipped) output, and return the counts that become the manifest sidecar. It is separated from :mod:symfonic.tools.corpus_scrub.scrubber -- which owns the redaction engine and the instance-scoped token mapping -- because nothing here knows a pattern from a placeholder; it knows files, lines, and totals.

scrub_jsonl_file 

scrub_jsonl_file(*, input_path: Path, output_path: Path, scrubber: CorpusScrubber | None = None, drop_on_doubt: bool = True) -> dict[str, Any]

Stream a JSONL file through :class:CorpusScrubber.

Input may be plain .jsonl or gzipped .jsonl.gz. Output extension determines compression (.gz -> gzip). Per architect spec, the canonical output shape is .jsonl.gz so the file is transferable without further compression.

Returns a summary dict

{ "input_rows": int, "output_rows": int, "dropped_rows": int, "match_counts": dict[str, int], }

The summary is also the natural shape for a manifest sidecar.

Source code in src/symfonic/tools/corpus_scrub/jsonl_io.py 
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def scrub_jsonl_file(
    *,
    input_path: Path,
    output_path: Path,
    scrubber: CorpusScrubber | None = None,
    drop_on_doubt: bool = True,
) -> dict[str, Any]:
    """Stream a JSONL file through :class:`CorpusScrubber`.

    Input may be plain ``.jsonl`` or gzipped ``.jsonl.gz``.  Output
    extension determines compression (``.gz`` -> gzip).  Per architect
    spec, the canonical output shape is ``.jsonl.gz`` so the file is
    transferable without further compression.

    Returns a summary dict:
        ``{
            "input_rows": int,
            "output_rows": int,
            "dropped_rows": int,
            "match_counts": dict[str, int],
        }``

    The summary is also the natural shape for a manifest sidecar.
    """
    if scrubber is None:
        scrubber = CorpusScrubber()

    input_open = gzip.open if input_path.suffix == ".gz" else open
    output_open = gzip.open if output_path.suffix == ".gz" else open

    input_rows = 0
    output_rows = 0
    dropped_rows = 0
    total_counts: dict[str, int] = {}

    with input_open(input_path, "rt", encoding="utf-8") as f_in, \
            output_open(output_path, "wt", encoding="utf-8") as f_out:
        for line in f_in:
            line = line.strip()
            if not line:
                continue
            input_rows += 1
            row = json.loads(line)
            result = scrubber.scrub_row(row, drop_on_doubt=drop_on_doubt)
            for k, v in result.match_counts.items():
                total_counts[k] = total_counts.get(k, 0) + v
            if result.scrubbed is None:
                dropped_rows += 1
                continue
            f_out.write(json.dumps(result.scrubbed, ensure_ascii=False))
            f_out.write("\n")
            output_rows += 1

    return {
        "input_rows": input_rows,
        "output_rows": output_rows,
        "dropped_rows": dropped_rows,
        "match_counts": total_counts,
    }