symfonic.services.shadow¶
shadow ¶
T2.3.7 — governed shadow execution and record/replay comparison.
Reading order, because the pieces only make sense as a chain:
effects / classification
The exhaustive effect-port table. Fourteen families, one disposition each
(deny or deterministically stub), and a fail-closed refusal for anything
unclassified.
trust
The second axis. Ports are not the whole story — an in-process tool can
open a socket — so every tool, plugin, and contributed stage is
port-mediated (provably, reviewed, by construction) or opaque (the
default), and opaque means non-shadowable and non-replayable.
sentinel
What makes a port-mediated claim falsifiable: direct-effect probes that
turn a mis-declaration into a loud abort instead of a silent leak.
gateway / state / context / harness
The effect-suppressed run itself, its suppressed replacement state writes,
the single surface the run body can reach, and the ledger a reviewer reads
to believe the suppression claim.
redaction / crypto / admission / recorder / store / privacy
Capture, governed by threat-model §6: eligibility, allowlist and
classification-aware redaction, minimization and sampling, encryption at
rest, tenant scoping, access audit, retention, and the erasure contract
wired into the legacy privacy-deletion path.
replay / comparator / cutover
Replay through the same harness, a comparator that refuses unsafe inputs,
and the recorder that files each capability's cutover evidence under the
one path it is entitled to use.
AccessAuditEntry
dataclass
¶
AccessAuditEntry(at: datetime, actor: str, action: str, recording_id: str, tenant_id: str, allowed: bool, reason: str = '')
Append-only: every read attempt, allowed or not.
AccessGrant
dataclass
¶
Who may read recordings, for which tenants, and why.
AllowedField
dataclass
¶
AllowedField(path: str, data_class: DataClass, redaction: Redaction = Redaction.NONE, keep_chars: int = 64)
One dotted path that may be captured, and how it must be transformed.
CaptureAdmissionPolicy ¶
CaptureAdmissionPolicy(*, grants: Mapping[str, TenantCaptureGrant] | None = None, allowlist: FieldAllowlist | None = None, cipher: EncryptionPort | None = None, minimizer: PayloadMinimizer | None = None, response_allowlist: FieldAllowlist | None = None, clock: Any = None)
Decides whether one invocation may be recorded, and in what shape.
Source code in src/symfonic/services/shadow/admission.py
project_response ¶
Govern a recorded port answer, not just the invocation payload.
Provider completions and tool results are the largest tenant-data surface in a recording and a common carrier of integration credentials, so they get the same treatment as the payload: credential-shaped keys dropped at every depth, then minimization.
Field allowlisting is opt-in here and configured separately from the
payload allowlist, because a recorded answer is also the replay's
stubbed answer: projecting every response through the payload
allowlist would silently change what a replay can serve. A capability
that records real tenant traffic configures response_allowlist;
without one the answer is scrubbed and minimized but not projected,
and the decision says so.
Source code in src/symfonic/services/shadow/admission.py
CaptureDecision
dataclass
¶
CaptureDecision(admitted: bool, reason: str, request: CaptureRequest, payload: Mapping[str, Any] = dict(), dropped_fields: tuple[str, ...] = (), redacted_fields: tuple[str, ...] = (), grant: TenantCaptureGrant | None = None)
Admitted with a minimized, redacted payload — or refused with a reason.
CaptureOutcome
dataclass
¶
Admitted (with a session) or refused (with a reason). Never both.
CaptureRefusedError ¶
Bases: ShadowError
Safe capture could not be established; the invocation is unrecorded.
Recording fails closed. Callers catch this and continue serving the tenant's request — never the other way round.
CaptureRequest
dataclass
¶
CaptureRequest(invocation_id: str, tenant_id: str, payload: Mapping[str, Any] = dict(), kind: str = 'invocation', at: datetime | None = None)
One invocation offered to the recorder.
ComparisonUnsafeError ¶
Bases: ShadowError
TM-29d — this comparison would duplicate an externally visible effect.
ConstructionProof
dataclass
¶
ConstructionProof(method: ProofMethod, verified_by: str, injected_ports: frozenset[str] = frozenset(), audited_module: str = '', self_declared: bool = False)
Why this extension's effects cannot leave the classified ports.
CutoverCriteriaRecorder ¶
Files cutover evidence and enforces which path a capability may use.
Source code in src/symfonic/services/shadow/cutover.py
CutoverEvidence
dataclass
¶
CutoverEvidence(capability: str, path: CutoverPath, recorded_at: datetime, criteria: dict[str, Any] = dict(), opaque_dependencies: tuple[str, ...] = (), comparison: ComparisonReport | None = None)
One capability's cutover evidence, filed under exactly one path.
CutoverPathError ¶
Bases: ConfigurationError
Cutover evidence was filed under a path the capability may not use.
DataClass ¶
Bases: StrEnum
Threat-model §6.2 classes, in ascending order of "must not capture".
DeletionWiringAttestation
dataclass
¶
DeletionWiringAttestation(participant_id: str, registry_id: str, verified_at: datetime, probe_tenant: str, seeded: int, residual: int)
Evidence that erasure actually reaches this participant.
DeterministicStub ¶
A pure function of (port, operation, payload). No state, no clock.
Determinism is the requirement, not realism: a comparator can only call a divergence real if re-running the same input twice gives the same answer.
DirectEffectSentinel
dataclass
¶
DirectEffectSentinel(subject: str = '<extension>', block: bool = True, witnesses: list[DirectEffectWitness] = list())
Records — and by default blocks — direct effects during a call.
block=True (the default, and what the harness uses) makes the probe
raise, so a mis-declared extension does not get to perform the effect it
was not supposed to be able to perform. block=False is observe-only:
the witness is recorded and the call is passed through to the real entry
point, which is what an audit of a candidate declaration wants — it
learns what the extension actually touches without changing its behaviour.
Either way the witness list is checked afterwards by assert_clean, so
an extension that swallows the exception, or one that was merely observed,
is still detected and still loses its declaration.
assert_clean ¶
Raise if anything was witnessed, even if the extension caught it.
Source code in src/symfonic/services/shadow/sentinel.py
watching ¶
Install the probes for the duration of one extension call.
The probes themselves are process-global and reference-counted, so
overlapping windows install once and restore once; ownership of any
effect they see is decided by the context variable, which is scoped to
this with block and therefore to this task.
Source code in src/symfonic/services/shadow/sentinel.py
DirectEffectWitness
dataclass
¶
One observed effect that did not cross a framework port.
EffectAttempt
dataclass
¶
EffectAttempt(seq: int, port_id: str, operation: str, request_digest: str, outcome: EffectOutcome, family: EffectFamily | None = None, externally_visible: bool = True, detail: str = '')
One attempted crossing of a framework effect port.
EffectFamily ¶
Bases: StrEnum
Every effect family the architecture declares. Exhaustive by contract.
EffectLedger
dataclass
¶
EffectOutcome ¶
Bases: StrEnum
How a single attempted effect resolved.
EffectPort
dataclass
¶
EffectPort(port_id: str, family: EffectFamily, disposition: ShadowDisposition, rationale: str, externally_visible: bool = True)
One classification row: a port, its family, and its shadow disposition.
rationale is required. A disposition with no recorded reason is a
decision nobody can review later, and this table is cutover evidence.
EffectPortClassification
dataclass
¶
An immutable table of port_id -> disposition.
Immutable because a classification that could be widened at runtime would let a shadow run mint its own permission halfway through, which is exactly what the fail-closed rule is protecting against.
assert_classifies ¶
Every named port has a row. Complements :meth:assert_exhaustive.
Family coverage proves the taxonomy is complete; it says nothing about whether the ports the framework actually declares are in the table. A port the runtime crosses but the table has never heard of is not "fail-closed" — it is simply never observed, because nothing routes it through the gateway. This is the check that names them.
Source code in src/symfonic/services/shadow/classification.py
assert_exhaustive ¶
Every declared family has at least one classified port.
Source code in src/symfonic/services/shadow/classification.py
classify ¶
The classification row, or a fail-closed refusal. Never a default.
Source code in src/symfonic/services/shadow/classification.py
extended_with ¶
A new table with extra rows. Never mutates the receiver.
EncryptionPort ¶
Bases: Protocol
Seal and unseal recording bytes.
ErasureParticipantRegistry ¶
The set of stores the tenant-erasure path must sweep.
Source code in src/symfonic/services/shadow/privacy.py
erase_all
async
¶
Sweep every participant. Returns participant_id -> rows removed.
sweep
async
¶
Erase from every participant, surviving one that fails.
A participant is somebody else's store — a remote backend having a bad minute must not abort the tenant's erasure, skip every participant sorted after it, and lose the completion trail. Each failure is logged and reported by id instead.
Source code in src/symfonic/services/shadow/privacy.py
ErasureSweep
dataclass
¶
The outcome of one sweep: what was erased, and what refused to be.
A sweep is deliberately not all-or-nothing. Erasure is the one operation where a partial success must still be reported rather than rolled back — the rows that went are gone — so a participant that raises is recorded by id and the sweep continues to the rest.
ExtensionRecord
dataclass
¶
ExtensionRecord(extension_id: str, kind: ExtensionKind, origin: ExtensionOrigin, trust: TrustClass, reason: str, proof: ConstructionProof | None = None, approval: ReviewerApproval | None = None)
What the registry knows about one extension.
ExtensionTrustRegistry ¶
Assigns and enforces trust classes. Default-deny by construction.
Source code in src/symfonic/services/shadow/trust.py
classify_port_mediated ¶
classify_port_mediated(extension_id: str, kind: ExtensionKind, *, proof: ConstructionProof, approval: ReviewerApproval, origin: ExtensionOrigin = ExtensionOrigin.FRAMEWORK) -> ExtensionRecord
Promote to port-mediated. Refuses everything short of the bar.
Source code in src/symfonic/services/shadow/trust.py
demote ¶
Force an extension to opaque and bar re-promotion under this id.
Source code in src/symfonic/services/shadow/trust.py
record_of ¶
The record, or a synthesized opaque one. Never raises for unknown.
Source code in src/symfonic/services/shadow/trust.py
register ¶
register(extension_id: str, kind: ExtensionKind, *, origin: ExtensionOrigin = ExtensionOrigin.ADOPTER, reason: str = 'registered without a construction proof') -> ExtensionRecord
Register an extension as opaque. This is the only bulk entry point.
Source code in src/symfonic/services/shadow/trust.py
FieldAllowlist ¶
Allowlist-first projection of a payload.
Source code in src/symfonic/services/shadow/redaction.py
HmacStreamCipher ¶
HMAC-SHA256 CTR keystream with encrypt-then-MAC.
Encryption and authentication use separately derived subkeys, and the MAC covers key id, nonce, and ciphertext, so neither a swapped nonce nor a relabelled key can be passed off as a valid record.
Source code in src/symfonic/services/shadow/crypto.py
IncompleteClassificationError ¶
Bases: ConfigurationError
An effect family the architecture declares has no classified port.
MisdeclaredExtensionError ¶
Bases: ShadowAbortedError
An extension declared port-mediated performed a direct effect.
The declaration was wrong; every suppression claim that depended on it is withheld, and the extension is demoted to opaque.
OpaqueExtensionError ¶
Bases: ShadowAbortedError
An opaque tool, plugin, or contributed stage is non-shadowable.
Raised before the extension runs. Pretending an opaque extension's effects were suppressed is the failure mode this type exists to prevent.
PayloadMinimizer
dataclass
¶
Payload minimization: cap string length and collection width.
ProofMethod ¶
Bases: StrEnum
The only two ways an extension can be port-mediated by construction.
RecordComparator ¶
Compares a recording against an effect-suppressed candidate run.
RecordedEvent
dataclass
¶
RecordedEvent(seq: int, port_id: str, operation: str, request_digest: str, response: Any = None, family: str = '')
One non-idempotent port crossing, with the answer it produced.
RecordedResponder ¶
Answers a stubbed port call from the recording, in recorded order.
Answers for one key are consumed FIFO rather than looked up, because the recorded ports are non-idempotent by definition: the same request made twice may have produced two different answers, and replaying the second answer to the first call is a divergence the comparator cannot catch. Once a key's recorded answers are exhausted the responder refuses, so a replay that calls a port more times than the original is a refusal — never a silently repeated answer.
Source code in src/symfonic/services/shadow/replay.py
unserved ¶
Every recorded answer the replay never asked for, repeats included.
Source code in src/symfonic/services/shadow/replay.py
Recorder ¶
Recorder(*, policy: CaptureAdmissionPolicy, store: RecordingStore, actor: str, mode: RecordingMode = RecordingMode.SYNTHETIC, clock: Callable[[], datetime] | None = None)
Binds the admission policy to the governed store.
Source code in src/symfonic/services/shadow/recorder.py
begin ¶
Admit or refuse. A refusal means the invocation proceeds unrecorded.
Source code in src/symfonic/services/shadow/recorder.py
commit ¶
Seal and store. Any capture failure leaves the invocation unrecorded.
The refusal is deliberately broad. Serialization is the obvious way a
commit fails for a reason the store never sees — an allowlisted leaf
holding a datetime makes json.dumps raise TypeError, not
RecordingStoreError — and a cipher or backend can fail its own way
too. All of them are capture failures, and the contract is that
capture failure never reaches the invocation being served. The double
commit above still raises: that is a caller bug, not a capture failure.
Source code in src/symfonic/services/shadow/recorder.py
Recording
dataclass
¶
Recording(recording_id: str, tenant_id: str, mode: RecordingMode, captured_at: datetime, events: tuple[RecordedEvent, ...] = (), extensions: tuple[str, ...] = (), payload: Mapping[str, Any] = dict(), metadata: Mapping[str, Any] = dict())
A tenant-scoped, redacted trace of one invocation.
from_bytes
classmethod
¶
Data-only decode. json.loads instantiates nothing but builtins.
index ¶
(port, operation, request_digest) -> last response — lossy.
Kept for callers that only need to know whether a key was recorded.
Replay uses :meth:response_queues, which does not collapse repeats.
Source code in src/symfonic/services/shadow/recording.py
response_queues ¶
(port, operation, request_digest) -> every answer, in order.
A list, not a single value, because the ports a recording exists to stub are the non-idempotent ones: a tool called twice with the same arguments legitimately answers differently the second time. Collapsing those into one entry would serve the last answer to both calls and steer the replacement down a path the original never took — the exact duplication-of-effect the comparator cannot see, since it diffs requests rather than responses.
Source code in src/symfonic/services/shadow/recording.py
RecordingAccessError ¶
Bases: ShadowError
TM-29a — an actor read (or tried to read) beyond its authorization.
RecordingMode ¶
Bases: StrEnum
Whether a recording holds real tenant traffic or fabricated traffic.
The distinction is load-bearing: SEC-PRIV-5 gates PRODUCTION behind
verified privacy-deletion wiring, while SYNTHETIC fixtures must stay
usable in CI on day one.
RecordingSession
dataclass
¶
RecordingSession(recording_id: str, tenant_id: str, mode: RecordingMode, decision: CaptureDecision, captured_at: datetime, events: list[RecordedEvent] = list(), extensions: list[str] = list(), committed: bool = False, project_response: Callable[[Any], ResponseProjection] | None = None, response_dropped: list[str] = list(), response_redacted: list[str] = list())
Accumulates events for one invocation, then commits once.
A session is normally built by :class:Recorder, which binds the
admission policy's response projection to it. A session built without one
still governs its answers — the fallback scrubs credential-shaped keys and
minimizes — because "nobody wired a projector" must not mean "write the
provider's answer verbatim".
observe ¶
observe(port_id: str, operation: str, request: Any, response: Any, *, family: str = '') -> RecordedEvent
Record one non-idempotent port crossing and its governed answer.
The request never enters the recording — only its digest — and the answer goes through the same governance as the payload before any of it is serialized.
Source code in src/symfonic/services/shadow/recorder.py
RecordingStore ¶
RecordingStore(*, cipher: EncryptionPort | None = None, retention: timedelta = timedelta(days=7), grants: tuple[AccessGrant, ...] = (), deletion_wiring: DeletionWiringAttestation | None = None, clock: Callable[[], datetime] | None = None)
An in-memory governed store. The seams are the cipher and the clock.
Source code in src/symfonic/services/shadow/store.py
attest_deletion_wiring ¶
Accept an attestation. Refuses one that does not prove erasure.
Source code in src/symfonic/services/shadow/store.py
RecordingStoreError ¶
Bases: ShadowError
The recording store refused a write it cannot govern.
RefusingCipher ¶
The absence of a configured cipher, made explicit and loud.
ReplayRunner ¶
Replays one recording through the shadow harness.
Source code in src/symfonic/services/shadow/replay.py
replayable
staticmethod
¶
A replay is usable as comparison input only when nothing leaked.
ResponseProjection
dataclass
¶
ResponseProjection(value: Any, dropped_fields: tuple[str, ...] = (), redacted_fields: tuple[str, ...] = ())
What survives of one recorded port answer, and what was removed.
ReviewerApproval
dataclass
¶
A named human from the owning capability signing off on a promotion.
SealedPayload
dataclass
¶
Ciphertext plus everything needed to verify and open it. No plaintext.
ShadowAbortedError ¶
Bases: ShadowError
The shadow run stopped before (or instead of) performing an effect.
Raised on its own when a run is aborted for a reason that has no more specific type; the fail-closed subclasses below are the usual cause.
ShadowContext ¶
ShadowContext(*, gateway: ShadowEffectGateway, state: SuppressedStateWriter, trust: ExtensionTrustRegistry, watch: bool)
What the body of a shadow run is given. Nothing else is reachable.
Source code in src/symfonic/services/shadow/context.py
claim_defects
property
¶
Reasons the suppression claim is void, independent of exceptions.
Detection must not depend on the exception reaching the harness: a
body with a bare except would otherwise buy back the claim the
sentinel just refused. Every detection path records here first and
raises second.
call_extension ¶
Run a synchronous extension only if its trust class permits it.
Opaque extensions raise before fn is touched. Port-mediated ones
run inside the sentinel, so a wrong declaration becomes a
MisdeclaredExtensionError instead of a silent effect. An extension
that returns an awaitable is refused: its body would run after the
probes came down. Use :meth:call_extension_async for those.
Source code in src/symfonic/services/shadow/context.py
call_extension_async
async
¶
Await an async extension inside the sentinel scope.
In an async-first framework the common tool shape is a coroutine
function; calling one only builds the coroutine, so a synchronous
call_extension would uninstall the probes before a single line of
the extension ran. This awaits under the probes, which is the whole
point of watching.
Source code in src/symfonic/services/shadow/context.py
ShadowDisposition ¶
Bases: StrEnum
What shadow mode does with a classified port. There is no third option.
DENY refuses the call. STUB answers it deterministically without
reaching anything external. "Let it through" is deliberately not
representable — that is the whole point of classifying.
ShadowEffectDenied ¶
Bases: ShadowError
A classified DENY port refused an effect inside a shadow run.
Not an abort: denial is the port doing its job. The caller decides whether the denial is fatal to the scenario it was exercising.
ShadowEffectGateway ¶
ShadowEffectGateway(classification: EffectPortClassification, *, run_id: str, responder: StubResponder | None = None, ledger: EffectLedger | None = None)
Routes port calls through the classification; records every attempt.
Source code in src/symfonic/services/shadow/gateway.py
abort ¶
invoke ¶
Deny, stub, or abort. Never performs the effect.
Source code in src/symfonic/services/shadow/gateway.py
ShadowError ¶
Bases: SymfonicError
Root of the shadow/replay taxonomy. Never raised directly.
ShadowHarness ¶
ShadowHarness(*, trust: ExtensionTrustRegistry, classification: EffectPortClassification = DEFAULT_EFFECT_CLASSIFICATION)
Builds and supervises shadow runs.
Source code in src/symfonic/services/shadow/harness.py
ShadowRunResult
dataclass
¶
ShadowRunResult(run_id: str, tenant_id: str, status: ShadowStatus, ledger: EffectLedger, state_intents: tuple[StateWriteIntent, ...], extensions_executed: tuple[str, ...], body_executed: bool, suppression_claim: bool, claim_defects: tuple[str, ...] = (), abort_reason: str | None = None, error: BaseException | None = None, value: Any = None)
The reviewable artifact of one shadow run.
ShadowRunSpec
dataclass
¶
ShadowRunSpec(run_id: str, tenant_id: str, extensions: tuple[str, ...] = (), baseline_state: Mapping[str, Any] = dict(), backing_state: Mapping[str, Any] | None = None, responder: StubResponder | None = None, watch_extensions: bool = True)
What a shadow run is allowed to be before it starts.
StateWriteIntent
dataclass
¶
A write the replacement would have performed. Never applied.
StoredRecording
dataclass
¶
StoredRecording(recording_id: str, tenant_id: str, mode: RecordingMode, captured_at: datetime, expires_at: datetime, digest: str, sealed: SealedPayload)
What the store actually holds: metadata plus ciphertext.
StubResponder ¶
SuppressedStateWriter
dataclass
¶
SuppressedStateWriter(baseline: Mapping[str, Any] = dict(), backing: Mapping[str, Any] | None = None)
Copy-on-write overlay over a read-only baseline.
assert_backing_untouched ¶
Prove the replacement's real store is byte-identical to before.
Source code in src/symfonic/services/shadow/state.py
TenantCaptureGrant
dataclass
¶
TenantCaptureGrant(tenant_id: str, authorized_by: str, purpose: str, expires_at: datetime, sampling_rate: float = 1.0)
Per-tenant capture eligibility. Absence of a grant means "no".
TenantErasureParticipant ¶
TrustApproval
dataclass
¶
TrustApproval(extension_id: str, capability: str, reviewer: str, approved_at: str, proof_method: str)
A port-mediated assignment, its reviewer, and its capability.
TrustDeclarationError ¶
Bases: ConfigurationError
A trust-class assignment does not meet the port-mediated bar.
A ConfigurationError on purpose: a self-declared or unreviewed
port-mediated claim is a misconfiguration of the evidence pipeline, and
adopters already catch that taxonomy.
UnclassifiedEffectError ¶
Bases: ShadowAbortedError
SEC-FCP-5 — an effect was attempted through a port nobody classified.
Fail-closed by construction: an unclassified port is not "probably safe", it is a hole in the evidence, and the run that found it is void.
UndigestibleValueError ¶
Bases: ShadowError
A value's identity cannot be established, so no digest is produced.
The digest is the comparator's only notion of "same request". A value the canonicaliser cannot see inside would digest identically to every other instance of its type, and the comparator would read that collision as parity. Refusing is the fail-closed answer: an unrecorded or aborted comparison is recoverable, a false parity claim is not.
canonical_json ¶
A stable JSON rendering: sorted keys, no whitespace drift, no NaN.
digest_of ¶
A sha256 over the canonical rendering of every part, in order.
Source code in src/symfonic/services/shadow/digest.py
scrub_credential_keys ¶
Drop credential-shaped keys from any nested structure, at every depth.
The allowlist walk covers mappings, but an allowlisted leaf can still be a list of mappings (a tool result, a message array), and a recorded port answer is not walked by the allowlist at all. SEC-CRED-2 has to hold for those too, so this is the one credential check both paths call.
Source code in src/symfonic/services/shadow/redaction.py
verify_deletion_wiring
async
¶
verify_deletion_wiring(participant: TenantErasureParticipant, *, registry: ErasureParticipantRegistry = PRIVACY_DELETION_PARTICIPANTS, probe_tenant: str = 'deletion-wiring-probe', seed: int = 1) -> DeletionWiringAttestation
Run a real erase round-trip and attest to the result.
Raises when the participant is not registered — an attestation for a store the deletion path cannot reach would be worse than none at all.