Skip to content

CORE-199 chunked content erasure rollout

A native PostgreSQL subtree forget erases at most 1,024 rows per table in one transaction. Above that it refuses with MemoryContractError("native subtree budget exceeded: stored rows") and changes nothing. scripts/migrations/core199_erase_content.sql lets the composed PostgresNativeEraser erase such a subtree in chunks instead. This page is the operator procedure. It is not a deployment record.

What changes once the script ran

  • The forget records a durable erase job and erases nothing else in that transaction. From that commit on, every native read hides the subtree, in the same statement that reads the rows, and every native write under it refuses with MemoryContractError("native subtree erase in progress"). The erase is all or nothing to every reader: none of the subtree is ever returned once the job exists, whether its rows are erased yet or not.
  • Each later forget of the same scope returns ForgetInProgress and erases content chunks of at most 1,024 rows, each its own transaction, for about one second. Edges go first, then workset progress, then nodes, then the claim history. Only the call that empties the range returns a LifecycleReceipt. It lists no ids (discarded is empty), and the database keeps only a keyed digest of the erased prefix.
  • Chunks take no table lock beyond ROW EXCLUSIVE, so writers of other tenants do not wait on them. Only the job's first transaction holds the EXCLUSIVE graph lock, within the forget's 1 s deadline.
  • Rows held by a job still count toward the read budgets. A query_subtree whose range still stores more than 1,024 rows, held ones included, refuses as it refused before the forget, until chunks bring it under the budget. Recall drops held owners before it reads any row, so held rows do not count toward its 4,096-row budget.
  • Rows whose key lies inside the subtree's key range but is not a valid descendant key stay, exactly as an inline forget leaves them. No native read returns such a row.

The forget must still be graph-only: a store composing vectors or a recall port refuses a large forget, as it did before.

Deploy order

  1. Ship the release that hides erasing subtrees from every read and fences graph updates and deletes to every runtime, writer and eraser host. An older runtime would read a partly erased subtree. No code can prove this fleet barrier; it is an operator step.
  2. Run the script, after core199_erase_jobs.sql and core199_erase_chunks.sql:
psql "$POSTGRES_DSN" -X -v scope_schema=public \
  -f scripts/migrations/core199_erase_content.sql

Do not pass -1 or wrap it in a transaction: the two keyset indexes are built CONCURRENTLY. A failed build leaves an INVALID index, which the script refuses and never drops. Recover with DROP INDEX CONCURRENTLY and run it again. Running it again on a good install changes nothing. 3. As the table owner, re-run postgres_eraser_roles.install(conn, eraser=<login>). It grants the eraser UPDATE on the three new job columns. Until then the eraser's startup check refuses.

What the script adds

  • memory_erase_jobs.phase (edges, progress, nodes or claims; existing jobs read claims), content_scope and content_key.
  • The trigger memory_erase_jobs_phase_guard, enabled ALWAYS. While a job's phase is not claims it refuses deleting the job and changing its claim cursor, and no phase can move back. An eraser older than this script only knows those two writes, so on a content job it fails closed instead of lifting the fence over content still stored.
  • idx_nodes_erase_keyset_c and idx_edges_erase_keyset_c on (tenant_id, scope_path, id), all "C". Each chunk resumes from its cursor with one seek; progress uses its primary key.

Abandoned jobs

Nothing advances a job in the background. Until the same scope is forgotten again or an operator resumes it, its subtree reads as erased, writes under it refuse, a flush or discard selecting its pending rows refuses, ancestor range reads over more than 1,024 rows refuse, and any overlapping forget (a tenant-wide one included) refuses as overlapping. Watch for jobs left erasing:

SELECT id, phase, created_at FROM memory_erase_jobs
WHERE state = 'erasing' AND created_at < now() - interval '15 minutes'
ORDER BY created_at;

Resume each through the host's composed eraser (same login, schema and receipt key), repeating until it reports completion:

step = await eraser.resume(job_id)  # chunks for up to CONTENT_SECONDS
while not step.complete:
    step = await eraser.resume(job_id)

Each chunk is its own transaction; a failed one rolls back and the next call resumes from the persisted cursor.

Rollback

Drop nothing while any job has a phase other than claims:

SELECT id, phase FROM memory_erase_jobs WHERE phase <> 'claims';

With no such job, the trigger, its function, the three columns and the two indexes may be dropped. An oversized content forget then refuses again before it changes anything.

Evidence

  • tests/memory/integration/test_core199_chunked_content_erase_live.py: erasure to zero with a digest receipt and no ids; every read and a public Agent turn see nothing of the subtree between chunks; writes under it refuse; a pre-script eraser fails closed; without the script the forget refuses unchanged; a failed chunk rolls back and the next forget resumes; recall below a held owner neither reads nor counts its rows.
  • tests/memory/integration/test_core199_held_content_flush_live.py: a vector-composed flush at, below or above an erasing owner refuses before its embedder or in-memory vector store sees held content, and the selecting statement returns no held label, properties or embedding; discard refuses; a flush outside the job and one after completion publish normally. Private scope: the test overrides the store's native composition check and mints the turn admission, because a public vector-composed native store refuses before this selection.
  • tests/memory/integration/test_core199_erase_content_migration_live.py: the script against the module DDL, re-runs, a wrong-shaped index, and the eraser re-install.
  • tests/memory/integration/test_core199_content_keyset_plans_live.py: each chunk is one ordered keyset seek in custom and generic plans.
  • tests/architecture/test_core199_erase_hiding_inventory.py and test_core199_erase_fence_inventory.py: every reader of erasable content hides held rows or is classified with its reason, and every graph update and delete is fenced.