Operator-installed index pair holding exactly the rows a candidate read can rank.
The partial predicate drops pending, retracted and unreviewed procedural rows;
the key separates the owner path, the layer, the audience (empty for rows every
session reads, else the SHA-256 of the row's own session string, so an id of
any length fits a btree tuple) and the profile slot, then orders by importance
rank and id. One equality seek per requested (path, layer, audience, profile)
reads that combination's rows, best first. The digest only narrows a seek: the
exact session check runs inside it, before its LIMIT, so a digest collision
neither returns nor counts another session's row, and the row budget counts
nothing a caller could not read. The TTL barrier is a function of another
table and stays a filter on that seek too.
That key holds tenant_id, scope_path, layer and id whole, and
none of them is length-bounded on write. Rows whose four values exceed
KEY_BYTES therefore sit in a companion index instead, keyed by a fixed
32-byte owner digest and the audience, so no write gains a size limit. Every
read also seeks that companion per (path, layer, audience), with the same
exact checks and no rank order: it reads every wide row of the combination up
to the budget, and each counts. Without both indexes, or with either of
another shape, reads keep the visible-row budget; installing them is additive
and invisible to older binaries.
audience
Index key of a row's readers; a non-string session key is NULL, never sought.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def audience() -> str:
"""Index key of a row's readers; a non-string session key is NULL, never sought."""
session = rows.session()
text = f"({session} #>> '{{}}')"
return (f"(CASE WHEN {rows.shared()} THEN {SHARED} "
f"WHEN jsonb_typeof({session})='string' THEN {session_digest(text)} END)")
|
eligible_index_ready
async
eligible_index_ready(conn: Any) -> bool
True only for both valid btrees rendered exactly as the DDL above, on PG16 8 KiB pages.
The renderings are too long to pin verbatim, so their SHA-256 are pinned;
any edit to an eligibility fragment changes them and retires older indexes.
KEY_BYTES is derived for the 8 KiB page's btree tuple limit. An index
pair can survive pg_upgrade to another major, where it is uncertified.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| async def eligible_index_ready(conn: Any) -> bool:
"""True only for both valid btrees rendered exactly as the DDL above, on PG16 8 KiB pages.
The renderings are too long to pin verbatim, so their SHA-256 are pinned;
any edit to an eligibility fragment changes them and retires older indexes.
``KEY_BYTES`` is derived for the 8 KiB page's btree tuple limit. An index
pair can survive ``pg_upgrade`` to another major, where it is uncertified.
"""
ready = await conn.fetchval(
"SELECT count(*)=2 AND bool_and(i.indisvalid AND i.indisready AND i.indislive "
"AND i.indrelid='memory_nodes'::regclass AND NOT i.indisunique "
"AND i.indnkeyatts=v.atts AND i.indnatts=v.atts AND encode(sha256(convert_to("
"regexp_replace(pg_get_indexdef(i.indexrelid),' ON [^ ]+ USING btree ',"
"' ON memory_nodes USING btree '),'UTF8')),'hex')=v.sha) "
"AND current_setting('block_size')='8192' "
"AND current_setting('server_version_num')::integer/10000=16 "
"FROM (VALUES ($1,$2,7),($3,$4,2)) v(name,sha,atts) "
"JOIN pg_index i ON i.indexrelid=to_regclass(v.name)",
ELIGIBLE_INDEX, ELIGIBLE_INDEX_SHA256, WIDE_INDEX, WIDE_INDEX_SHA256,
)
return ready is True
|
index_ddl
The eligible index DDL, built from the current eligibility and digest fragments.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def index_ddl() -> str:
"""The eligible index DDL, built from the current eligibility and digest fragments."""
return (
f"CREATE INDEX {ELIGIBLE_INDEX} ON memory_nodes "
f'(tenant_id COLLATE "C", scope_path COLLATE "C", layer, {audience()}, '
f'{PROFILE_KEY}, {rows.RANK} DESC, id COLLATE "C") WHERE {PREDICATE} AND {FITS}'
)
|
owner_digest
owner_digest(tenant_sql: str, path_sql: str, layer_sql: str) -> str
SHA-256 of tenant, path and layer bytes; text holds no NUL, so the split is unique.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def owner_digest(tenant_sql: str, path_sql: str, layer_sql: str) -> str:
"""SHA-256 of tenant, path and layer bytes; text holds no NUL, so the split is unique."""
parts = f"||{_SEPARATOR}||".join(_utf8(part) for part in (tenant_sql, path_sql, layer_sql))
return f"sha256({parts})"
|
seek_source
seek_source(tenant: int, paths: int, layers: int, cap: int, expiry: str, whole: int, paths_sql: str | None = None) -> str
Per-combination ordered seeks of at most cap rows, then the wide seeks.
The OFFSET-free per-row LIMIT keeps each subquery parameterized by its
combination in custom and generic plans alike, so no scan reads an owner,
layer or audience the request did not name. The session equality is the
ranked statement's own and precedes the LIMIT, so a row sharing only the
caller's digest is filtered before it can fill or count against the seek.
A wide seek has no rank order, so it reads up to whole rows: past the
budget the read refuses, under it every wide row is ranked. paths_sql
replaces the owner array, as when erasing owners leave it (CORE-199).
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def seek_source(tenant: int, paths: int, layers: int, cap: int, expiry: str,
whole: int, paths_sql: str | None = None) -> str:
"""Per-combination ordered seeks of at most ``cap`` rows, then the wide seeks.
The OFFSET-free per-row LIMIT keeps each subquery parameterized by its
combination in custom and generic plans alike, so no scan reads an owner,
layer or audience the request did not name. The session equality is the
ranked statement's own and precedes the LIMIT, so a row sharing only the
caller's digest is filtered before it can fill or count against the seek.
A wide seek has no rank order, so it reads up to ``whole`` rows: past the
budget the read refuses, under it every wide row is ranked. ``paths_sql``
replaces the owner array, as when erasing owners leave it (CORE-199).
"""
owned = (f"AND {PREDICATE} AND {expiry} "
f"AND ({rows.shared()} OR {rows.session()}=to_jsonb($3::text))")
return f"""combos AS (
SELECT c_path, c_layer, c_audience, c_profile
FROM unnest({paths_sql or f"${paths}::text[]"}) c_path,
unnest(${layers}::text[]) c_layer,
(VALUES ({SHARED}),({session_digest("$3::text")})) a(c_audience),
(VALUES (TRUE),(FALSE)) f(c_profile)
), seeks AS (
SELECT n.* FROM combos CROSS JOIN LATERAL (
SELECT * FROM memory_nodes WHERE tenant_id COLLATE "C"=${tenant}
AND scope_path COLLATE "C"=c_path AND layer=c_layer
AND {audience()}=c_audience AND {PROFILE_KEY}=c_profile AND {FITS} {owned}
ORDER BY {rows.RANK} DESC, id COLLATE "C"
LIMIT ${cap}::bigint) n
UNION ALL
SELECT w.* FROM (SELECT DISTINCT c_path, c_layer, c_audience FROM combos) o
CROSS JOIN LATERAL (
SELECT * FROM memory_nodes
WHERE {owner_digest("tenant_id", "scope_path", "layer")}
={owner_digest(f"${tenant}::text", "c_path", "c_layer")}
AND {audience()}=c_audience AND tenant_id COLLATE "C"=${tenant}
AND scope_path COLLATE "C"=c_path AND layer=c_layer AND NOT {FITS} {owned}
LIMIT {int(whole)}) w
),"""
|
session_digest
session_digest(text_sql: str) -> str
SHA-256 of a text's UTF-8 bytes.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def session_digest(text_sql: str) -> str:
"""SHA-256 of a text's UTF-8 bytes."""
return f"sha256({_utf8(text_sql)})"
|
wide_index_ddl
The companion DDL: every eligible row the eligible index cannot key.
Source code in src/symfonic/memory/backends/postgres_candidate_index.py
| def wide_index_ddl() -> str:
"""The companion DDL: every eligible row the eligible index cannot key."""
owner = owner_digest("tenant_id", "scope_path", "layer")
return (f"CREATE INDEX {WIDE_INDEX} ON memory_nodes ({owner}, {audience()}) "
f"WHERE {PREDICATE} AND NOT {FITS}")
|