Native Semantic Graph Foundation

Purple8's first RDF/LPG integration slice stores one authoritative record per asserted quad in Cortex. RDF statements and governed LPG relationship views refer to that record; there is no second, independently writable edge copy.

The foundation has a native SDK, policy-aware REST/MCP queries and governed single-quad assertion/retraction endpoints, and bounded N-Quads import. It is not a complete RDF database or a released SPARQL endpoint. It introduces no external service or new runtime Python dependency.

Create and project a statement

from purple8_graph import GraphEngine, RDFStatement, RDFTerm

with GraphEngine("./knowledge", embedding_dimension=128) as engine:
    semantic = engine.semantic_graph()
    assertion = RDFStatement(
        subject=RDFTerm.iri("urn:example:alice"),
        predicate=RDFTerm.iri("urn:example:worksFor"),
        object=RDFTerm.iri("urn:example:acme"),
        graph=RDFTerm.iri("urn:source:document-1"),
    )
    assertion_id = semantic.assert_statement(assertion)

    subject, relationship, target = semantic.project(assertion)
    assert relationship.edge_type == "urn:example:worksFor"
    assert subject.node_id == semantic.term_id(assertion.subject)

    # The LPG-side operation addresses the same canonical statement.
    same_id = semantic.add_relationship(
        assertion.subject, assertion.predicate, assertion.object,
        graph=assertion.graph,
    )
    assert assertion_id == same_id

Projection returns detached Node, Edge, Node models. Mutating those Python objects does not modify storage. Resources retain their full IRI; predicates retain their full IRI rather than being shortened into colliding edge names. Literal objects are projected as literal vertices, not coerced scalar properties. Their lexical form, datatype and language remain available in node.properties["rdf_term"].

The same resource term has the same tenant-local projection identity across graph contexts. Statement identity includes graph membership, so removing an assertion from one named graph does not remove its other graph memberships.

Term fidelity

age = RDFTerm.literal(
    "01", datatype="http://www.w3.org/2001/XMLSchema#integer",
)
english = RDFTerm.literal("bank", language="en-GB")
anonymous = RDFTerm.blank("b1", scope="document-1")

• Terms and statements are immutable, reject unknown fields and do not coerce numbers into literal strings.
• "01" and "1" remain different lexical terms, including when typed as integers. Datatype lexical spaces are not evaluated or validated in this slice; accepting an assertion does not establish that its literal is well-typed.
• Plain strings normalize to xsd:string. Language tags normalize to lowercase and imply rdf:langString. Contradictory datatype/language combinations fail.
• Blank nodes require an explicit scope. Reuse a scope only when intentionally referring to the same blank nodes; independent imports need distinct scopes.
• Blank-node identifiers currently use a documented ASCII subset: letters, digits, underscores, internal periods and hyphens, with no trailing period. Identifiers are at most 256 characters; scopes use 1-128 ASCII letters, digits, underscores or hyphens.
• IRIs must be absolute, at most 4096 UTF-8 bytes, and exclude whitespace, controls, unpaired surrogates, forbidden IRI-reference characters and malformed percent escapes. This is not a full RFC conformance validator or a network fetch operation.
• Literal values are at most 65536 UTF-8 bytes; language tags are at most 128 characters. Generalized RDF, quoted triples and directional literals are not supported.

Matching and graph selection

from purple8_graph import GraphSelection

matches = semantic.statements(subject=assertion.subject, limit=100)
default_only = semantic.statements(graph=None, limit=100)
all_contexts = semantic.statements(graph=GraphSelection.ANY, limit=100)
document_only = semantic.statements(graph=assertion.graph, limit=100)

statements() also accepts predicate= and object_term= filters. All supplied filters are combined. The limit must be an integer from 1 to 1000.

Reads select a tenant/component label index, fetch at most 128 records per page, and observe the existing cooperative query deadline/cancellation control. They do not load the whole corpus into a Python list. Complex filter intersections may still examine many candidates: there is no RDF join optimizer or latency guarantee for statements(). Use match() below for adaptive pattern joins. Results are unordered and are not snapshot-isolated against concurrent writes; repeated calls are not a pagination cursor.

get_statement(assertion_id) is a direct lookup. retract_statement(assertion) deletes exactly that quad and returns whether it existed. Repeated assertion is idempotent within the engine and preserves the existing record instead of overwriting its timestamps or ownership.

Processing: indexed pattern joins

match() evaluates a conjunction of quad patterns against the same canonical records, without materializing a second graph. Shared variables connect relationships into fixed-length paths and joins. This is an opt-in SDK execution path; it does not change existing Cypher semantics.

from purple8_graph import RDFPattern, RDFVariable, SemanticQueryLimits

person = RDFVariable(name="person")
organization = RDFVariable(name="organization")
region = RDFVariable(name="region")
patterns = [
    RDFPattern(
        subject=person,
        predicate=RDFTerm.iri("urn:example:worksFor"),
        object=organization,
        graph=RDFTerm.iri("urn:source:document-1"),
    ),
    RDFPattern(
        subject=organization,
        predicate=RDFTerm.iri("urn:example:locatedIn"),
        object=region,
        graph=RDFTerm.iri("urn:source:document-1"),
    ),
]

plan = semantic.explain(patterns)
result = semantic.match(
    patterns,
    bindings={"person": RDFTerm.iri("urn:example:alice")},
    limit=100,
    limits=SemanticQueryLimits(max_candidates=10000),
)
for row in result.matches:
    print(row.bindings["region"])
    # One supporting canonical statement ID for each input pattern, in order.
    print(row.statement_ids)
    support = semantic.get_statement(row.statement_ids[0])
    subject, relationship, target = semantic.project(support)

print(result.stats.to_dict())
print(plan.to_dict())

Query contracts:

• Supply a list of 1-16 RDFPattern, LPGNodePattern and/or LPGEdgePattern objects. Variable names use 1-64 ASCII letters, digits and underscores, starting with a letter or underscore.
• Variables bind entire RDFTerm values, not strings or scalar coercions. Repeated names enforce equality within a pattern and across patterns. Language, datatype, lexical form and blank-node scope remain part of identity.
• bindings= accepts initial RDF terms for variables used in the query. A literal bound to a subject, predicate or graph variable makes that branch unsatisfiable rather than converting it into a resource.
• A pattern's omitted graph selects only the default graph, unlike the all-context default of statements(). graph=GraphSelection.ANY ignores graph membership; graph=RDFVariable(...) binds named graphs only, including scoped blank graph names. A shared graph variable requires the same context. Separate variables or ANY explicitly permit cross-context joins.
• Results preserve every matching combination of supporting statements. Different contexts can produce identical bindings with different evidence. There is no implicit DISTINCT; one statement may satisfy multiple patterns. Disconnected patterns produce a Cartesian product, subject to the same budgets.
• The result limit is 1-1000, with no ordering guarantee. Execution stops once the limit is reached. limit_reached=True means the limit was reached, not that an additional match is known to exist. No cursor is returned.
• supports has one entry per input pattern, in caller order, with model (rdf, lpg for a node, or lpg_edge) and record_id. statement_ids retains the RDF supports in their relative input order; node_ids retains LPG node supports and edge_ids retains LPG edge supports in theirs. Execution reordering never changes evidence order. Statement IDs are tenant-local canonical IDs usable with get_statement() and project(). They identify supporting assertions, not independently verified evidence or inferred provenance.

Complementary LPG and RDF knowledge

Shared execution does not require pairing every RDF record with an LPG record. Operational entities and semantic concepts can contain different facts. An explicitly declared property interpretation can connect them without saying that the entity and concept are identical:

from purple8_graph import LPGNodePattern

# Operational fact: this supplier has an overdue payment and a classification.
engine.add_node(
    "supplier:42", labels=["Supplier"],
    properties={
        "state": "payment_overdue",
        "class_iri": "urn:class:regulated-supplier",
    },
)
# Complementary semantic fact: that classification requires enhanced review.
semantic.assert_statement(RDFStatement(
    subject=RDFTerm.iri("urn:class:regulated-supplier"),
    predicate=RDFTerm.iri("urn:requires"),
    object=RDFTerm.iri("urn:obligation:enhanced-review"),
))

classification = RDFVariable(name="classification")
result = semantic.match([
    LPGNodePattern(
        node_id="supplier:42",
        label="Supplier",
        equals={"state": "payment_overdue"},
        iri_properties={"class_iri": classification},
    ),
    RDFPattern(
        subject=classification,
        predicate=RDFTerm.iri("urn:requires"),
        object=RDFVariable(name="obligation"),
    ),
])
row = result.matches[0]
assert row.node_ids == ("supplier:42",)
assert row.bindings["obligation"] == RDFTerm.iri("urn:obligation:enhanced-review")
assert [support.model for support in row.supports] == ["lpg", "rdf"]

The supplier is not identified with its classification or obligation. iri_properties explicitly interprets selected LPG string property values as absolute RDF IRIs and compares/binds those terms. It does not interpret the node ID, convert other strings, create RDF assertions, copy the node or persist an identity mapping. Multiple LPG entities may share a classification without being merged. A query with no shared variables is a conjunction, not an inferred identity or relationship. This is exact relation-value joining, not automatic ontology reasoning.

LPG node patterns support direct, traversal-bound or label-selected, tenant-local LPG nodes, using the existing REST-compatible validation for supplied IDs. node_id=LPGNodeVariable(...) reads an ID bound by an edge pattern or by node_bindings=; when unbound, an explicit label= permits indexed label selection and binds that variable to each matching node ID. Without an explicit label or a reachable ID binding, the pattern is rejected before execution. This does not introduce a whole-store fallback, property index build or independent LPG query executor. LPG-only queries also use this same API. For an already bound ID, label= remains an optional label check, not a scan. equals= checks top-level scalar properties with exact type and value, including distinguishing True, 1 and 1.0. A present null is different from a missing property. Float constants must be finite. Together equals and iri_properties allow at most 64 conditions, with property names of 1-256 characters.

An IRI-property condition can contain a shared RDFVariable or constant IRI RDFTerm. Missing properties, missing nodes, nonmatching labels and valid but different values produce no match. A present malformed or non-string declared IRI property logs and raises SemanticPropertyError when evaluated, not a silent conversion or empty success. Non-IRI terms bound to an IRI-property variable make the branch unsatisfiable. All comparisons use the same term bindings as RDF patterns and observe the same cancellation and budget controls.

lpg_direct appears in explain output with one estimated lookup, including when the node may not exist; explain does not fetch it. Each executed node lookup charges one candidate and no index entries. Supports identify records used at execution time, not immutable LPG snapshots. Subsequent LPG property updates change subsequent query results without rewriting the RDF facts.

Label-selected LPG starting points

The same complementary query can start without knowing "supplier:42":

from purple8_graph import LPGNodeVariable

supplier = LPGNodeVariable(name="supplier")
result = semantic.match([
    LPGNodePattern(
        node_id=supplier,
        label="Supplier",
        equals={"state": "payment_overdue"},
        iri_properties={"class_iri": classification},
    ),
    RDFPattern(
        subject=classification,
        predicate=RDFTerm.iri("urn:requires"),
        object=RDFVariable(name="obligation"),
    ),
])
for row in result.matches:
    print(row.node_bindings["supplier"], row.bindings["obligation"])

This selects operational suppliers by their label and properties, then joins the classification value to complementary RDF knowledge. The supplier ID and classification IRI remain different bindings. A selected node can also anchor incoming/outgoing LPGEdgePattern paths in the same query. Reusing its node variable in another node pattern requires the same node to satisfy both sets of conditions; independent node variables form an explicit cross product.

Explain reports lpg_label, the label and an estimate of live tenant-local candidates. It does not read records or reveal other tenants' label counts. Execution uses the common hot/cold label readers, not a full corpus scan. Hot windows scan at most 256 raw membership IDs; all scanned IDs are charged before any candidate from that window is consumed, even when tenant filtering removes them. Thus limit=1 can charge 256 index entries, and a window can be prefetched before budget exhaustion is reported. Cold catalog lines charge one entry each, before prefix, liveness and generation filtering; stale and foreign memberships cannot evade the scan budget. Repeated openings charge their scan work again. Live candidate estimates do not bound this raw work.

Scalar and IRI-property conditions are checked against each tenant-local candidate after label selection; there is no new property-index pushdown. Canonical RDF records are never treated as LPG entities, including when an internal semantic label is explicitly requested. Candidates examined and raw membership entries scanned are separate counters. Early limits and failures close the cold catalog iterator; failure never returns partial success.

Anchored LPG paths in the same execution

LPGEdgePattern follows one directed operational relationship. Compose patterns for fixed-length paths; a discovered endpoint can feed another edge or node pattern, which can in turn bind semantic property values.

from purple8_graph import LPGEdgePattern, LPGNodeVariable

order = LPGNodeVariable(name="order")
item = LPGNodeVariable(name="item")
category = RDFVariable(name="category")
path_query = [
    LPGEdgePattern(source="supplier:42", target=order, edge_type="HAS_ORDER"),
    LPGNodePattern(node_id=order, label="Order", equals={"state": "overdue"}),
    LPGEdgePattern(source=order, target=item, edge_type="HAS_ITEM"),
    LPGNodePattern(node_id=item, iri_properties={"category_iri": category}),
    RDFPattern(
        subject=category,
        predicate=RDFTerm.iri("urn:requires"),
        object=RDFVariable(name="obligation"),
    ),
]
# Assumes the operational edges/nodes and semantic requirements already exist.
result = semantic.match(path_query)
for row in result.matches:
    print(row.node_bindings["order"], row.node_bindings["item"])
    print(row.bindings["obligation"])
    print(row.edge_ids, row.statement_ids)

There are two deliberately distinct binding domains:

• RDFVariable / bindings= / row.bindings contain full RDF terms.
• LPGNodeVariable / node_bindings= / row.node_bindings contain tenant-local LPG node ID strings. These IDs are not implicitly interpreted as RDF IRIs.

Variable names must not overlap between those domains. An initial node binding must name a declared LPG node variable and contain a valid ordinary node ID. Every LPG traversal component must be reachable from a supplied constant node ID, initial node binding or label-selected node variable. Cycles without any such root, isolated unbound node patterns without a label, and disconnected unanchored components are rejected before execution, rather than falling back to a whole-store scan.

At each step, a bound source permits outgoing adjacency access; otherwise a bound target permits incoming access. With both endpoints bound, outgoing access is used and the target is checked. Direction, the optional edge_type, both endpoint constraints, repeated node variables and property conditions must all match. edge_type=None permits all edge types on that anchored adjacency. Supplied edge types contain 1-256 characters. Edge patterns support the same strict scalar equals and explicit iri_properties conditions as node patterns.

Parallel edges produce separate supports. Self-loops can bind the same variable at both endpoints. One edge may satisfy multiple input patterns: there is no implicit edge uniqueness, simple-path condition or deduplication. The input order does not force an unbound node or edge to execute too early; the planner defers it until an earlier executed access supplies its anchor.

Adjacency reads reuse the existing native get_edges_page implementation: at most 128 adjacency IDs per page, without cloning the full adjacency. max_index_entries charges every scanned adjacency ID, including edges rejected by edge-type filtering, stale IDs and foreign-tenant edges filtered from a shared engine. Charging occurs after the bounded native page is read and before any matches from that page are yielded. Consequently:

• A rejected page can prefetch up to 128 IDs beyond the remaining budget; exhaustion still raises without returning partial successful results.
• limit=1 can charge 128 scanned IDs even if only one matching edge is used.
• Large all-nonmatching adjacency lists cannot evade the scan budget.
• Repeated adjacency openings and repeated scan work are charged again.

Successful statistics count scans actually charged, not just returned edges. Candidate counts include emitted edge candidates checked by the join. Node lookups from a separate node pattern charge separately. Cooperative cancellation is passed into the native page read and checked between candidates and joins.

Shared-mode traversal namespaces the anchor and filters edge IDs and both endpoints before exposing or binding records. Evidence and node bindings use tenant-local IDs. LPG path reads retain the existing non-snapshot behavior under concurrent LPG writes; the semantic lock does not turn them into historical snapshots.

Planning and bounded execution

For each partial binding, execution chooses the cheapest available component label by its current cardinality and then chooses the next pattern. A fully bound quad uses direct canonical-ID lookup. Other supplied constraints and repeated-variable conditions are checked against the decoded record. Unconstrained all-context RDF patterns use the tenant's semantic-record label, not an ordinary full-store scan. There is no whole-corpus Python join table.

explain() reports adaptive_index_nested_loop, an initial_accesses entry per input pattern, and first_pattern_index. Each access reports its path (direct, s, p, o, g, tenant, lpg_direct, lpg_label, lpg_outgoing, lpg_incoming, lpg_deferred or empty), estimated candidates and optional index label. These are initial choices, not a promise of the later execution order: bound values can change the best index at each join step. Explain reads index counts, not matching records, and performs no writes. lpg_deferred is not an empty result: it identifies a currently unbound access. Its estimate is null. Adjacency cardinalities are also unknown (null); these anchored accesses currently use a scheduling rank of 2, after known zero/one lookups. Nonempty LPG label accesses have a minimum scheduling rank of 3: a small local candidate count can still require scanning a large global label membership. This preserves preference for anchored adjacency rather than unnecessarily opening a broader label reader. Zero-cardinality accesses retain rank zero; remaining known accesses use their cardinalities. These are documented heuristics, not measured physical-work estimates or a global cost optimum. Existing anchored queries can keep using traversal-bound node lookups.

SemanticQueryLimits defaults:

Budget	Default	Meaning
max_candidates	10000	Semantic/edge candidates and RDF/LPG node direct lookup attempts
max_index_entries	10000	Admitted RDF label cardinalities plus scanned LPG label memberships and adjacency IDs
max_expansions	10000	Partial bindings expanded into a next-pattern access
max_result_bytes	8388608	Cumulative compact JSON bytes of individual returned matches

The first three budgets may be configured from 1 to 1000000; the byte budget from 1 to 67108864. Booleans, strings and fractional budgets are rejected. Statistics expose the consumed counters. Result-byte accounting includes bindings and statement IDs, not the surrounding result envelope or Python object overhead; it is not a process-RSS cap.

Common storage reads now use bounded native hot-ID windows and bounded cold physical-page windows. Label iteration no longer clones the full hot membership list or retains an ever-growing set of visited cold IDs. Cold memberships carry write generations, so re-ingesting the same ID does not produce duplicate rows; legacy catalogs are rebuilt once through bounded cold windows. A durable dirty marker precedes cold mutations; reopening after an interrupted publication rebuilds the catalog even if the node count is unchanged. Flush serializes with writes and drains pending commits before persisting membership files and counts and clearing that marker. Tenant reads preselect labels from tenant counts, filter prefixes before hydration, and use each node's primary label for unlabelled listings. These remain global label memberships, not a separate per-tenant membership index.

Index admission deliberately retains its existing conservative contract: it charges the whole RDF label cardinality, even if limit=1 consumes only one record; repeated openings charge again. This is now a work-admission policy, not compensation for full-label allocation. An oversized label fails before iteration opens. Broad queries may need tighter bindings rather than a larger result limit. Bounded working windows do not make deep offset scans constant time, remove stale/foreign-entry scan costs, or provide snapshot pagination under concurrent LPG writes.

Budget exhaustion logs and raises SemanticQueryLimitExceeded; no partial successful result escapes. Storage/integrity failures, deadline expiry and explicit cancellation also propagate rather than becoming empty results. Matching checks the existing cooperative query control throughout execution. Storage calls and lock acquisition are not arbitrarily preemptible.

match() and explain() hold the engine's semantic lock while executing, so managed assertions/retractions cannot invalidate admitted label cardinalities or interleave with a join. Shared tenants currently share that lock. It grants no reserved-record write capability. This serializes managed semantic writes; it is not a whole-database snapshot or a transaction against administrative clear, raw trusted storage access or concurrent LPG operations. Mixed queries can observe concurrent LPG changes; their node supports must not be treated as a consistent historical snapshot. The simpler statements() API retains its existing non-snapshot read behavior.

Output: unified RDF and LPG evidence

Use match_output() with the same patterns, bindings, node bindings and processing limits as match(). It runs the same indexed processor and renders only the selected evidence, not a whole-graph export:

output = semantic.match_output(
    patterns,
    bindings={"person": RDFTerm.iri("urn:example:alice")},
    limit=100,
    limits=SemanticQueryLimits(max_candidates=10000),
    max_output_bytes=8 * 1024 * 1024,
)

for row in output.matches:
    print(row.bindings, row.node_bindings)
    print(row.supports)  # Still in input-pattern order, including repeated uses.

    for evidence in row.rdf:
        print(evidence.statement_id, evidence.statement.to_dict())

    for item in row.lpg.nodes:
        print(item.origin, item.node.node_id, item.node.properties)
    for item in row.lpg.edges:
        print(item.origin, item.edge.edge_id, item.edge.source_id, item.edge.target_id)

payload = output.to_dict()

The output contract is SemanticQueryOutput, containing tenant_id, matches, stats and limit_reached. Each SemanticOutputMatch preserves the existing match's typed RDF bindings, operational node bindings, ordered supports, statement_ids, node_ids and edge_ids, and adds two views:

• rdf contains canonical statement IDs and their asserted quads. It preserves lexical literal forms, datatypes, languages, scoped blank nodes and default/named graph membership. Operational properties are not silently converted to RDF assertions. An LPG-only match has an empty RDF view.
• lpg contains detached Node and Edge models, each with an explicit origin of rdf or lpg. RDF projections retain their canonical statement IDs and full predicate IRIs; operational records retain their actual IDs and properties. Matching IRI-valued properties do not collapse operational nodes into RDF resources. Edge endpoints are included so the view is closed; endpoint hydration does not invent additional query supports.

Views deduplicate records within each match. Ordered supports still retain repeated uses, and different supporting combinations remain separate matches, even when bindings are identical. Named graph contexts therefore remain distinguishable. Mutating a returned Python model does not write to storage; governed RDF projections still cannot be persisted through ordinary graph CRUD.

RDF term vertices are derived, not independent stored entities. In a single project() call their timestamps come from that assertion. When multiple assertions in one match_output() match share a term, its detached vertex uses the earliest supporting assertion created_at and latest supporting assertion updated_at. This is a selected-evidence time span, not a term creation or modification history. Unselected assertions do not widen it; assertion edges retain their own timestamps. Conflicting RDF term facts still fail.

max_output_bytes accepts an integer from 1 to 64 MiB, defaulting to 8 MiB. It bounds the complete compact UTF-8 JSON envelope produced from to_dict(), including both views, bindings, evidence, tenant identity and statistics. Rendering checks records progressively and verifies the final envelope size; exhaustion raises SemanticQueryLimitExceeded, never partial success. Existing SemanticQueryLimits still constrain processing independently. stats.result_bytes retains its original binding/support-row accounting; it is not the full output size. These are payload/work bounds, not a process-RSS cap, and reading or serializing one large property value is not preemptible.

Operational supports are captured from the records actually matched rather than re-read after the query. Missing edge endpoints, unavailable RDF evidence, reserved operational records and conflicting observations of the same record within a match raise explicit errors. Endpoint hydration uses tenant-scoped direct lookups. Managed RDF reads retain the semantic lock, but concurrent LPG writes are not snapshot-isolated. Endpoints can be read later than their edge supports, so this is not a whole-database or historical snapshot. Conflict checking includes persisted system timestamps as well as facts, labels, embeddings and relationship fields for stored LPG nodes and relationships. Different timestamp observations of one stored record fail explicitly rather than silently selecting one. Derived RDF term vertices use the selected-evidence time span above.

The to_dict() representation is SDK JSON, not JSON-LD or SPARQL results JSON. Selected RDF evidence can also be exported as N-Quads below. REST/MCP query and mutation access is described after the export contract; cursors, streaming responses and other RDF syntax imports remain subsequent slices.

Standards output: N-Quads

SemanticQueryOutput.to_nquads() serializes the union of asserted quads supporting the selected matches in the W3C N-Quads format, including default and named graph contexts:

nquads = output.to_nquads(max_output_bytes=8 * 1024 * 1024)
print(nquads, end="")

The result is a string; encode it as UTF-8 when writing or transmitting it. The standard media type is application/n-quads. The serializer preserves literal lexical forms, datatypes, language tags and full IRIs. Quotes, backslashes, CR/LF line breaks and control characters in literal values are escaped. It does not normalize "01"^^<http://www.w3.org/2001/XMLSchema#integer> to "1".

Each distinct quad appears once, in first-encounter order. Duplicate supports and matching combinations remain in the SDK envelope, not as duplicate RDF lines. Default/named graph membership is part of quad identity. Operational LPG properties and relationships are not invented as RDF assertions. An empty selection or LPG-only selection produces an empty string.

Blank nodes receive valid opaque labels derived from tenant, native import scope and identifier. The same native blank node has one label across subject, object and graph positions; equal identifiers from different scopes or tenants remain distinct. These labels are stable for a selected dataset but are not global RDF identifiers. Native scope strings and statement IDs are not standard RDF terms and are not exported as metadata: dataset fidelity is up to blank-node renaming. Retain output.to_dict() as a sidecar when bindings, operational views, native identities, support multiplicity or statistics are needed.

This is an export of the selected evidence, not the entire tenant corpus or a backup. Check output.limit_reached when deciding whether the query selection meets your completeness requirements; that flag is SDK metadata, not part of an N-Quads document. The serializer reads only the detached result, does not query or mutate storage, and does not add inferred statements.

Its independent max_output_bytes limit uses the same 1-byte to 64-MiB range and 8-MiB default as SDK output. It counts the exact UTF-8 bytes, including syntax and final line feeds, before admitting each line. Exhaustion raises SemanticQueryLimitExceeded without returning a partial document. Cooperative cancellation/deadlines still apply. This is bounded, buffered serialization, not streaming or a process-RSS guarantee.

Generated output is validated in development tests with RDFLib's independent N-Quads parser, with literal normalization disabled to check lexical fidelity. RDFLib is a development-only dependency; export has no new runtime dependency. The exporter is not RDF dataset canonicalization, a signature format, an RDF importer, or a claim of complete RDF standards conformance.

Read-only REST and MCP access

POST /semantic/query and the MCP tool graph.semantic_query use the same tagged request schema and shared processor. Every pattern has an explicit model: rdf, lpg or lpg_edge. Term constants use the native RDFTerm JSON shape; variables use {"name": "variable_name"}. RDF graph: "any" selects all contexts; omitted/null graph still means only the default graph.

For example, send this JSON body to the REST endpoint or use it as the MCP tool's arguments:

{
  "patterns": [
    {
      "model": "lpg_edge",
      "source": "supplier",
      "target": {"name": "order"},
      "edge_type": "OWNS"
    },
    {
      "model": "lpg",
      "node_id": {"name": "order"},
      "iri_properties": {"kind_iri": {"name": "kind"}}
    },
    {
      "model": "rdf",
      "subject": {"name": "kind"},
      "predicate": {"kind": "iri", "value": "urn:requires"},
      "object": {"name": "action"},
      "graph": "any"
    }
  ],
  "limit": 100,
  "output_format": "json"
}

The example assumes matching operational records and asserted RDF already exist. These read endpoints do not create them or convert ordinary ingestion into RDF. Create assertions through the native SDK above or the governed single-quad mutation endpoints below. bindings, node_bindings, limits and max_output_bytes are also supported. The MCP input schema is generated from the active REST request model rather than maintained as a separate set of fields.

With output_format: "json" (the default), the response retains tenant identity, the SDK's typed bindings, ordered evidence and detached mixed LPG views, and limit_reached. Network stats is explicitly null: raw work counters can include denied records and must not expose hidden index cardinalities. The trusted local SDK retains its normal work statistics.

With output_format: "nquads", the response is still an application/json envelope, containing tenant_id, media_type: "application/n-quads", nquads, limit_reached and stats: null. The nquads field is the standard UTF-8-encodable document, not the entire HTTP body. The envelope preserves the selection flag for agents and applications; inspect it before treating an export as complete. Query multiplicity/native support metadata requires the JSON view, not duplicate RDF lines.

Authorization and evidence filtering

• Authentication selects the tenant; caller-supplied tenant_id is rejected. Existing query:traverse, node:read and edge:read permissions are required. N-Quads additionally requires data:export. No new application role or admin-only bypass is introduced. MCP role and allowed-tool restrictions continue to apply.
• Existing row, field, PII and caller-classification filters run before matching, not after private values have influenced joins. API-key label scopes and matching per-label read denials also constrain records and endpoints. The acting principal's existing record read/block policy applies as well. A directed LPG edge is excluded if either endpoint is inaccessible.
• LPG patterns operate on the filtered property view. A denied property cannot be matched by guessing its original value. A masked value is not an original IRI: explicitly interpreting an invalid masked IRI property fails rather than inventing an RDF binding.
• RDF assertions are all-or-nothing. If the canonical record, a projected endpoint or its projected relationship is hidden or altered by these filters, the entire assertion is excluded before matching. Redacted RDF is not returned as a new asserted fact with the old statement identity.
• Field/PII rules retain their existing top-level property-name semantics, applied to canonical record fields and projection fields such as statement, rdf_term and rdf_graph. This is not automatic per-predicate authorization, nested RDF-literal PII detection, or ACL inference from another operational node that happens to carry the same IRI.

SecureGraphEngine.semantic_graph() provides this policy-aware read-only view. Direct GraphEngine.semantic_graph() remains the trusted local SDK, including its existing assertion/retraction methods.

Network limits and errors

max_output_bytes limits the complete compact UTF-8 JSON response body, including export envelope metadata and JSON escaping. Its default is 8 MiB and its ceiling is 64 MiB. The selected SDK evidence is separately materialized under a cap of max(8 MiB, max_output_bytes); this retains the SDK's normal working-payload bound while allowing a small final transport limit. N-Quads text also passes its own byte check before envelope serialization. Existing binding/support result limits and query work budgets still apply.

Denied/stale index memberships still consume admission/scan budgets. A query may exhaust its work limit even when few authorized matches are returned. Counters are withheld, not replaced with misleading zeros. Cancellation and the configured cooperative query deadline cover processing and serialization; individual storage reads/serialization calls are not preemptible. This does not promise instantaneous cancellation on HTTP disconnect, a process-RSS cap, or snapshot isolation.

The byte cap describes the REST JSON body, not additional MCP protocol framing or the client's rendering of tool output. Oversized output never becomes a partial successful response.

Status	Meaning
401 / 403	Authentication or required read/export/tool scope denied
422	Invalid/unsupported query or invalid explicitly interpreted LPG IRI
409	Selected semantic evidence is inconsistent or unavailable
413	Processing or output budget exhausted
408 / 504	Cooperative cancellation / query deadline exceeded

Unexpected storage failures remain server errors, not empty or partial success. Successful queries use the existing audit logger with pattern count, format and response byte count, without recording raw query literals or result payloads.

Governed single-quad REST and MCP mutations

Operation	REST	MCP	Required permissions
Assert	POST /semantic/assert	graph.assert_statement	node:read, edge:read, node:create, edge:create
Physically retract	POST /semantic/retract	graph.retract_statement	node:read, edge:read, node:delete, edge:delete, plus admin/super_admin role

Both operations accept the same strict request schema:

{
  "statement": {
    "subject": {"kind": "iri", "value": "urn:example:alice"},
    "predicate": {"kind": "iri", "value": "urn:example:worksFor"},
    "object": {"kind": "iri", "value": "urn:example:acme"},
    "graph": {"kind": "iri", "value": "urn:source:document-1"}
  }
}

Omitted/null graph means the default graph, never all graphs. Blank terms require an explicit scope; "any" and query variables are not mutation terms. Lexical literal values, datatypes and language tags retain the native identity rules. One request targets exactly one quad. Caller-supplied tenant, record IDs, labels, arbitrary properties and owner/access metadata are rejected. Authentication selects the tenant and the engine stamps creation ownership.

Assertion returns {"tenant_id": "...", "statement_id": "...", "asserted": true}. Authorized retries return the same identifier without overwriting the existing record or transferring its owner. This confirms the assertion, not that the caller created a new record. Contributor/editor/admin MCP roles can assert; REST uses the existing effective permissions.

Retraction returns {"tenant_id": "...", "statement_id": "...", "retracted": true}. An authorized absent retry returns retracted: false. It physically removes the canonical assertion, not a recoverable soft delete, and leaves other graph memberships and assertions intact. Matching the existing destructive MCP tools, graph.retract_statement is human_only: API-key agent callers are blocked by the MCP gate even when they hold admin roles. An authenticated human admin can use the existing username/password/JWT MCP connection or the REST endpoint. REST retains the existing admin hard-delete rule; this does not introduce a new blanket JWT-only restriction on administrator service keys.

Before mutation, the canonical existing record (or proposed creation record) and its projections must pass the same whole-assertion visibility rules as queries, including row, field, PII, classification, record blocks and API-key label restrictions. Matching per-label permission entries must include both node:read and the relevant node:create/node:delete for the canonical node and projected endpoints. A hidden or changed assertion is denied, never overwritten, partially persisted or rewritten under its original identity. These remain top-level property filters, not automatic RDF predicate ACLs or literal-content PII detection.

Retraction checks active legal holds on the canonical record and both projected endpoint IDs/labels. The hold check and deletion serialize with hold placement in this process. Holds block removal even for super administrators; idempotent assertion of an unchanged held fact remains allowed. The semantic mutation lock covers authorization, existence checks and the native durable write. This is not an atomic multi-quad transaction or a cross-process policy snapshot.

SecureGraphEngine.semantic_graph() remains read-only. Its explicit assert_semantic_statement() and retract_semantic_statement() methods require a tenant-bound acting principal and perform these checks; trusted local native SDK methods keep their existing behavior. Network retraction checks do not retrofit a new legal-hold contract onto the trusted raw SDK.

Invalid terms or unsupported fields return 422 before engine access. Permission or visibility denials return a generic 403; inconsistent canonical evidence or a legal hold returns a generic 409 without private record/hold details. Storage failures remain server errors. Successful operations use the existing audit logger with operation, statement ID, tenant and actor, plus the retraction outcome; raw RDF terms are not copied into that event. Mutations use the write executor and existing WAL, without implying query deadline or HTTP-disconnect cancellation guarantees.

Bounded N-Quads import

Externally produced RDF can enter through the same native canonical write path, without DocIntel or an external RDF service:

• SDK: semantic_graph().import_nquads(text, blank_node_scope="document-one")
• REST: POST /semantic/import
• MCP: graph.import_nquads

The REST body and MCP arguments share one schema:

{
  "nquads": "<urn:alice> <urn:worksFor> <urn:acme> <urn:document-one> .\n",
  "blank_node_scope": "document-one",
  "max_input_bytes": 8388608,
  "max_statements": 1000
}

This accepts an inline Unicode document, not a URL, server-side path or streaming upload. Input follows the RDF 1.1 N-Quads grammar and the existing native term constraints: absolute IRIs, scalar Unicode, native term-size limits and supported language tags. It does not accept Turtle, TriG, RDF/XML, JSON-LD, RDF-star/triple terms or RDF 1.2 extensions. Escapes are decoded once; literal lexical values such as typed integer "01" are not numerically normalized. Existing language-tag case rules still apply. Default, IRI-named and blank-named graphs remain distinct.

blank_node_scope is required even when a document has no blank nodes. Choose a distinct scope for each independent source document and keep it stable across retries. Every source blank label, including Unicode/colon/long labels outside the native identifier subset, maps to a stable opaque native identifier within that scope. Sharing across subject, object and graph positions is preserved. Different scopes or tenants do not share blank identities. N-Quads carries no original native scope metadata: export/reimport preserves a dataset up to blank node renaming, not the previous native blank IDs.

Admission, permissions and preflight

• The decoded document's complete UTF-8 encoding, including whitespace and comments, must fit max_input_bytes. Default and hard ceiling: 8 MiB. This is the document size, not its surrounding HTTP JSON representation.
• max_statements defaults to and cannot exceed 1,000 input statements. Duplicate source statements consume this budget too. Requests can lower either limit; overflow fails before any assertion is written, never truncates. Empty/comment-only documents return an explicit successful zero-count result.
• Full syntax parsing precedes storage writes. The secure path also authorizes every distinct assertion before the first write, under the shared semantic lock. Canonical/projection visibility, record blocks, classification and action-specific API-key label restrictions use the same checks as single-quad assertion. Authorization is checked again before each write.
• Network and explicit secure imports additionally require data:import, node:read, edge:read, node:create and edge:create, even for empty input. Default admin roles have these permissions; default editor/contributor roles do not have data:import and need an explicit grant. MCP's mutation role and allowed-tool gates also apply.

SecureGraphEngine.import_semantic_nquads() provides the explicit governed method for a tenant-bound acting principal. Its query view stays read-only. Trusted raw SDK imports retain trusted SDK authorization behavior, but still parse the full input and validate existing assertions before writing.

Completion, partial progress and recovery

Successful output is:

{
  "tenant_id": "default",
  "blank_node_scope": "document-one",
  "input_statements": 1,
  "unique_statements": 1,
  "acknowledged_statements": 1,
  "complete": true,
  "atomic": false
}

Duplicate quads coalesce in first-encounter order. Acknowledged counts include unchanged assertions that already existed; they are not counts of new nodes. Retries do not overwrite existing facts or transfer ownership.

The importer is deliberately not an atomic dataset transaction. Each assertion uses the normal WAL-durable write. Invalid syntax, admission failures and initial policy/integrity preflight failures write no assertions. A later engine, storage or policy failure can leave a durable prefix:

• Known write failures return a non-success status with detail.error: "semantic_import_interrupted" and a typed progress object containing complete: false and atomic: false. Permission changes map to 403, integrity conflicts to 409, other handled write failures to 503.
• acknowledged_statements is a lower bound on completed input assertions. The failing write may have become durable before it raised. Do not resume by skipping that many source lines.
• After resolving the cause, retry the entire original document with the same scope and authenticated tenant. Canonical identity makes that replay idempotent. Reusing a different scope would create different blank nodes.
• A process crash, lost response or unexpected server failure may provide no progress response at all. The same whole-document retry remains the recovery contract. No automatic rollback, import job checkpoint or exactly-once delivery claim is made.

REST returns 422 for invalid request fields, unsupported terms or invalid syntax, 413 for admitted document byte/statement-budget exhaustion, and generic 403/409 for initial authorization/integrity failures. Errors do not echo private canonical records or storage paths. Successful imports audit only summary input/unique/acknowledged counts; interrupted writes log the tenant, acknowledged count and exception type, not the RDF document.

Memory is bounded by the admitted request and statement caps, not a promise of constant memory for arbitrary files or whole-store snapshot isolation. The import uses the write executor; it adds no query-deadline or HTTP-disconnect cancellation guarantee. Parsing uses existing in-process Lark; RDFLib remains development-only. Tests independently parse exported datasets and common input forms with RDFLib, account for its plain-string/language-tag representation differences, and explicitly check Unicode blank labels that its legacy N-Quads lexer cannot read. The grammar ships with the pip package data and is explicitly copied into the commercial compiled package.

Reviewed document-to-output proof

The local HTTP acceptance proof also exercises actual DocIntel jobs through Graph's multipart preview and polling routes, with an explicitly controlled inference transcript. It reuses the reviewed evidence/output assertions below; it does not claim model quality.

The two-service TCP restart proof then exercises the complete HTTP handoff with separate Graph and DocIntel processes. Both receive SIGKILL after acknowledged work. Shared and dedicated tenant cases reopen their original stores and retain completed extraction results, independent canonical schemas, original/quote evidence, mixed DTOs and selected N-Quads, actual BM25 retrieval and source-policy enforcement. It reuses persisted connection settings and original credentials rather than re-ingesting to recover. This remains a controlled inference transcript, not a live model or an arbitrary mid-job crash/power-loss qualification.

tests/test_semantic_document_proof.py exercises a real public-domain excerpt, not a synthetic invoice or a claim of whole-document extraction. The source fixture contains the dateline and opening paragraph of the National Archives Declaration transcript. It excludes the website's navigation and the rest of the Declaration. Its recorded parser handoff pins the SHA-256 of those exact excerpt bytes.

The proof separates three responsibilities:

1. Local parsing: DocIntel's actual HTML parser and deterministic Markdown renderer produce text and page ranges. No LLM extraction, OCR, classifier, extraction job service or external model is invoked.
2. Explicit review: the checked-in declaration-dateline-v1 mapping selects July 4, 1776 from In Congress, July 4, 1776. It creates a reviewed LPG observation and a native RDF literal with that unchanged lexical value. The example predicate urn:example:document:dateline means the source's dateline; it does not infer a publication/creation date or an xsd:date. The RDF subject addresses the excerpt bytes by SHA-256, and the named graph identifies this reviewed mapping and source. This is a reviewed example, not an automatic date extractor or a general review-approval service.
3. Governed output: real application handlers authenticate service keys, retain and download the original through the file API, commit the reviewed observation/Markdown/pages, and separately assert the selected quad through /semantic/assert. Ordinary commit creates no native RDF. A mixed /semantic/query joins the observation's explicitly interpreted IRI properties to the quad and follows MENTIONED_IN and HAS_CHUNK through the retained file.

The JSON evidence contains the canonical assertion, operational observation, chunk and file, with the source quote recoverable from retained offsets. Mention edges carry document-relative Unicode-code-point spans and the exact Markdown SHA-256; original HTML bytes have a separate fingerprint. An assertion retry retains its canonical ID. Selected N-Quads round-trips to exactly the reviewed quad; operational edges do not become extra triples.

Hiding the file, chunk, observation, assertion, or required join fields removes the mixed row and its selected export for that reader. This is not a new source-dependent assertion ACL: a standalone RDF query can still return an otherwise authorized assertion when an operational support is hidden. Include the required support patterns when the application requires source-backed answers.

Reproduce the proof

From the Graph checkout, the recorded DocIntel handoff runs without installing DocIntel into Graph. Live-parser cases are explicitly skipped without the interpreter setting:

PYTHONPATH=src .venv/bin/python -m pytest \
  tests/test_semantic_document_proof.py tests/test_semantic_document_restart.py \
  tests/test_file_read_policy.py tests/test_mention_spans.py \
  -q --no-cov --tb=short

To include actual local DocIntel parsing, use its own prepared environment:

PYTHONPATH=src \
P8G_DOCINTEL_PYTHON=../Purple8-DocIntel/.venv/bin/python \
P8G_DOCINTEL_SOURCE=../Purple8-DocIntel/src \
.venv/bin/python -m pytest \
  tests/test_semantic_document_proof.py tests/test_semantic_document_restart.py \
  tests/test_file_read_policy.py tests/test_mention_spans.py \
  -q --no-cov --tb=short

The two P8G_DOCINTEL_* paths resolve relative to the Graph checkout before the parser subprocess switches to a temporary working directory, isolating package-initialization artifacts. An installed DocIntel package does not require P8G_DOCINTEL_SOURCE. Its HTML dependencies must already be installed. Parser output must match the recorded handoff exactly; output changes require reviewing the fixture and mapping, not silently accepting new offsets.

Both processes block external socket/DNS access during the proof. Graph calls use the real application in-process through its ASGI test client, not a live TCP deployment or a deployed DocIntel service. Authentication, ingestion, policy checks, Cortex storage, file downloads and semantic processing are real. Embedding outcomes are explicitly not_configured; this does not qualify vector retrieval. The test stores are temporary. This narrow proof does not qualify OCR, arbitrary formats, LLM extraction accuracy, full-document completeness, historical reconstruction, packaged distribution or production readiness.

Restart and recovery qualification

tests/test_semantic_document_restart.py extends the same proof across separate operating-system processes, for shared and dedicated tenant storage. It runs with both the recorded handoff and actual local DocIntel parsing when configured.

The writer obtains successful ingestion and assertion responses, captures the authorized evidence, persists independent File-row and mention-field policies, then either shuts down cleanly or calls os._exit() without application shutdown or engine close. Two successive fresh reader processes reopen that same application data directory. They verify:

• The same service keys still authenticate against persisted users/key hashes. Temporary plaintext keys travel only through captured process pipes, not a checked-in fixture or a plaintext on-disk manifest.
• The original download is byte-identical; the source hash, Markdown hash, precise quote spans, source version, canonical assertion ID, literal lexical value, named graph and selected N-Quads remain unchanged.
• File, Chunk, Page and reviewed-observation IDs survive. Repeating the original commit and assertion after reopening does not duplicate them or change the source-backed semantic result.
• Saved row and field policies each continue to suppress unauthorized mixed evidence and selected export. Authorized output remains available.

Before/after source-node, source-edge and mixed-output comparisons now include system created_at and updated_at, without timestamp exclusions. Source properties and event dates are also compared exactly. This qualifies the newly exercised writes and replays, not reconstruction of timestamp history already discarded by older code.

The proof exposed and fixes positional-property forwarding in shared tenant node writes: add/update/upsert now bind labels and properties consistently with GraphEngine, preserving tenant stamps without modifying caller dictionaries. It also exposed a source-download policy gap; the file visibility contract now covers metadata, history and originals instead of protecting only semantic query output.

The clean/abrupt cases above prove reopening acknowledged document and assertion writes after clean or abrupt process exit. They do not simulate power loss, filesystem corruption, an interruption inside every ingestion step, cross-resource transactions, node failover or a deployed HTTP service. A successful ingestion commit and a subsequent RDF assertion remain separate operations, not one atomic document-plus-RDF transaction.

Interrupted re-extraction and a lost commit response

The same test module also interrupts an authenticated secondPOST /ingest/commit before its caller receives a response. It starts from the complete, asserted document proof, then constructs a deliberate re-extraction candidate: whitespace-only Markdown normalization, an adjusted Page range and a test-only review metadata marker. The original bytes, dateline literal, reviewed RDF assertion and exact quoted character span do not change. This candidate is not presented as another actual DocIntel parser result or an extraction-accuracy measurement.

Test-only wrappers terminate the entire writer with os._exit() immediately after each of these real operations returns:

Interruption point	Required first-open result
Entity update, with its prior snapshot journaled	Original entity and source graph restored
Chunk creation and source/mention linkage	Candidate chunks/edges removed; original graph restored
Structured Page replacement	Original Page properties and graph restored
Retirement of the original Chunk and its incident edges	Original Chunk and edges reinserted; candidate graph removed
After confirmed finalization and cleanup, before the HTTP caller receives a response	Completed candidate preserved, not rolled back

Each point runs for shared and dedicated storage, with recorded and configured local-parser variants. Checkpoint assertions inspect actual graph mutations and the pending journal; reaching the hook alone is not accepted as proof. At the unfinished write checkpoints, the journal must still exist. The lost-response checkpoint runs after cleanup, when it is absent. Under the current completion protocol below, a confirmed commit may also retain a completed journal awaiting cleanup.

Fresh readers reuse the original credentials and independently verify:

• Recovery completes before the tenant's source graph is exposed. All source node properties and outgoing edges match the pre-attempt snapshot after an unfinished attempt, including the original system timestamps.
• The original bytes, quote, canonical statement, named graph and selected N-Quads remain faithful, and saved row/field policies still apply.
• Retrying the interrupted candidate succeeds, and repeating that same retry converges on the same source IDs, properties, links and mixed output. Another fresh process then reopens and verifies the completed candidate.
• If the process died after the commit point, the new Page/Chunk state is already present on the first reopen even though the caller received no success response. Repeating that request remains safe.

These cases originally passed without a production recovery change; the current I/O finalization protocol is described below. They provide controlled stage-boundary process-exit qualification, not arbitrary instruction-level SIGKILL, interruption inside a batch or during recovery itself, disk-full/ fsync failure, power-loss durability, concurrent-writer isolation, exactly-once post-commit lineage/notification delivery, or a document-plus-RDF transaction. The RDF assertion already exists before this interrupted re-extraction. Content-addressed blob cleanup and inline-original creation are not exercised by this existing-file retry scenario.

Partial batch writes and re-interrupted recovery

Further cases in tests/test_semantic_document_restart.py exercise two partial-batch fault models inside the authenticated re-extraction request:

• Node batch: a test-only storage wrapper durably writes the first of two planned nodes, confirms the second is absent, then terminates the process before the engine batch returns. This manufactures a durable prefix at the Python storage boundary. It does not assert that the underlying native batch itself can partially commit or qualify a crash inside the Rust call.
• Edge batch: the writer exits immediately after the first new edge's actual storage write, inside the existing per-edge batch loop. The journal must already contain both planned edge IDs, while only one edge exists.

Two explicitly synthetic RecoveryProbe entities supply the plural batch inputs. They are marked test-only, are not parser-extracted facts or native RDF assertions, and have no MENTIONED_IN links. The original reviewed dateline mapping and original file bytes stay unchanged.

The first fresh reader must recover the complete original source-node and outgoing-edge snapshot, including absence of leftover probes or links. It then retries the interrupted candidate twice; a second reader independently verifies the completed state and another idempotent retry.

Separate cases first interrupt the writer after prior-content retirement, then terminate a fresh recovery process at these undo boundaries:

Recovery interruption	State required before the forced exit
Candidate Chunk deletion	Candidate absent; original Chunk still absent
Original Page restoration	Exact original Page properties restored; original Chunk still absent
Original Chunk reinsertion	Chunk restored, but its deleted incident edges still absent
First incident-edge reinsertion	One original edge restored; another still absent

At each boundary the undo operation must have actually succeeded, the journal must remain, and the tenant engine must not yet appear in either published engine cache. The parent process verifies that the retained journal is byte-identical after the crash. The next open must complete recovery before serving document reads, then pass the same source, policy, RDF/N-Quads and retry checks.

In addition to the individual boundaries, a sequence case interrupts four successive recovery processes against the same journal before allowing recovery to finish. Shared and dedicated storage run with both recorded and configured local-parser variants.

These fault models originally passed without a production recovery change. They extend the stage-boundary evidence above; they still does not prove arbitrary native instruction-level interruption, power-loss durability, disk-full/fsync-failure handling, concurrent-writer isolation or exactly-once post-commit side effects. The existing system separate document/RDF transaction boundaries remain. Timestamp comparisons are now exact under the persistence contract below.

Durable completion and storage-I/O failures

Ingestion finalization now uses an explicit completion protocol:

1. Append a terminal completion record and sync the journal file.
2. Rename the journal from .jsonl to .complete, then sync its directory.
3. Only then confirm completion. Closing, unlinking and syncing the removal are cleanup, not the commit point.

Directory-sync errors and short writes are explicit failures, not silently ignored durability downgrades. The filesystem and platform must support the required directory synchronization; unsupported configurations fail rather than receiving an unqualified durability promise.

If finalization starts but completion cannot be confirmed, the request does not compensate the data or claim it rolled back. A reported sync failure does not prove that bytes were absent: fresh recovery preserves data when a valid terminal completion record exists, but undoes the attempt when only an unterminated marker fragment exists. A .complete filename without a valid terminal record fails closed. Malformed complete records or data after the terminal record also require investigation.

Incomplete undo or uncertain finalization blocks new tenant-engine admission, including cached-engine access. Governed REST reads return 503 with reason=commit_recovery_required; MCP retains the HTTP failure details. Other tenants are not quarantined merely because they share physical storage. Existing trusted engine references and already-running operations are not revoked: this is not a new concurrent-reader/writer isolation guarantee.

Known storage failures return 507 with explicit outcome, rollback and retry details. Repair storage and restart before retrying the identical payload against an unconfirmed tenant. Do not delete the retained journal to unblock access. After recovery, use the same IDs, file_ref, source content and assertion keys to confirm or converge the result.

Once completion is confirmed, failed cleanup cannot turn committed data into a rollback. A successful response instead includes journal_cleanup: "pending"; the ingestion UI warns that data is committed and an administrator must check storage. New recovery cleans completed journals without undoing their data. The normal success value is journal_cleanup: "complete".

The focused journal tests cover creation/open/sync failures, short writes, completion publication failures, cleanup failures, malformed completion, recovery I/O failures, undo-execution failure and live tenant quarantine. Document subprocess cases cover creation and append errors, full-but-reported- unsynced and torn markers, failed undo, and committed cleanup pending. These run across shared/dedicated tenants and recorded/actual local DocIntel parsing; fresh readers verify source snapshots, original bytes, governed evidence, RDF/N-Quads and identical-payload retry.

Downgrades require care. New recovery supports old pending .jsonl and .failed journals. Older permissive readers may ignore the new terminal record and undo a still-.jsonl attempt. The durably published .complete suffix keeps acknowledged completions outside their legacy pending scan, but this is not blanket downgrade compatibility. Resolve pending work and stop the instance cleanly before downgrading; do not run old and new writers against the same storage.

These are controlled I/O fault-injection and process-restart checks, not power-loss, filesystem-corruption or deployed-service certification. Directory-publication errors have focused unit coverage, not an additional document subprocess cutpoint. The transaction-isolation, separate document/RDF operation and exactly-once-side-effect limitations remain.

Persisted system timestamps

Native hot writes now preserve the model's created_at and updated_at through the existing opaque-property encoding, without a Rust rebuild. Decoding restores them as top-level model fields, not user properties. Single writes, normal and fast node batches, edge batches, cold-to-hot promotion, WAL replay, checkpoint/rewrite and exact-snapshot recovery use the same timestamp-preserving path. The homogeneous fast-batch path also persists acknowledged graph records and default vectors without requiring an explicit checkpoint; see the recovery contract below.

For GraphEngine writes:

• Updates and create-or-replace writes preserve an existing record's creation timestamp. Deleting and creating a new record is a new lifecycle.
• Node/edge updates, upserts and duplicate-update batches advance updated_at only when stored content changes. Identical retries keep both timestamps. Label, embedding, weight and validity changes count; mere dictionary-key order does not. Boolean, numeric and string property changes remain distinct.
• A create-or-replace call can refresh ingestion context or replace omitted properties; it is not necessarily a no-op simply because its ID is unchanged.
• Exact snapshot restoration preserves the snapshot's timestamps rather than recording recovery as a new content edit.
• Unchanged structured Pages, Tables and Rows are not rewritten merely because they carry engine-owned context, access properties or a shared-tenant stamp.

The private storage properties __p8_created_at__ and __p8_updated_at__ are engine-owned. Public engine writes strip attempts to supply them and log the reserved keys. They are removed from decoded properties and partial property projection. Source/event dates, including ordinary properties named created_at or updated_at, remain separate caller-supplied facts.

Legacy native top-level timestamps remain readable. Replay can preserve timestamps still present in an older WAL, but no migration invents missing history or reconstructs dates already replaced by an earlier replay or checkpoint. Use an authoritative backup or audit source for that history. These are UTC system write times, not trusted document dates, a monotonic clock or a cross-replica clock-equivalence guarantee.

The document proofs compare complete source and output DTOs across fresh readers, interruptions, recovery retries and identical-payload replay. Focused tests also cover native/timestamp precedence, malformed timestamp rejection, cold-to-hot movement, no-op writes, property-type changes, create-only/upsert locking and explicit output conflicts. None of this adds whole-database snapshot isolation or extends the power-loss qualification.

Transaction-time history status

Creation/update timestamps alone are not a bitemporal version store. Existing valid-time traversal filters the currently stored edges; it does not recover superseded or deleted beliefs. The Python WAL's timestamp helper is not a historical database query or a permanent history-retention contract.

An isolated transaction-time foundation is being qualified in src/purple8_graph/temporal.py. It keeps version headers, archived snapshots and historical adjacency in Cortex, with a durable mutation intent and commit publication. Current snapshots are archived on change rather than duplicated on initial ingestion. Large archived values use bounded cold records; history does not create one label or an in-memory catalog per entity. Historical traversal reuses the existing bounded traversal implementation and selects the system-time version before applying the independent valid-time window.

This foundation is not yet installed on GraphEngine writes, REST, MCP or tenant/security facades. Do not treat it as a production bitemporal API. Its callers must currently supply all affected keys, including cascaded edges. Automatic capture, transaction/rollback boundaries, retention and legal holds, source erasure, backup/restore and public policy enforcement remain release requirements. Automatic activation is intentionally not wired while those requirements remain open.

The agreed activation contract is default-on for new databases and explicit activation for existing databases, with coverage beginning at activation. Without an expiry policy history is retained until explicitly pruned; configured expiry permits automatic pruning, legal holds override destruction, and reads outside retained coverage must explicitly report unavailable history. Those are integration requirements, not claims that these policy surfaces are already implemented.

Homogeneous batch acknowledgement and vector recovery

GraphEngine.add_nodes_batch() now writes complete new-node records, including timestamps and the supplied default vectors, through one batched WAL write and synchronization before native graph mutation. Vectors use the existing packed-float32 WAL encoding; no native rebuild or new WAL operation is required. Checkpoints retain hot-node recovery vectors instead of assuming a later clean HNSW save has happened. Normal batch WAL records also retain their supplied vectors.

The default search index is derived state. On opening, the engine streams durable nodes and repairs it in bounded batches, including when a saved index predates the acknowledged batch. Without a saved index, durable vectors supply its dimension. Read-only recovery uses temporary, tiered index storage and removes it on close, rather than modifying source index files. This recovery concerns the default vector slot; it does not certify named-slot, BM25, notification or replica recovery.

The homogeneous API is create-only:

• Existing IDs, including cold-resident records, keep their original facts, ownership, timestamps and vectors.
• For repeated new IDs in one call, the first input occurrence wins.
• The result is the number of inserted nodes, not the number of requested rows. A fully repeated batch returns zero and does not rewrite vectors. Use an update/upsert API when replacement is intended.
• Storage sequencing drains pending single-record writes before deciding which IDs are new. Capacity, license and property-index maintenance also apply to the fast path.

A write, short-write or synchronization failure is not an acknowledgement and is not proof of rollback. A valid WAL prefix or a fully synchronized batch may recover even if native/vector mutation never finished. WAL I/O failures and fast-batch vector mutation failures block further writes and checkpoints on that open engine. Preserve storage and diagnose the fault; reopen and verify recovery before replaying the original IDs and payload. Detected truncated, checksum-damaged or undecodable WAL frames also block new writes/checkpoints after reopening. A damaged WAL needs repair or a consistent backup, not blind retries or deletion. Read-only/closed WAL writers now reject attempts rather than applying mutations without a log.

Local subprocess tests exit before WAL append, after synchronization, after a native-insert prefix, after native mutation, after acknowledgement and after explicit/automatic checkpoints. Fresh shared/dedicated readers check exact node DTOs and timestamps, vector rankings and no-op replay. Fault-injection tests cover write/sync/short-write failures, short checkpoint writes, vector failure, skipped duplicates, read-denial preservation and recovery across a bounded-batch boundary. Recovered nodes can participate as complete evidence in mixed RDF/LPG output. These are controlled local process/I/O proofs, not power-loss or arbitrary-corruption certification.

Resource boundary: source vectors now occupy durable storage and native node properties rather than disappearing from this fast path's recovery state. This path remains hot-tier ingestion, not seal-and-evict ingestion. The recovery pass scans durable nodes and may rebuild existing vectors; its bounded working buffer is not a total-process RSS guarantee. Previous throughput, startup-latency and bounded-RSS results are not requalified by these functional tests. Use bounded batches and measure this path on the target corpus before sizing a deployment. Older writers/checkpointers can discard embeddings again; do not downgrade and assume this contract holds.

Restart cost and actual HTTP regression coverage

Existing-vector repairs are grouped by segment within each recovery batch. A cold segment is loaded, updated and persisted once for that group, not once per vector. Replacing vectors does not consume new HNSW capacity; saved growth settings survive reopening, and older index metadata remains readable. This changes derived-index maintenance, not the homogeneous SDK's first-new-ID-wins contract.

A small isolated local diagnostic used float32 vectors with 128 dimensions, 500-row input batches, 1,000-vector segments, one hot-cache slot and BM25 disabled. For 5,000 nodes, writable reopen fell from 18.80 seconds to 1.88 seconds in a paired run. RSS after opening/querying remained approximately 1.1 GiB. Stage measurements locate most of that allocation in writable storage opening, before vector repair. These observations are not production throughput results, a statistical performance guarantee or proof of bounded total RSS. Fast graph records still remain hot-tier, and recovery still scans durable nodes and repairs existing vectors.

tests/e2e/test_restart_socket.py starts a real Uvicorn subprocess on an inherited OS-assigned loopback listener. Separate authenticated tenants exercise SDK homogeneous ingestion, REST writes, vector/hybrid queries and mixed RDF/LPG output. After successful HTTP responses, the server receives SIGKILL; a new server verifies exact node DTOs, timestamps, lexical RDF values, tenant isolation and no-op replay of the original SDK batch. Dedicated and shared storage are covered. External DocIntel/LLM calls, browser behavior, graceful shutdown, power loss and replication are not qualified by this test.

Shared-isolation schema blocker closed: single-node POST /nodes now uses the canonical tenant schema enforced inside the engine. The TCP proof uses single-node creation for both shared and dedicated REST writes, without forwarding a global schema. REST/MCP schema definitions and SDK definitions now use the same native authority; see Schema & Data Model. HTTP bulk duplicate handling is also not the SDK's zero-insert retry contract.

Source-bound reviews across replacements

Replacing the original does not authorize replaying old reviewed extraction over newer evidence. The source-binding contract checks original-byte hashes and optional version pins before writes. Conflicts preserve current chunks, historical evidence and independently asserted RDF facts; they do not automatically retract facts or ingest a historical version. Identical Markdown in different File versions has distinct strict chunk IDs.

Portable source-evidence backups

Source-backed evidence can also be restored into a different file-store root: the portable backup contract includes referenced original bytes and versions, remaps only internal storage locators and preserves graph/RDF identity and source coordinates. tests/test_backup_files.py makes the original store unavailable before restore, checks shared/dedicated tenant and encrypted/plain cases, and exercises restored original download plus the reviewed mixed RDF/LPG and N-Quads evidence. It also requires corrupt or incomplete bundles to fail before graph replacement.

BM25 ingestion publication

The heterogeneous batch_add_nodes() path used by document entities, document chunks and structured-content creation previously persisted nodes without queuing the integrated BM25 update. Single-node and homogeneous batch ingestion already queued those updates; that mismatch could leave ingested content stored and vector-searchable but absent from BM25.

Batch creation and merged upserts now enqueue snapshots after successful storage writes, including a committed prefix before a later duplicate error. Skipped duplicates and failed writes are not published. Clearing indexed fields removes obsolete terms, and shared views expose tenant-prefixed BM25 lookup rather than an unscoped base-store search. The TCP proof checks actual method: "bm25" responses after authenticated POST /ingest/commit creates both reviewed entities and document chunks, with no configured embedding provider. A bounded-scan fallback is not accepted as proof that indexing worked.

BM25 remains asynchronous and best-effort. Full-queue losses are counted and logged, and indexing failures are logged at error level. Indexed properties are name, title, description, body, text, bio and content; unrelated custom fields are not automatically indexed. P8G_FULLTEXT=0 and read-only engine opening disable the integrated index. The index is saved on clean close, but persisted postings are no longer trusted as the authoritative state on reopening. This does not promise synchronous search visibility or durable/exactly-once background indexing.

BM25 reconstruction before readiness

Every writable engine with BM25 enabled now rebuilds the integrated index from durable graph records before starting the full-text worker or becoming ready. It streams both hot and cold records, uses the same indexed-property extraction as live updates, and does not enqueue the entire corpus or retain a corpus-sized set of visited IDs. Saved tokenizer and scoring settings are retained, but stale postings are not restored. Missing or damaged derived snapshots are reported and rebuilt from source; source-scan or indexing failures propagate instead of publishing a partial index.

This repairs missing queue work, old snapshots, changed labels/text and deleted records after local process death. Optional Rust accelerator mutation failures disable that incomplete mirror and explicitly report the complete Python-index path. No compiled artifact is rebuilt.

The real TCP proof now requires BM25 entity, chunk and REST-write results after SIGKILL as well as before it. Separate process tests suppress background publication through injected queue overflow, retain a stale snapshot, and verify repeated writable reopen with and without a graph checkpoint, including shared isolation and cold records. Disabled and read-only engines keep their prior behavior: no integrated BM25 recovery.

Startup/resource trade-off: this deliberately adds a content-sized scan on every writable open, including clean restarts. In a small paired local diagnostic with BM25 enabled, 5,000 records, 128-dimensional vectors, 500-row batches and repeated three-token text, BM25 initialization rose from 3.4 ms to 114.7 ms. Total open time rose from 1.88 s to 3.20 s, while the separate vector-repair stage varied from 1.54 s to 2.69 s; the total change cannot be attributed solely to BM25. BM25-stage RSS rose by about 4.6 MiB; total RSS remained around 1.1 GiB, predominantly in storage opening. These are diagnostic samples, not production limits or latency guarantees. The inverted index itself and legacy snapshot JSON decoding remain content-sized; bounded source iteration is not bounded total RSS. This qualifies local process recovery, not power-loss, arbitrary source corruption, concurrent writers or replication.

Tenant and mutation boundaries

Obtain the view from the engine already selected for the authenticated tenant:

engine = tenant_manager.get_engine(authenticated_tenant_id)
semantic = engine.semantic_graph()

Shared-mode engines preserve the existing tenant storage prefixes; dedicated engines retain their separate paths. Term and statement IDs also incorporate tenant identity. The standalone GraphEngine view uses the default tenant. Do not construct a semantic view over an unscoped shared base engine for a network request.

The __rdf_ ID namespace and __RDF labels are reserved. Ordinary graph CRUD, bulk writes, rollback helpers and Cypher mutation paths cannot forge or modify these records or persist a detached projection as an independent entity. Change semantic relationships through assert_statement(), add_relationship() and retract_statement().

Internal statement records are excluded from corpus graph algorithms and full-text corpus rebuilding, and their labels are hidden from normal label pickers. Generic raw graph reads can still inspect canonical records within the selected tenant. Existing Cypher traversal does not yet traverse the virtual projected relationships.

Durability and recovery

Each assertion is one ordinary engine node write, using Cortex's existing WAL, ownership checks, cache invalidation, audit hooks and replication shipping. There is no multi-record materialization transaction to repair after a crash. Semantic mutation serialization is shared by views over the same engine.

WAL replication validates semantic records before replaying them through a narrow internal mutation scope. Snapshot replication validates semantic payloads before clearing the destination and rebuilds the same canonical records. Unsupported record versions, changed identities, inconsistent index labels or tenant mismatches raise explicit integrity errors rather than returning partial semantic results.

The scope is per assertion, not an atomic dataset transaction. Concurrent list reads have no dataset snapshot guarantee. Administrative whole-store clear still removes semantic records along with other data. Direct low-level storage access is trusted internal code, not a public integrity or authorization boundary.

Mixed RDF/LPG load qualification

benchmarks/semantic_mixed_qualification.py is a dedicated same-engine, closed-loop SDK harness. It alternates three workloads across concurrent clients:

• RDF: a two-statement join, retaining a named source graph and an exact typed literal lexical form.
• LPG: a supplier-to-order-to-item traversal, with two allowed edge types, depth two and at most three visited nodes. An empty Query() deliberately bypasses the traversal-result cache.
• Cross-model: the anchored operational path joined through explicit IRI properties to both asserted quads, returning unified evidence, JSON and selected N-Quads.

Every completed operation is checked against deterministic expected bindings, IDs and ordered supports. Traversals check nodes, edges, paths and visit bounds; unified output additionally checks exact assertions, record snapshots, RDF term vertices, edge topology, endpoint closure and lexical/context-preserving export. All cases and workload kinds pass an untimed preflight before measurement. No provider, DocIntel service or vector search is involved.

For a short smoke only, from the repository root:

P8G_FULLTEXT=0 PYTHONPATH=src .venv/bin/python \
  benchmarks/semantic_mixed_qualification.py \
  --duration-seconds 10 --concurrency 4 --cases 64 \
  --artifact-label working-tree-sdk-installed-native \
  --output ./p8-semantic-smoke.json

Choose an unused output path. The harness refuses to overwrite either the JSON report or its sibling .jsonl timeline. It creates and removes only its own temporary database. Increase the same duration, case-count and concurrency flags for an explicitly scheduled sustained run; a short smoke is not sustained-load acceptance.

The report records actual native artifact paths and SHA-256 digests, selected execution-source fingerprints, benchmark identity and nonsecret execution settings. It checks that those files remain unchanged during measurement. --artifact-label is an operator description, not release certification. Caller-selected PYTHONPATH is preserved: an approved isolated artifact must be selected explicitly and qualified separately from installed artifacts. Nothing is installed or rebuilt. The isolated CPU-expensive HNSW loader experiment is not selected by default; the HNSW fingerprint is context, not evidence that vector loading or search was exercised.

Measurement boundaries

• Query latency includes pattern construction, SDK execution and serialization, but excludes correctness verification. Engine lock waiting is included. P50/P95/P99 are fixed-histogram upper bounds, with 5% bucket spacing above the one-microsecond floor. Overflow is counted explicitly.
• Verified throughput and whole-process CPU/query include the driver, verification and sampling overhead. Calling-thread CPU is also reported separately per workload; it is not all native/background thread CPU.
• Sampled RSS, process CPU, thread counts and per-window workload counters and latency summaries are streamed to JSONL. Histograms and in-memory summaries have fixed size rather than retaining every query or resource sample. Sampled peak RSS is not a continuously observed high-water mark.
• Request failures stop further work, remain explicit failures in the report, and produce a nonzero CLI exit. Setup/preflight or infrastructure exceptions also exit unsuccessfully rather than emitting a successful report.
• The duration stops new submissions, not arbitrary in-flight native execution. Draining can exceed it; the report records the overrun. Existing cooperative query controls do not turn this into a hard process-level timeout.

The current engine holds an exclusive per-engine RLock throughout semantic queries. RDF and cross-model requests therefore serialize with each other. Ordinary LPG traversal does not take that semantic lock and is included to exercise the shared engine alongside them. Reported in-flight overlap means requests overlap, not that their native work runs in parallel or that throughput scales with client count. The harness does not bypass or replace this synchronization.

A passing report gates verified outputs, stationary fixture counts, workload coverage, unchanged artifact fingerprints and (with multiple clients) observed mixed-request overlap. It does not apply unagreed latency, CPU or memory thresholds. This trusted-SDK, stationary synthetic workload does not qualify HTTP/auth overhead, tenant-policy isolation under load, concurrent mutations, diverse document extraction, crash recovery or release packaging. Those remain separate acceptance evidence; no production-completeness claim follows from a smoke run.

The initial local harness checks used four clients, 64 synthetic cases and ten measured seconds per runtime, after the shared-term projection repair:

Runtime	Verified operations	Failed operations
Working-tree SDK + installed native core	17,516	0
Same SDK + previously qualified isolated storage core	17,459	0

Both used the unchanged installed HNSW binary. Each observed four in-flight requests, but average whole-process CPU use was approximately one core; these checks do not establish parallel scaling. Sampled RSS increased during both short runs, from approximately 75 MiB to 79 MiB. That is neither a leak diagnosis nor sustained memory-envelope qualification. The coupled harness, query, evidence, REST and MCP regression selection passed on both runtimes. These are functional smoke observations, not a performance ranking or production acceptance thresholds.

Sustained installed-native observation

A subsequent local run used the unchanged installed storage and HNSW binaries, 16 closed-loop clients, 4,096 stationary cases and 1,800 measured seconds:

PYTHONPATH=src .venv/bin/python \
  benchmarks/semantic_mixed_qualification.py \
  --duration-seconds 1800 --concurrency 16 --cases 4096 \
  --interval-seconds 5 --artifact-label "installed native; local sustained qualification" \
  --output ./p8-semantic-sustained.json

It completed 3,169,940 verified operations with zero failures. All three workloads were exercised, source/native fingerprints remained unchanged, and the retained resource windows reconciled with the aggregate operation, CPU and latency accounting. No build, inference or other heavy qualification work was run alongside the measurement; the host was shared, not dedicated benchmark hardware.

The run observed 16 in-flight requests and approximately one process CPU core. The semantic RLock remained intact: this does not establish parallel semantic execution. Sampled RSS was approximately 135.1 MiB initially and 123.4 MiB finally; the sampled peak was 135.1 MiB. The latter half ranged from approximately 123.4 to 125.2 MiB. These finite observations do not prove leak freedom or an arbitrary-corpus memory bound.

Aggregate RDF and mixed-query P99 histogram upper bounds were approximately 22.1 ms; the largest mixed-query P99 upper bound in a resource window was approximately 48.2 ms. These are bounded-histogram observations from this specific workload, not exact raw quantiles, agreed production SLOs or the product's general query-latency claim. Extraction-model quality, compiled distribution/runtime validation and deployment acceptance remain separate gates.

Explicitly not implemented yet

• Other RDF syntax imports; Turtle, N-Triples, RDF/XML and JSON-LD serializers. N-Quads export covers selected asserted evidence only. The typed models expose to_dict() and validation for the SDK contract; that JSON is not JSON-LD. Generic graph import is not a semantic restore interface.
• SPARQL, property-index selection for LPG conditions, Cypher integration of projections, variable-length paths, OPTIONAL/UNION, aggregation or general filter expressions. LPG edge paths require an anchor; node patterns use direct or traversal-bound selection, scalar equality and explicitly declared IRI-property joins. The SDK conjunctive executor is not a SPARQL engine.
• RDFS/OWL inference, SHACL validation, entity resolution or ontology alignment.
• Assertion assessments, confidence, independent source-occurrence records or inference-support tracking. A named graph supplies context, not automatic evidence verification.
• Storage of empty named graphs independently of assertions.
• Unbounded/streaming RDF import, atomic dataset transactions, new application RBAC roles, raw N-Quads HTTP downloads, and cursor/streaming query APIs. Queries, single-quad mutations and bounded imports use the existing authorization model.

This foundation proves term fidelity, shared RDF/LPG identity, guarded changes, tenant separation, replication and abrupt-process recovery. It does not claim full RDF standards conformance, a complete hybrid query engine or competitive RDF workload performance.