Schema & Data Model

Purple8 Hyper Graph operates on a data-first model — you can write data immediately without defining a schema. The schema is optional, and when you do define one, it can be discovered from existing data rather than declared upfront.

The spectrum

Purple8 operates across a full spectrum from completely schemaless to strictly enforced:

schemaless          warnings only         strictly enforced
    │                    │                      │
GraphEngine()       SchemaValidator(        SchemaValidator(
                      strict_mode=False)      strict_mode=True)

Pole 1 — Fully schemaless (default)

from purple8_graph import GraphEngine

engine = GraphEngine("./data")

# Any label, any property, any shape — accepted without restriction
engine.add_node("n1", labels=["Person"],   properties={"name": "Alice", "age": 30})
engine.add_node("n2", labels=["Document"], properties={"title": "Q4 Report", "pages": 42})
engine.add_node("n3", labels=["Whatever"], properties={"foo": "bar", "nested": {"a": 1}})

No schema file. No DDL. No migration. Just write.

One canonical schema per tenant

Every tenant owns one typed GraphSchema, persisted as a protected Cortex record through the normal durable write path. SDK, REST, MCP and administrative schema reads use that authority. Shared storage does not mean a shared schema: two tenants can define Customer with different required properties in one Purple8 instance.

from purple8_graph.core.models import NodeSchema, PropertySchema, PropertyType

schema = engine.schema  # detached snapshot, not a live mutable configuration
schema.add_node_schema(NodeSchema(
    label="Customer",
    properties=[PropertySchema(name="account_number", type=PropertyType.STRING, required=True)],
    immutable_properties=["account_number"],
))
engine.set_schema(schema)  # durable replacement before acknowledgement

Import the model types from purple8_graph.core.models. Passing schema=... to a new GraphEngine also persists the definition. On reopening, the durable definition wins; a conflicting constructor schema is rejected rather than silently replacing it. Use set_schema() for an intentional change. Mutating a snapshot returned by engine.schema does not change the database.

Existing tenant schema.json definitions are imported on the first writable open that has no native schema record. The original file is retained, but is not read again as an authority after successful migration. Damaged definitions and conflicting legacy endpoint aliases fail explicitly rather than disabling validation. Read-only opens never persist a migration.

The REST/MCP envelope retains its existing shape (node_types, edge_types, and an integer version revision). graph_version carries the SDK schema's version string. Type definitions preserve application/classification metadata, audit fields, embedding declarations and property metadata.

What is enforced

All ordinary engine node/edge creation, updates, upserts and both batch paths use SchemaValidator. REST and MCP writes therefore cannot bypass a declared schema. Cypher property/label removal goes through the same engine checks.

• Required properties, declared property types and enums.
• Required additional labels and immutable node properties.
• Allowed outgoing edge types and source/target label constraints.
• Relationship cardinality: many_to_one limits outgoing relationships, one_to_many limits incoming relationships, and one_to_one limits both. Updating the same edge does not count it twice.

Constrained writes serialize within their tenant authority to make cardinality checks effective under concurrent writes in one process. Unconstrained tenants retain the existing concurrent write path. This does not establish multi-process or distributed transaction isolation.

Schema changes do not rewrite existing records. Use GET /schema/validate or MCP schema.validate to inspect current data with the same validator. Historical immutability cannot be inferred by scanning current data. Application identity, classification and defaults are retained declarations, not new IAM grants, automatic policy stamping or property backfill. Existing authorization and record policies still apply. Unsupported generic constraints expressions are rejected; they are not silently treated as enforced rules.

Schema persistence failures block further writes until the engine is reopened. Schema records cannot be changed through ordinary graph CRUD and are excluded from ordinary graph enumeration/counts. Single-engine backup and replica snapshots carry the schema separately from graph nodes; schema WAL replication has an explicit operation.

Aggregate tenant backup and restore

IncrementalBackupManager.create_backup(tenant_manager) stores separate tenant snapshots, not a flattened graph. Each snapshot includes the tenant definition, canonical schema and shard-local graph DTOs. Every registered non-deleted tenant is included, even if its engine has not been opened or its status is suspended. Tenant definitions retain isolation, quotas, capabilities, IAM configuration and metadata; record properties and system timestamps are preserved.

Restore the archive into a TenantManager, not a single engine:

manifest = await backups.create_backup(source_tenant_manager)
result = await backups.restore_backup(manifest.backup_id, target_tenant_manager, strategy="merge")

The target must have matching embedding dimensions and shard counts. Existing tenant IDs must use the archived isolation level. Restore validates every schema, record shape, identifier, endpoint and shared ownership mapping before replacing graph contents. Missing tenants are created through normal tenant admission, including licensing checks. Tenant schema revisions and metadata are restored rather than regenerated; suspended tenants remain suspended.

With replace=True, only tenants present in the archive have their graph contents replaced. Other target tenants are untouched. replace=False merges graph records, while restoring the archived schema and tenant definition. Vector and named-slot indexes are updated, and enabled BM25 indexes receive publication tasks; BM25 visibility remains asynchronous.

This is a privileged, quiesced maintenance operation, not an atomic distributed snapshot or restore. Stop concurrent traffic, retain the original archive and retry a failed restore before serving the target: a storage failure can leave a partially restored target. In v0.72.0, BackupManager supports streaming gzip/JSONL backups and inspects only the 4-byte P8KM envelope magic header before invoking KMS decryption (P8G_BACKUP_MAX_ARCHIVE_MB ceiling). Aggregate tenant-manager backups still materialize tenant snapshot DTOs in memory; this is not a backup of the separate authentication database, nor does it automatically reseed replicas. Legacy flattened aggregate archives have no reliable tenant ownership mapping and are rejected explicitly rather than guessed. Legacy single-engine graph archives without original-file references remain readable. Archives with file references but no document bundle cannot certify source recovery and are rejected; create a portable backup while the source store is available. These checks do not establish power-loss or distributed failover guarantees.

Portable source evidence: originals and versions

The existing /backup/create and /backup/restore routes automatically attach the application's trusted file store and authenticated tenant scope. Archives carry each referenced File's current original and stored versions, not just its graph record. File IDs, content hashes, RDF terms, quote offsets and graph timestamps are retained; the internal blob_ref storage locator is deliberately mapped into the target file-store root.

SDK callers attach the store explicitly:

from pathlib import Path
from purple8_graph.file_store import FileStore

backups.attach_file_store(FileStore(Path("/srv/source/documents")))
manifest = await backups.create_backup(source_tenant_manager)

backups.attach_file_store(FileStore(Path("/srv/target/documents")))
result = await backups.restore_backup(manifest.backup_id, target_tenant_manager)
print(result["document_files_restored"])

For a single tenant engine, pass its actual scope: backups.attach_file_store(store, tenant_id="alpha"). Aggregate manager backups derive scopes from the archived tenant definitions. A backup containing referenced originals fails explicitly if no source store is attached, a source is missing, its current checksum disagrees with the File record, or required versions are absent. Source paths must stay inside that tenant's store; symlinks and cross-tenant paths are rejected.

Backup format 3.0.0 records portable paths, checksums, sizes, version-file timestamps and de-duplicated file artifacts. Incremental backups can inherit their document artifacts from the same parent that owns the graph data. Original bytes are copied in bounded blocks; an attached KMS codec encrypts bounded frames rather than loading a whole large document for encryption. Keep the corresponding KMS keys available separately; they are not included with the archive. Graph records and archive metadata still materialize in memory, so this is not a bounded-total-RSS backup guarantee.

Restore verifies every bundled file's decoded checksum and size before changing graph contents, even when optional graph-artifact checksum verification is disabled. It rejects missing originals, incomplete required history, malformed encoding, traversal, ownership mismatch and unreferenced document artifacts. Native FileVersion histories are also validated before backup publication and before restore changes any tenant: all sequential versions, exact predecessors, current pointers, chosen bases and matching ownership/ancestry edges must be present and consistent. Missing, duplicated, foreign or cyclic native ancestry is rejected rather than reconstructed or silently relinked. Legacy Files without native history remain supported; an unknown historical base is not invented. Both a same-File v3 based on v1 and an independent linked File retain their exact ancestry, originals, archived records and current-view exclusions after restore and reopen. Equal bytes do not collapse their distinct File/version identities. Older readers do not understand portable file bundles: do not restore these archives with an older graph-only backup reader.

Restore has three explicit strategies; merge is the default:

Strategy	Contract
merge	Append compatible missing native history and retain independent target data. Newer/divergent target history, changed immutable versions and unrepresented extraction produce an actionable HTTP 409 before graph, schema or original-file mutation. Identical replay is idempotent. Compatible extensions retire obsolete current extraction anchors.
replace	Destructively apply the snapshot to the restored tenant's graph and canonical schema, removing target-only timeline records. Choose this explicitly; no branch is selected automatically.
fork	Restore the complete snapshot into explicitly named new dedicated tenants, leaving active target tenants untouched. Publish the new tenants read-only only after graph and original-file hydration completes.

Merge is history-aware, not a generic three-way merge of arbitrary application records. The archived canonical schema and ordinary records with matching IDs are still applied; review these changes during offline maintenance. Existing target tenant administration settings are retained in merge mode. The SDK's explicitly supplied legacy replace=True / replace=False flags still mean replace / merge; do not combine them with strategy.

Fork requires a writable TenantManager and an exact source-to-new-tenant mapping, with no existing destination tenant, graph directory or original-file directory:

result = await backups.restore_backup(
    manifest.backup_id,
    target_tenant_manager,
    strategy="fork",
    fork_tenants={"alpha": "audit-alpha", "beta": "audit-beta"},
)

REST accepts the same strategy and fork_tenants fields on POST /backup/restore. Fork requires tenant administration permission in addition to backup permission. The GDPR erasure cutoff applies to all three strategies, including forks.

Fork keeps internal node, edge, File/version and RDF identities, terms, literals and relationships unchanged. Tenant ownership and original-file paths move to the destination namespace. The original semantic identity/index namespace is retained separately so RDF lookup, joins and projection continue to work without rewriting statement or term IDs. Snapshot provenance and read-only admission persist across restart; interrupted extraction is not resumed. Source live IAM configuration is not copied. Snapshot archives can themselves be forked again; restoring them into a writable timeline is intentionally rejected. Selective promotion is not implemented.

Snapshots provide tenant/storage isolation, not network air-gapping, filesystem tamper protection or cryptographic immutability. Fork currently requires matching dedicated shard geometry; it does not repartition data. Incomplete snapshots fail admission, and failed hydration/publication leaves new tenants suspended with an explicit restore error. Keep them offline for operator recovery; a destination collision is never silently overwritten.

With strategy="replace", stale stored versions of restored File IDs are removed after publication. Merge mode retains unrelated stored artifacts, but rejects incompatible native timeline extensions rather than moving current pointers backwards. Other tenants' files and unrelated unreferenced blobs are not swept; this is not a garbage-collection or erasure operation. Single-tenant shared restores use a scoped engine restore, never a global storage clear.

The target remains offline throughout restore. A file-publication/storage failure after graph restoration can leave a partial target; the operation raises and must be retried from the same archive before serving. It does not return success or silently fall back to the old source path. Separate authentication/policy/configuration stores, external cold-storage objects, replica reseeding, live cloud-provider transfers and power-loss behavior are not qualified by this source-evidence bundle.

Referenced history, retention and explicit source erasure

Routine DELETE /files/{file_id} refuses deletion with HTTP 409 and reason file_history_referenced when another native File version depends on that history, or another File shares the original. The conflict identifies the source's referenced versions, not potentially hidden derivative File IDs. Use the File lifecycle API rather than generic node deletion; managed File roots, versions and erased endpoints cannot be purged through bulk node calls.

An authorized human super_admin can explicitly request:

POST /files/{file_id}/erase
Content-Type: application/json

{"reason": "Approved source erasure request"}

The SDK equivalent is FileVersionStore.erase(file_id, reason=..., actor=...). SDK callers must serialize the operation with other writes to that tenant. The existing human-only graph.delete_node MCP tool supports file=true for routine File deletion, or file=true, erase=true, reason="..." for explicit File erasure. Defaults retain ordinary node-deletion behavior; erase=true without file=true delegates to the protected node-purge endpoint.

Erasure removes the source File, its owned extraction (including archived chunks/pages/tables/rows), and unshared original bytes. A version still needed by a surviving derivative becomes an immutable ErasedFileVersion at its original ID. This is a lineage endpoint, not an available historical version. Its properties contain only file_id, tenant_id, version and lineage_status="erased": no original hash, filename, text, author, timestamp, path or embedding. Outgoing source ancestry is removed; surviving derivatives keep their exact base ID and incoming derivation relationship. When the last dependent is deleted, the unneeded marker can be removed with it.

The response reports retained_lineage_markers, derivatives_to_review and cleanup_completed. Derivatives retain their own bytes, which may still contain identical or personal information; review them independently. Version history identifies an erased base as base_state="erased" and the File Library displays "source erased; lineage only", without implying its content survives. Subject-scoped erasure also uses this lifecycle when a matching record belongs to a File and reports surviving derivatives rather than silently erasing them.

Legal holds on source history, owned content, incident edges or dependent evidence block the operation. File-root holds extend to owned versions and extraction. The generic retention sweeper includes archived history but reports native File/version endpoints as blocked (file_lifecycle_required) instead of silently purging or anonymizing immutable ancestry; held records remain held.

A restore cutoff is persisted before graph mutation. Merge, Replace and Fork all reject archives predating that cutoff; existing erased endpoints also prevent reviving their source content without cutoff metadata. Backups validate minimal marker shape and preserve surviving ancestry, including isolated forks. Keep cutoff metadata with operational configuration; it is not a replacement for protecting old archives and external copies.

Original-byte cleanup is journaled before graph publication. A cleanup failure is an explicit error with graph_erased=true, not success: retry the same erasure request. Writable tenant admission retries committed cleanup before serving; a failed graph publication retains the originals and restores graph history, or requires explicit recovery if rollback cannot be confirmed. Read-only snapshot engines cannot resume destructive cleanup.

This is scoped source erasure, not physical secure overwrite or a blanket GDPR-compliance guarantee. Independent RDF facts, shared entities, audit records, other-tenant snapshots, old backups/media/WAL/slabs and external systems require separate policy and review.

Dedicated multi-shard File lifecycle

Native File lifecycle qualification covers shared, dedicated single-shard and dedicated multi-shard tenants. The multi-shard cases use real native storage, including REST document commit, RDF/LPG preservation, replacement visibility, legal holds, tenant-isolated erasure and restart recovery.

Each node has one owner shard. Cross-shard ancestry edges have matching copies on both endpoint shards; endpoint nodes are not duplicated. WAL replay preserves these remote-endpoint edge records. Aggregate backups export canonical outgoing edges; Merge, Replace and Fork restore both endpoint copies and retain logical, not physical-mirror, restored-edge counts. Restore still requires matching shard geometry and does not repartition.

A failed revision publication restores the previously current graph and leaves its original bytes unchanged. New immutable bytes may already be staged: reload the File and retry the identical payload. Unpublished bytes do not become retrievable File history. A conflicting payload in an occupied immutable version slot requires explicit operator recovery, not an overwrite.

Current retrieval across dedicated shards

Default and named-slot vector searches merge native per-shard candidates into a global top-k. Archived records are excluded inside native ANN eligibility, before they can consume a result window, even if stale index entries remain. Labels, property filters and node-ID prefixes remain effective. A named slot may exist on only some shards; its declared dimension comes from the shards owning that slot, not an unrelated default dimension.

Hybrid retrieval selects global vector seeds, expands through logical owner adjacency across shards, and returns native HybridSearchResult records with scores, hop distances and seed paths. Optional graph reranking operates on the global candidate set, not independently on each shard. Vector APIs return VectorSearchResult records; use the result types' to_dict() methods in SDK integrations. REST and the graph/hybrid RAG retrievers use the same native paths.

Traversal applies current visibility and record read restrictions before admitting neighbors: archived or hidden records cannot bridge to otherwise reachable context. It supports native edge filters, direction, paths and caller-supplied max_nodes budgets. Depth and execution deadlines retain native safeguards; a result count is not a new hard cap on all graph expansion.

Once a File is replaced, old extraction remains absent from live retrieval while the current extraction is missing. There is no fallback to archived content, even after warming search caches. Explicit history reads use a separate vector-cache mode; request-specific blocked-record results are not shared through that cache. Invalid filters and shard search failures propagate as errors, not empty or partial-success results.

The facade accepts native query_embedding= and the legacy embedding= keyword, but not both. Native traversal positional order is (start_node_id, edge_type, depth, direction); the old depth-first positional order is also recognized. New integrations should use named arguments.

Current keyword retrieval across dedicated shards

Integrated fulltext_search() uses tenant-wide indexed document counts, average document length and query-term frequencies for BM25. Shard-local scores are not merged as though they were comparable: each shard scores with the same corpus statistics before global top-k selection. Tokenizer/scorer settings must match, and indexing must be consistently enabled or disabled across shards; inconsistent settings and shard failures are explicit errors.

Current-record and record-access checks run before top-k selection, so stale postings cannot fill the result window with archived or erased records. The eligibility-enforced search path streams matching postings and retains bounded top-k heaps instead of allocating a full candidate set and score list. Index locks use a consistent order while gathering statistics and scoring, preventing concurrent updates from invalidating posting iteration. Statistics describe the indexed corpus: asynchronous posting removal can still affect relevance scores temporarily, but cannot make archived content eligible.

REST /search/text additionally applies row/field visibility and checks that the term still matches visible text before BM25 top-k selection. Its exact-ID shortcut is discovery, not permission to retrieve an archived record; explicit point/history APIs remain separate. If the index has no usable match or is disabled, fallback streams tenant-owned current records through native iteration/label indexes rather than materializing a label or repeatedly rescanning logical pages. It retains only top-k scored results and reports truncated=true when the endpoint's current-record scan cap is exceeded. Archive records do not consume that cap.

The integrated RAG keyword adapter delegates to this search API without copying a physical shard index or rebuilding the corpus. Native shard tests cover global score equivalence, tenant isolation with identical IDs, field policies, stale postings, replacement gaps, REST discovery, concurrent index updates and recovery after reopening.

Graph-first shard and segment selection

The SDK's graph_guided_search() traverses the logical graph across shard boundaries before ranking vectors. It keeps at most max_candidates graph nodes, including the seed. Edge types, direction and tenant prefix constrain the walk; current-record/access visibility applies before admission. Labels are checked against bounded candidates, without materializing an entire label. Only physical owners of eligible candidates receive vector searches, and only their candidate-owning segments are probed. Exact ownership overrides centroid heuristics; an unrelated active segment is not automatically included. Non-segmented indexes use their filtered base ANN directly for bounded searches, avoiding a corpus-wide quantized pre-screen. Logical storage and owner lookups read durable node records without loading vector segments merely to hydrate embeddings. Explicit public point reads retain their existing embedding-hydration behavior. Eligibility checks use live metadata projections, omitting primary and named vector reconstruction while preserving labels, properties and system times. When the native core exposes get_node_metadata, hot reads also omit reserved vector fields before JSON serialization. Older binaries retain the compatible Python projection; native read errors propagate rather than silently falling back. Cold slabs are still decrypted, but packed vectors are not unpacked for these checks. Owner-only probes use the hot/cold existence indexes, retaining the fallback for migrated records. Returned nodes and traversal results remain complete, not metadata projections. Graph-first candidate discovery uses a private metadata-only traversal mode, with a separate key in the existing bounded traversal cache. Public traversals retain full primary/named embeddings; complete nodes are materialized for vector-ranked result candidates, not every BFS visitor. The hnswlib backend's pre-filtered per-segment result requests are capped by known candidate ownership, without reducing the ANN search width. They also return at most the query's global candidate-pool size per segment, rather than over-fetching results that the global top-k merge must discard. This avoids repeated impossible requests and redundant per-result eligibility reads. Post-filtering backends retain their over-fetch requests; the same cap would not preserve their candidate coverage. Native hnswlib searches with Python eligibility callbacks process selected segments inline, avoiding nested worker contention. Unfiltered searches and other backends retain parallel fan-out and copied request context. The configured worker count remains an upper bound; a limit of one uses no executor.

A valid sparse seed retains the native behavior: widen once when eligible, then use explicit full-vector fallback if fewer than min_candidates graph nodes were reached. This threshold is evaluated before the result-label filter; it is not a minimum number of eligible embedded results. Fallback still applies current/access/tenant/label checks and reports fallback_used=true. Missing, archived or access-hidden seeds return no current results, not an unrelated full-search fallback. If a replacement has no current extraction, neither search mode substitutes archived content. Explicit history_reads() remains separate and does not bypass access controls.

Results retain the native GraphGuidedSearchResult shape, global top-k ranking and hop distances. segments_searched counts actual non-empty segment-search calls across selected shards, including fallback; total_segments describes all shard indexes, including empty active segments. These calls are uncached, so metrics do not infer probes from candidate mappings or cached vector results. Selected-shard failures propagate rather than returning partial success. Native tests verify physical probes, cold-segment loading, restart, budgets, scope and archival gaps. No fixed speedup or large-corpus load claim follows from these correctness tests; benefit depends on graph and segment locality.

Retrieval typing and reproducible load qualification

The retrieval boundary is strict-type-checked across the native engine, vector backends, policy wrapper, tenant manager, sharded facade and shared retrieval helpers. Install development dependencies, including the declared third-party stubs, then run:

.venv/bin/python -m mypy --follow-imports=silent \
  src/purple8_graph/core/engine.py src/purple8_graph/core/vector.py \
  src/purple8_graph/security.py src/purple8_graph/tenancy.py \
  src/purple8_graph/sharding.py src/purple8_graph/core/retrieval.py

This checks those six modules, not the entire repository or the installed native extension internals. Backend contracts include node-ID eligibility predicates, batch thread selection and distance-space metadata. Quantized pre-screening applies eligibility before selecting its candidate window, so archived vectors cannot crowd current records out of that window.

The native sharded load harness runs ordinary and graph-first retrieval over identical synthetic topic vectors and held-out queries, with exact global and candidate-subgraph cosine top-k references. It instruments actual physical shard and segment searches and verifies real File replacement gaps during overlapping queries. For example, from a source checkout:

PYTHONPATH=src P8G_VECTOR_BACKEND=hnswlib \
  OPENBLAS_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 OMP_NUM_THREADS=1 \
  .venv/bin/python benchmarks/graph_guided_qualification.py \
  --nodes 100000 --queries 12 --placement clustered \
  --output benchmarks/results/graph_guided_clustered_100k_20261022_metadata_reads.json

Repeat with --placement scattered to remove favorable segment locality. Results include workload settings, actual per-segment backend implementation, index parameters, configured worker cap, actual over-fetch policy, per-query latency/recall/probes, concurrent throughput and process RSS. Query/traversal caches and full-text indexing are disabled for this comparison. The application-cold phase limits the sealed-segment LRU; it does not flush OS disk caches. RSS includes the benchmark's reference arrays and instrumentation. These finite synthetic runs establish a reproducible local baseline, not a semantic relevance benchmark, production SLA or proof of corpus-independent RSS.

Measured local baseline

The completed local comparison used 100,000 embedded documents, 64 dimensions, four shards, 100 segments, 250 documents per topic, k=10, and 12 held-out queries per scenario. It ran on macOS arm64 with 10 logical CPUs and Python 3.12.12. The explicitly selected and recorded vector backend was the embedded C++ hnswlib implementation (purple8_graph.core.vector.VectorIndex), not an external database and not the opt-in Rust HNSW backend. Cortex remained the durable storage engine. The over-fetch policy stayed at its legacy default.

Before-fix outputs remain in benchmarks/results/graph_guided_{clustered,scattered}_100k_20261022.json. Final outputs use the corresponding _qualified.json suffix. Intermediate _raw_reads.json and _bounded_k.json outputs retain the individual iterations. The first pass's graph-constrained retrieval measurements were:

Placement	Workload	Before P50 (ms)	Final P50 (ms)	Final P95 (ms)
Clustered	Warm, one worker	80.239	14.659	15.571
Clustered	Four workers	536.944	69.238	97.682
Clustered	Application-cold LRU	80.857	16.880	17.268
Clustered	Concurrent replacement	648.940	120.497	149.271
Scattered	Warm, one worker	1067.128	116.276	119.866
Scattered	Four workers	4482.548	500.200	541.124
Scattered	Application-cold LRU	11808.894	178.864	191.158
Scattered	Concurrent replacement	5524.972	660.254	760.867

Every measured query retained global and candidate-subgraph recall@10 of 1.0, with no graph-first fallback. Graph-first queries probed one segment on one shard for clustered placement and 25 segments on one shard for scattered placement; ordinary vector queries probed all 100 segments on four shards. Each final run verified four empty current-extraction gaps during real File replacements. Six clustered and four scattered timed queries overlapped the writer, but those timed queries addressed independent stable topics, not the replaced File. These checks do not establish all possible replacement races.

That pass exposed material limits. Four-worker throughput was lower than single-worker throughput: 44.121 versus 61.197 queries/s clustered, and 7.758 versus 8.575 queries/s scattered. Ordinary full-fan-out vector search had warm P50 of 1871.981 ms clustered and 1887.990 ms scattered. Peak process RSS was 1110.64 MiB clustered and 1129.75 MiB scattered, including benchmark arrays and instrumentation; these finite points do not establish a memory-scaling law. With only 12 queries per scenario, P95/P99 values describe these samples, not a robust production tail-latency qualification.

Filtered fan-out follow-up

Repeating the same workload with a single segment worker isolated avoidable contention in filtered searches. The production path now runs native Python eligibility callbacks inline and bounds each segment's returned result pool without narrowing the configured ANN exploration width. It does not cache eligibility decisions or remove current-record/access checks.

The final follow-up explicitly retained a worker cap of 10 and the legacy over-fetch policy. Outputs use the _filtered_fanout.json suffix; _one_segment_worker.json and _inline_filters.json retain the isolation runs. Compared with the preceding _qualified.json pass:

Workload	Prior P50 (ms)	Follow-up P50 (ms)	Follow-up P95 (ms)
Clustered, warm graph-first	14.659	13.570	13.774
Scattered, warm graph-first	116.276	23.596	24.579
Scattered, four-query graph-first	500.200	271.254	291.766
Scattered, application-cold graph-first	178.864	75.737	79.113
Clustered, warm full-vector	1871.981	864.009	870.038
Scattered, warm full-vector	1887.990	899.662	959.321

All measured queries again retained both recall@10 references at 1.0, with unchanged physical segment counts and no graph-first fallback. Both runs verified four empty replacement gaps; 12 clustered and four scattered timed stable-topic queries overlapped replacement. Backend identity and configured worker caps are recorded in each output. Optional --profile PATH captures one extra query after timed/resource measurements; --profile-mode guided|vector selects that query without changing measured workloads.

Concurrency scaling is still not accepted: graph-first four-query throughput was 53.009 versus 74.611 queries/s single-worker clustered, and 15.169 versus 42.221 scattered. Full-vector four-query P50 remained 5086.074 ms clustered and 5317.167 ms scattered. Peak process RSS was 1123.36 MiB and 1122.11 MiB, respectively, including the harness. These results close specific contention and redundant-read costs, not the remaining end-to-end concurrency gap or production performance acceptance. The earlier small-sample, synthetic-data, application-cache and independent-replacement-topic limitations still apply.

Live metadata and indexed ownership follow-up

The next pass removed vector reconstruction from eligibility reads and record decoding from owner-only probes. Metadata is read again at each eligibility check, not cached: a regression test archives a node during search and verifies that it cannot survive the final eligibility check. Tests cover hot records, packed/encrypted cold slabs, malformed policy fields, migrated ownership and complete embeddings on returned results.

Final outputs use _metadata_reads.json; _live_metadata.json retains the projection-only iteration. The repeated scattered run is retained separately as graph_guided_scattered_100k_20261022_metadata_reads_repeat.json. Against the preceding _filtered_fanout.json pass:

Workload	Prior P50 (ms)	Metadata/ownership P50 (ms)
Clustered, warm graph-first	13.570	10.640
Scattered, warm graph-first	23.596	18.306 (repeat: 19.707)
Scattered, four-query graph-first	271.254	241.096 (repeat: 240.971)
Scattered, application-cold graph-first	75.737	67.169 (repeat: 67.039)
Clustered, warm full-vector	864.009	772.820
Scattered, warm full-vector	899.662	933.559 (repeat: 799.130)

The scattered full-vector slice includes an adverse first result; it is not discarded in favor of the faster repeat. These short runs do not establish a uniform full-vector latency improvement.

A separate, smaller high-dimensional slice used 10,000 documents, 768 dimensions, 12 physical segments, the same topic size and 12 queries per scenario. graph_guided_scattered_10k_768d_20261022_before_metadata.json and the corresponding _metadata_reads.json retain that comparison:

High-dimensional workload	Before P50 (ms)	After P50 (ms)
Warm graph-first	76.960	59.091
Four-query graph-first	218.938	162.880
Warm full-vector	174.992	152.098
Four-query full-vector	505.831	355.810

Both exact recall references remained 1.0 for every measured query in these runs, with no graph-first fallback. Physical segment selection was unchanged: one/25 of 100 segments for clustered/scattered graph-first at 100K, and three of 12 for the scattered 768-dimensional slice. Each run verified four empty replacement gaps, with stable-topic queries overlapping the writer.

This remains an incremental improvement, not accepted end-to-end scaling. At 100K, four-query graph-first throughput was 63.486 versus 95.866 queries/s single-worker clustered and 16.814 versus 54.000 scattered. The smaller 768-dimensional slice did scale positively, from 16.707 to 24.518 queries/s, but that does not qualify the larger workload. Peak process RSS was 1117.44/1119.62 MiB for the 100K placements. The high-dimensional peak increased from 920.42 to 933.09 MiB; no process-memory saving is claimed.

For this Python-only pass, the existing compiled core still serialized complete hot records, including folded vector fields. The Python projection avoided their reconstruction and validation, not their native serialization or JSON parsing. No Rust extension was rebuilt for that pass. The following isolated experiment qualifies native projection separately.

Isolated native metadata qualification

The native core now provides a borrowed get_node_metadata projection, excluding only __embedding__ and __embeddings__ before serialization. It preserves all other properties (including legitimate embedding and embeddings source fields), labels, IDs and timestamps. Serialization holds the node's read guard, so concurrent updates cannot mix labels from one revision with properties from another. The Python adapter resolves the reader from the current core on each call, preserving compatibility when the core is replaced. Full point reads, cold-slab decoding and persisted record formats are unchanged.

Qualification used two isolated release builds with the same compiler, default features, Python interpreter and offline locked dependencies. The baseline snapshot includes the existing local Rust changes; the candidate differs only by the metadata API in engine.rs. Neither build replaced the shared installed binary, whose SHA-256 was checked against its pre-build fingerprint. The harness records the actual native extension path, SHA-256 and metadata capability in every result, alongside the unchanged hnswlib vector backend. These candidate timings are not active in the shared runtime until the native artifact is separately promoted.

Outputs are graph_guided_{clustered_100k,scattered_100k,scattered_10k_768d}_20261022_native_metadata_{baseline,candidate}.json. The workload controls match the preceding pass: four shards, 12 queries per scenario, four concurrent query workers, legacy over-fetch, a segment-worker cap of ten, single-threaded BLAS and the same synthetic corpus seed. Builds, tests and timed runs did not execute concurrently. Scattered workloads were repeated in reverse candidate/baseline order; both original and _repeat.json results are retained.

Workload	Isolated baseline P50 (ms)	Candidate P50 (ms)
100K x 64, clustered warm graph-first	10.648	7.710
100K x 64, clustered warm full-vector	788.656	405.544
100K x 64, scattered warm graph-first	19.858	16.536
100K x 64, scattered warm full-vector	795.600	430.455
10K x 768, scattered warm graph-first	59.202	23.701
10K x 768, scattered warm full-vector	156.089	18.994
10K x 768, scattered four-query graph-first	170.358	102.771
10K x 768, scattered four-query full-vector	365.942	123.237

The repeated scattered warm graph-first pairs were 20.136 -> 16.927 ms at 100K and 58.680 -> 22.578 ms at 10K x 768. Scattered application-cold graph-first was adverse in the first 100K pair, 69.115 -> 73.292 ms, but improved in the repeat, 71.158 -> 64.578 ms. Neither result is discarded; these short runs do not establish a uniform cold-read improvement.

All measured queries retained both exact recall references at 1.0 with no graph-first fallback. Guided probes remained one/25 of 100 physical segments for clustered/scattered 100K, and three of 12 at 10K x 768. Each run verified four empty replacement gaps while stable-topic queries overlapped replacement. A large-vector regression fixture (a 4,096-dimensional primary vector and two named slots) measured 147,909 -> 394 serialized bytes; tests require metadata below 1 KiB and below 1% of the full payload, without discarding governance.

Concurrency remains unqualified. Candidate single-worker versus four-query graph-first throughput was 128.125 versus 75.234 queries/s clustered and 60.241 versus 17.030 scattered at 100K. Even the smaller 768-dimensional candidate now has negative scaling: 42.203 versus 38.866 queries/s (repeat: 44.459 versus 38.134), although both absolute throughput figures improve on the baseline's 16.881/23.000. Faster individual queries do not establish useful concurrent scaling. Nor are fewer probes a universal latency win: warm full-vector search beats graph-first on this smaller candidate slice. This change does not alter graph-first selection or fallback policy.

First-pair peak process RSS was essentially unchanged: 1118.59 -> 1118.58 MiB clustered, 1120.47 -> 1120.12 MiB scattered, and 951.03 -> 950.95 MiB for 10K x 768. Repeated peaks varied, especially for the smaller corpus; no process-memory saving is claimed.

The candidate passed 891 coupled retrieval, lifecycle, tenant and storage regressions, including native payload/update consistency, legacy dispatch, explicit error propagation and complete result embeddings. The untouched installed binary passed 889 with the two native-API-only tests explicitly skipped. Six-module strict typing and focused lint also passed. These are finite local checks, not production SLA, soak, whole-repository typing or Rust vector-backend qualification. The broader storage-module import check still reports existing import ordering and an unused shutil import outside this change; it is not claimed clean.

Metadata-only candidate traversal follow-up

Profiling the isolated native candidate found another discarded cost: graph-first BFS decoded full vectors for every visited node even though candidate selection uses only node IDs and path lengths. The candidate walk now requests metadata at queue processing as well as neighbor admission. It retains live visibility checks, node/edge scope, budgets and cancellation. A regression archives a bridge after admission but before processing and verifies that it cannot expose its descendants.

The projection is internal; public traversal results and search results remain complete, including named embeddings. Projected and full traversals use distinct keys in the same bounded cache. Tests exercise both cache population orders; no eligibility-decision cache or new cache pool is introduced.

This comparison reuses the same isolated native artifact on both sides, without another Rust rebuild or promotion. The baseline Python package is a local copy with only the changes to engine.py, retrieval.py and sharding.py reversed. The harness records paths and SHA-256 hashes for those loaded implementation files, and checks compare them against the source snapshots.

Those fingerprints caught a setup error: an imported standalone benchmark helper prepended repository src before SDK imports, overriding the requested baseline source. The first six _traversal_projection_{baseline,candidate}.json files are retained but are invalid A/B comparisons and are not used below. The harness now resolves the caller-selected SDK before importing that helper. A fresh-interpreter regression protects that ordering. Only the source-verified _traversal_projection_{baseline,candidate}_qualified.json outputs and their _qualified_repeat.json counterparts support this comparison.

The corpus, backend, worker, cache and reference controls match the native metadata pass. Runs were sequential, with scattered repeats in reversed candidate/baseline order. In one extra profiled query per first-pair run, complete native record serializations fell from 261 to 10: metadata-only BFS left only the ranked candidates requiring full records. This is a call-count measurement, not a claim about profiler-derived CPU percentages.

Graph-first workload	Baseline P50 (ms)	Projected walk P50 (ms)
100K x 64, clustered warm	7.316	6.147
100K x 64, clustered four-query	54.837	47.338
100K x 64, scattered warm	15.282	14.050
100K x 64, scattered four-query	209.701	208.512
10K x 768, scattered warm	22.166	8.197
10K x 768, scattered four-query	98.942	73.528
10K x 768, scattered application-cold	26.355	14.198

Repeated scattered warm pairs were 15.601 -> 14.232 ms at 100K and 21.300 -> 8.914 ms at 10K x 768. The repeated high-dimensional four-query pair was 102.988 -> 65.846 ms. Full-vector search is an unchanged control; its timing variation is not attributed to this BFS change.

Adverse samples are retained. Scattered 100K application-cold P50 was nearly unchanged, 60.057 -> 59.720 ms, while P95 worsened from 60.816 to 86.521 ms and QPS fell from 16.619 to 15.200. The repeat improved P95 from 64.784 to 59.598 ms. Mixed-query P50 also worsened slightly in the first 100K pair, 380.413 -> 384.803 ms, before improving in the repeat. These short samples do not establish uniform cold or mixed-load gains.

Every measured query in the valid comparisons retained both exact recall references at 1.0 and no fallback. Physical graph-first probes stayed one/25 of 100 segments at 100K and three of 12 at 10K x 768. Every valid run verified four empty replacement gaps, with stable-topic queries overlapping the writer.

Concurrency is still not accepted: candidate single-worker versus four-query graph-first throughput was 160.424/87.831 queries/s clustered, 70.099/19.875 scattered, and 119.895/55.649 at 10K x 768. Lower service latency is not positive concurrent scaling. The remaining bottleneck is not claimed diagnosed or resolved by removing BFS vector materialization.

Peak process RSS was 1118.44 -> 1114.81 MiB clustered and 1117.14 -> 1117.95 MiB scattered. The smaller high-dimensional pair measured 959.38 -> 937.22 MiB (repeat: 952.98 -> 938.81 MiB). These fixed-corpus observations do not establish a general process-memory or scaling guarantee.

The final coupled suite passed 897 tests, including cache-mode isolation, complete public results, bridge archival, shared/native/sharded visibility and benchmark source selection. The shared installed native binary is unchanged; these isolated-artifact timings are not a promoted production baseline.

Bounded native metadata scheduling follow-up

A separate live-read probe isolated a tiny native-call scheduling cost before HNSW was involved. With the previous isolated metadata implementation, raw reads measured 2.64-2.69 million/s with one worker but only 0.40-0.46 million/s in aggregate with four workers. SDK metadata reads measured 404-411 thousand/s versus 207-220 thousand/s. A deliberately non-live static-decoding control stayed near 540 thousand/s at either worker count; it is attribution evidence, not a proposed visibility cache.

The explicitly approved isolated Rust candidate avoids detaching/reacquiring the GIL for small, immediately available metadata records. It uses a nonblocking native read-lock attempt and a conservative 4096-byte/128-value budget, including a structural reserve, escaped-string charges, nesting and HashMap-capacity checks. Reserved folded vectors are skipped without walking their contents. Contended, large, deeply nested or oversized-capacity records drop any acquired guard and use the original GIL-detached path. Both paths serialize the same live record under its read guard, preserve user properties and propagate failures. There is no eligibility cache or changed ANN algorithm.

The candidate's raw live reads measured 2.61-2.66 million/s with one worker and 2.60-2.62 million/s in aggregate with four. SDK live reads measured 406-409 thousand/s and 404-408 thousand/s respectively. This removes the microbenchmark's contention collapse; flat aggregate throughput is not four-core scaling.

End-to-end comparisons use identical current Python source hashes and the retained previous native metadata artifact versus the bounded-GIL artifact. The corpus and retrieval controls above are unchanged. The harness additionally records its own source hash, calling-thread CPU and optional segment-guard acquisition/held times. Calling-thread CPU excludes delegated worker CPU. Guard acquisition includes scheduler/GIL wake-up, and held time includes HNSW work, eligibility callbacks and their waits; neither is a pure native mutex/compute breakdown. Hot-cache and native metadata-map locks are not timed.

The first six _metadata_gil_{baseline,candidate}.json files are retained but rejected for performance qualification: observer cleanup references retained evicted segment indexes, inflating application-cold memory and affecting timing. The corrected observer tracks segments weakly, restores still-live guards and has an eviction/cleanup regression. Accepted diagnostic files end in _metadata_gil_{baseline,candidate}_qualified.json. Independent runs without guard diagnostics end in _plain.json and reverse candidate/baseline order. The separate live metadata microbenchmark did not use that observer.

Graph-first workload, guard diagnostics disabled	Previous native P50 (ms)	Bounded-GIL P50 (ms)
100K x 64, clustered warm	6.246	6.050
100K x 64, clustered four-query	47.540	38.025
100K x 64, scattered warm	14.494	14.106
100K x 64, scattered four-query	209.660	193.414
10K x 768, scattered warm	8.081	8.631
10K x 768, scattered four-query	64.671	61.109

The corrected diagnostic four-query pairs were 47.673 -> 38.640 ms clustered, 215.089 -> 193.718 ms scattered, and 62.588 -> 58.477 ms at 768 dimensions. Their mean calling-thread CPU was 17.056 -> 11.861 ms, 89.393 -> 87.804 ms and 24.809 -> 22.128 ms respectively. In scattered graph-first queries, acquisition P50 was about 0.01 ms while summed guard-held P50 remained about 187 ms in the candidate. The measured delay is not predominantly at guard acquisition; the protected HNSW/callback interval still needs attribution. These independently aggregated medians are not an additive latency decomposition.

Ordinary full-vector retrieval also uses live metadata eligibility checks; its four-query P50 improved in both corrected comparisons, without changing ANN search width or physical probes. Improvements are not uniform: the 768-dimensional warm graph-first case worsened in both comparisons (diagnostic: 8.400 -> 8.889 ms). Diagnostic scattered mixed P50 worsened 388.728 -> 417.854 ms, although the reverse-order plain pair improved 393.041 -> 372.296 ms. Plain scattered application-cold P50 also worsened 58.956 -> 59.683 ms. All adverse observations remain in the raw outputs.

Across the 12 accepted runs, all 1008 measured queries retained both exact recall references at 1.0, no fallback and unchanged physical probes: one/25 of 100 segments at 100K, three of 12 at 768 dimensions. All runs verified four empty replacement gaps with positive writer/query overlap. Corrected diagnostic peak process RSS was 1121.72 -> 1119.11 MiB clustered, 1114.83 -> 1116.41 MiB scattered and 943.00 -> 912.08 MiB at 768 dimensions; the plain high-dimensional pair instead increased 917.27 -> 951.45 MiB. There is no general memory-saving or corpus-scaling claim.

Concurrency acceptance remains open. Plain candidate graph-first single/four worker throughput was 163.191/111.058 queries/s clustered, 70.005/20.027 scattered and 113.826/64.739 at 768 dimensions. These are measured improvements over the previous artifact, not positive scaling. Tiny native owner/existence probes and HNSW eligibility callbacks remain investigation targets, not established sole causes.

Validation passed 907 coupled Python tests and four focused Rust tests, including a deterministic contended-read/GIL-inversion regression. The untouched original binary passed 147 targeted tests with six native-API-only cases skipped. Six-module strict typing and focused lint passed. The shared installed native binary's SHA-256 is unchanged; the new artifact has not been installed or promoted. These finite synthetic local checks do not establish a production SLA, soak, Rust vector-backend qualification or whole-repository typing cleanliness.

HNSW callback-boundary attribution

The next pass changes only benchmark diagnostics, not production retrieval or native artifacts. --diagnose-callbacks optionally counts/times actual hnswlib.Index.knn_query calls, Python filter bodies and SDK storage-existence probes. Instrumentation is scoped, restored on query failure and disabled by default. Body timers begin after entering Python, so they exclude callback entry/GIL acquisition. They include observer bookkeeping; the residual between the ANN envelope and timed bodies includes HNSW, binding, scheduling/GIL waits and instrumentation, not exclusively native compute or GIL wait.

Six new source/artifact-verified runs use the retained bounded-GIL metadata candidate. Files end in _callback_attribution_{observed,plain}.json; plain controls disable both callback and segment-guard diagnostics. All 504 measured queries retained both exact recall references at 1.0, no fallback, unchanged physical probes and 24 empty replacement gaps with positive writer overlap.

Graph-first four-query workload, diagnostics enabled	Mean HNSW filter calls	Mean ANN envelope (ms)	Mean timed filter bodies (ms)	Mean existence-probe time (ms)
100K x 64, clustered	999.6	21.968	1.637	9.387
100K x 64, scattered	25000	330.599	13.215	4.972
10K x 768, scattered	2500	45.540	2.778	5.799

Each guided query made 251 storage-existence probes, with none inside the HNSW call in these fixtures. Those probes are a separate cost from the callback-heavy ANN interval. A disjoint-hot-ID live microbenchmark confirms their scheduling sensitivity: native present-node existence measured 13.93-15.74 million/s with one worker versus 1.84-2.07 million/s aggregate with four. SDK probes measured 10.40-11.25 versus 1.53-1.63 million/s. Missing-node probes also degraded; live deletion/reinsertion was checked. The same artifact's metadata control remained near 2.61-2.65 million/s with one worker and 2.63 million/s with four. This does not make existence probes the sole or dominant end-to-end bottleneck.

Per-callback instrumentation is intrusive, especially for cheap rejection callbacks. Observed/plain scattered warm P50 was 26.451/15.222 ms and four-query P50 was 357.049/258.284 ms. Plain clustered/high-dimensional four-query P50 was 37.423/60.948 ms. These are diagnostic controls, not an optimization A/B comparison; the scattered plain sample is also slower than the preceding 193.414 ms sample and is retained, not replaced. No new default-runtime performance improvement is claimed.

A separate boundary microbenchmark runs actual installed hnswlib over four independent read-only indexes, removing shared index-guard contention. Each index holds 1000 vectors of 64 dimensions, with k=10, ef=1000 and one native query thread. This wide setting visits all fixture nodes; it characterizes callback overhead, not sublinear search complexity. Three repeats alternate one/four-worker order:

Boundary control	One worker (queries/s)	Four workers aggregate (queries/s)
Native HNSW, no filter	20742-23020	61791-85288
Python accept-all callback, not live policy	7202-7536	1108-1119
Python static candidate mask, not live policy	7654-8021	1077-1104

Accept-all and unfiltered cases have identical queries, search width and expected top-k output. Both callback controls invoke Python 1000 times per probe; the mask uses its own exact subset reference. No metadata reads or owner lookups occur in this microbenchmark. Callback-boundary overhead can therefore cause a major concurrent collapse independently of live metadata decoding and shared segment guards. It does not separate every binding cost from GIL/scheduler wait or prove a production throughput multiplier.

Neither non-live control is a proposed retrieval implementation. Removing live filters, caching eligibility, changing ANN width, serializing requests or silently substituting exact scoring is not an accepted fix. A possible next experiment is a native candidate-membership gate before the Python callback, retaining live policy callbacks for selected candidates and final result checks. The installed hnswlib query interface exposes a callback but no native candidate-mask argument; this requires a separate design/build decision, not a hidden toggle. The opt-in Rust backend's existing allow-set does not itself provide live Python policy parity.

The targeted instrumentation/retrieval suite passed 113 tests, including callback error propagation, disabled/unfiltered operation, native-method restoration after failure and weak segment lifetime. No new native build, backend promotion or shared binary installation occurred in this pass.

Isolated native candidate-gate qualification

The explicitly approved prototype adds a native candidate-ID gate to an isolated hnswlib binding. It rejects graph non-candidates before the C++/Python boundary; selected IDs still invoke the existing live Python predicate, including access/current-content checks, and final result checks remain unchanged. It does not replace HNSW with exact scoring, change search width, weaken writer-safe locks or cache policy decisions.

The SDK uses a typed capability guard for the prototype's supports_native_candidates marker. Stock bindings retain the existing Python filter path. Candidate IDs are grouped by physical segment once, translated under that segment's existing lock and copied into a query-local native set. Additional candidate storage is bounded by the existing candidate limit, not the corpus size. Post-filter/opt-in Rust backends and ordinary full-vector queries do not silently acquire a different retrieval algorithm.

Matching baseline/candidate bindings were built from the same upstream source, headers, compiler settings and isolated build dependency. The HNSW algorithm headers are identical; the candidate difference is confined to the binding. Both builds also have a common exception-buffer ownership fix: upstream allocated result arrays before search without retaining cleanup ownership when a filter failed. Both now keep those buffers in RAII owners until successful NumPy capsule transfer. This common fix is not attributed to candidate-gate speedups. Across 10000 post-warmup failed queries, the finite cleanup probe observed zero virtual-memory growth and at most about 1.1 MiB RSS growth. That is a failure-allocation regression check, not a general RSS guarantee.

The harness now fingerprints the loaded HNSW artifact/capability as well as the storage core, its own source and four retrieval implementation files, including the vector adapter. All comparisons use identical current SDK sources and the same retained bounded-GIL storage-core artifact.

Graph-first four-query workload, diagnostics disabled	Matching baseline P50 (ms)	Native gate P50 (ms)
100K x 64, clustered	37.543	28.413
100K x 64, scattered	198.698	40.976
10K x 768, scattered	55.780	36.006

Reverse-order plain repeats were 36.755 -> 30.841 ms clustered, 188.075 -> 37.922 ms scattered and 56.351 -> 35.508 ms at 768 dimensions. Diagnostic scattered callbacks fell from 25000 to 250 per query, clustered from about 1000 to 250 and high-dimensional from 2500 to 250, with the same physical probes and ANN width. These counts cover callbacks inside HNSW; the existing Python post-search revalidation also remains.

A small-fixture regression exposed a benchmark-startup race: fast readers could finish before the separately scheduled writer began. Mixed workers now meet at a start barrier outside query/writer service-time measurements. The real timed-overlap assertion remains; barrier waiting is not substituted for measured overlap. Earlier valid results retain their exact frozen harness snapshot and positive overlap checks. A further plain pair with synchronized startup measured four-query P50 of 32.708 -> 32.758 ms clustered, 180.399 -> 44.278 ms scattered and 58.338 -> 30.404 ms at 768 dimensions. These harness revisions are identified separately, not pooled as one timing sample.

All 24 runs retained 2016 measured queries with both exact recall references at 1.0, no fallback, unchanged physical probes and 96 empty replacement gaps. Every run had positive writer/query overlap. Physical graph-first probes remained one/25 of 100 segments at 100K and three of 12 at 768 dimensions. Raw filenames end in _native_candidate_gate_{baseline,candidate} followed by _plain.json, _observed.json, _plain_repeat.json or _plain_synchronized.json.

Adverse samples are retained. High-dimensional application-cold P50 worsened 13.347 -> 13.798 ms in the first plain pair, before improving in repeats. High-dimensional warm P50 worsened 8.085 -> 8.370 ms in the reverse pair. Clustered reverse-pair four-query P95 worsened 43.725 -> 48.270 ms, and synchronized clustered P95 worsened 46.268 -> 57.441 ms despite almost unchanged P50. Fixed-corpus RSS varied in both directions; no uniform tail latency, cold-read or memory-saving claim follows.

The scattered concurrent workload is substantially faster, but general concurrency acceptance is still open. Candidate single/four-worker QPS was 83.685/100.556 scattered in the first plain pair and 94.270/99.859 in the reverse repeat, but 99.700/97.306 in the synchronized pair. Synchronized clustered QPS was 165.869/124.108 and high-dimensional 130.848/121.334. Positive aggregate scaling is not yet stable across these workloads.

The final coupled suite passed 934 tests, including native rejection before Python, live policy changes, invalid/empty/unknown candidates, independent concurrent masks, saved-index reloads, legacy dispatch and mixed-worker startup edge cases. Both untouched installed bindings passed 162 targeted tests with 18 explicitly native-capability-only skips. Six-module strict typing, focused lint/whitespace and documentation build passed. Existing broader vector-module style debt is not claimed resolved. The prototype HNSW binding paired with the original installed storage core also passed 174 targeted tests, with six metadata-API-only cases skipped.

The binding remains an isolated prototype, not a promoted dependency. Neither installed HNSW nor the installed Rust storage artifact changed. Only the experimental binding advertises the new capability; normal installed bindings continue through the legacy path. Distribution, upstream/vendored patch maintenance, broader load/soak, hardware acceptance and stable positive concurrency scaling still require separate qualification.

This is lifecycle and current-retrieval qualification, not full sharded database parity or a production load/power-loss acceptance. Globally sorted sharded node queries remain explicitly unsupported. Logical edge counts stream canonical edges rather than using an O(1) counter. The optional write coordinator, production soak, hardware-specific performance acceptance and power-loss qualification remain outside this boundary.

Post-gate CPU, NUMA and scheduling attribution

A whole-process CPU probe on the same scattered 100K fixture measured about 1.00 CPU core for sequential graph-first requests and 1.39 cores for four concurrent requests, with aggregate throughput 93.202 -> 100.627 QPS. The host reported ten physical/logical cores; aggregate host busy time was about 27-30% during those phases. These are short phase averages across all process threads, not instantaneous per-core occupancy, affinity availability or a sustained capacity test. The ordinary vector control used about 0.99 -> 1.35 cores while throughput fell 2.531 -> 1.430 QPS. All 84 measured retrieval queries matched both exact references, without fallback; all four replacement gaps remained empty, with positive writer/query overlap.

The host is an Apple M5 with four performance and six efficiency cores, not ten equivalent cores. The actual isolated storage binding reports HardwareTopology.affinity_active == False. Its printed subsystem core lists do not pin workers on macOS; pinning in topology.rs is Linux-only. Its reported single NUMA node is the non-Linux fallback, not a measurement of memory locality. The tiny native existence lookup runs directly on the calling thread, not through the two-thread Rust query pool. Changing that pool's allocation would therefore not address this particular lookup boundary.

A separate repeated control compared threads with spawned processes on this same hardware, with an independent HNSW index per worker in both modes. Spawn/import/index-build/warmup costs preceded a separate timed-start barrier. The query vectors, HNSW width and exact reference were unchanged. Medians of three runs were:

HNSW control	One worker QPS	Four-worker aggregate QPS	Four/one
Native, no Python callback, threads	22,786	53,724	2.36
Native, no Python callback, processes	22,112	63,599	2.88
Python accept-all callback, threads	6,928	1,056	0.15
Python accept-all callback, processes	7,037	23,617	3.36

Four callback processes used about 3.82 CPU cores; four callback threads used about 2.26 cores despite much lower throughput. All 24 measurements and their variation are retained; every worker's warmup/final output matched its exact reference across the 60,000 timed HNSW calls. The accept-all callback is a non-live attribution control, never a replacement for access policy. Separate processes remove the shared GIL but also change address-space isolation; this is not a proposal to replace the embedded architecture with multiple processes, nor a measurement of performance-core residency.

Intrusive sampled stacks provide more direct evidence for the remaining hot existence probe: approximately 78.21% of the four reader threads' samples were inside PyEval_RestoreThread, and another 16.06% inside PyEval_SaveThread. The already-qualified bounded-GIL metadata control had no samples inside either API; normal Python GIL scheduling still applies to that control. The native vector/query pool threads were waiting throughout both profiles. These are sampled wall-stack proportions, not CPU-time percentages or an additive decomposition of end-to-end query latency. The hot-ID profile had no concurrent writer or cold-slab lookup; live delete/reinsert was checked after sampling.

The evidence supports shared-GIL/native-boundary contention rather than a whole-machine CPU ceiling or NUMA pinning for these workloads. It does not exclude performance/efficiency-core scheduling, cache effects or other end-to-end costs. Session-only CPU, scheduling and stack probes retain source, artifact and report hashes in scheduling-attribution-audit.json. Neither installed native binary changed, and no additional Rust build or existence fast path was performed in this investigation.

Isolated bounded existence-read qualification

After separate explicit build approval, a matching baseline/candidate Rust pair tested a bounded fast path in GraphEngineCore.node_exists. IDs of at most 256 UTF-8 bytes use a nonblocking DashMap try_get: present/absent returns immediately, while a locked shard falls through to the original detached lookup. Longer IDs also retain the detached path. This is a scheduling threshold, not a new ID-length limit. No result is cached and no node payload is hydrated. The SDK's subsequent cold-slab check and live access/archival predicates remain unchanged.

Both cores were built with the same compiler, locked dependencies, release features and Python interpreter. Baseline sources matched the retained bounded-metadata candidate; only src/engine.rs differed in the new candidate. The previously qualified native HNSW candidate gate stayed identical in every retrieval run.

Repeated hot-ID controls recovered four-worker aggregate native presence throughput from 2.13M to 20.67M checks/s; single-worker throughput was 15.17M -> 20.56M. SDK hot-presence throughput was 1.72M -> 14.71M with four workers. These are three-run medians, not per-worker rates or four-core scaling: the tiny GIL-held path largely restores aggregate single-worker capacity. Missing-ID checks also recovered; large-ID and cold-tier behavior were correctness-tested, not performance-qualified by this hot-ID control.

End-to-end changes were smaller and not uniform. Plain aggregate four-query QPS in forward/reverse-order pairs was:

Fixture	Forward baseline -> candidate	Reverse baseline -> candidate
Clustered 100K x 64	130.917 -> 142.130	106.556 -> 133.812
Scattered 100K x 64	96.502 -> 106.673	104.099 -> 90.567
Scattered 10K x 768	117.271 -> 124.171	114.081 -> 121.399

The scattered reverse repeat regressed, as did high-dimensional forward four-query P50 (30.001 -> 30.261 ms) and mixed QPS (91.746 -> 91.215). All samples are retained. General positive concurrency scaling is still unproven; this is not an across-the-board query-speed or tail-latency win.

An additional clustered diagnostic pair measured mean SDK existence time per concurrent query falling 19.158 -> 0.091 ms for the same 251 probes. Its HNSW envelope increased 2.827 -> 6.653 ms, reinforcing that removing one handoff does not eliminate other scheduling costs. Component timers are intrusive and do not isolate pure GIL wait. Separate post-change stack profiles found no worker samples inside PyEval_SaveThread or PyEval_RestoreThread for the small hot existence path; normal Python GIL scheduling remains.

Across 12 plain and two diagnostic runs, all 1,176 measured queries matched both exact references without fallback, physical probes/search width stayed unchanged, and all 56 replacement gaps stayed empty with positive writer/query overlap. Result filenames contain _native_existence_gil_; source, artifact, raw-report and stack fingerprints are retained in the session qualification audit.

Ten Rust bounded-read tests passed, including deterministic writer-lock/GIL inversion checks, deletion before a waiting read resumes, long/empty/UTF-8 IDs, repeated delete/reinsert and payload independence. Combined candidate Python coverage passed 348 tests, with 15 optional live-DocIntel cases skipped and one explicitly deselected pre-existing cancellation-test stub failure. That failure was reproduced under the matching baseline before any storage access: the stub supplies write but the router eagerly also accesses delete/erase. It was not changed as part of this native scheduling work. The untouched installed bindings passed 61 targeted compatibility tests, with 12 explicit HNSW-prototype-only skips.

The native existence candidate remains isolated and unpromoted. Neither installed native binary changed. Production load/soak, stable concurrency acceptance, cold-tier contention and broader typing debt remain separate.

Remaining post-existence retrieval work

Two longer scattered-fixture comparisons used 48 requests per phase and opposite baseline/candidate execution order. Concurrent graph-first aggregate QPS was 104.376 -> 110.779 and 104.101 -> 111.968. The candidate used about 1.38 process CPU cores versus about 1.42 for the baseline; sequential requests used about one core. These short local observations do not establish statistical significance, sustained capacity or a universal improvement. Host background activity differed, and the earlier adverse scattered repeat remains part of the evidence.

Separate count-only instrumentation measured 1,225-1,235 live metadata reads per graph-first request for the 250-candidate fixture: 250 inside HNSW's live-filter callbacks and 975-985 outside them. Ten full-node SDK reads fetched final candidates. Per-request counts matched between sequential and four-request phases. The count probe adds overhead and is not used for a clean throughput comparison. These reads span traversal, eligibility and result revalidation; they are not all redundant, and caching policy results would change live-visibility semantics.

Across the four longer resource runs and one count run, 1,428 measured queries matched both exact references, without fallback or changed physical segment scope; twenty replacement gaps stayed empty with positive writer overlap. The backend was explicitly hnswlib, with the same isolated native candidate gate in baseline and candidate. "HNSW live-filter callback" means that in-process query boundary, not a separate ANN service or a claim about all retrieval backends.

Deterministic profiler output was rejected: simultaneous profilers initially conflicted, and later profiles produced inconsistent call accounting around native callbacks even with only one profiler. No component cost claim is derived from those profiles. Resource phases preceded the extra profiling, and count-only measurements were verified independently. Rejected artifacts and the accepted CPU/count evidence are retained in the session post-existence-attribution-audit.json.

The remaining target is bounded live-metadata/policy processing and Python/native scheduling, not an assumption of whole-machine saturation. No additional retrieval behavior or native build changed in this follow-up.

Decoder ownership-copy simplification

The next simplification removes a redundant outer-dictionary copy from both node and edge decoding. _decode_record_properties already creates private outer/properties dictionaries before restoring reserved timestamps. The decoders now update that owned outer dictionary instead of copying it again. Caller input remains unchanged, including when validation fails. Folded-vector precedence, metadata-only vector omission, temporal fields and full Pydantic validation are preserved. No visibility check, live read or native boundary was removed, and no new production helper/cache was added.

An exact-decoder microbenchmark used eight alternating-order pairs per case against the preserved source. Median time fell about 5.6% for small metadata, 3.3% for wider metadata, 1.4% for full vectors and 5.1% for temporal edges. These small local decoder savings are not an end-to-end query-speed, concurrency or statistical-significance claim.

Candidate-native regression groups passed 174 and 151 tests, with two skips in the latter group. Untouched installed binaries passed 146 decoder/timestamp regressions with six explicit native-metadata-capability skips. Success/failure input immutability and repeated folded/top-level embedding decoding are covered. Storage-module typing still reports the same thirty diagnostics as the exact pre-change source; this simplification introduces none but does not resolve that existing debt.

The archival predicate also avoids temporary label/intersection copies by using isdisjoint, without adding a cache/helper or changing check order. Strict superseded is True, mixed system-label priority, explicit history reads and fresh FileVersion parent lookups are regression-tested. Exact predicate microbenchmarks showed roughly 16-34% less time across Node/dict cases, or tens of nanoseconds per call; this is not a query-speed claim. Candidate coverage passed 251 tests; installed bindings passed 161 with six native-metadata-only skips. Scoped typing and lint passed.

Combined simplification end-to-end comparison

Both Python simplifications were compared together against the exact three preserved pre-change functions, using the same isolated native existence and HNSW candidates in every run. The shared ownership helper was AST-verified unchanged. Three fixtures ran in baseline-first and candidate-first order, with 48 queries per phase and no callback timers or deterministic profilers.

Observed graph-first throughput changes (baseline-first / candidate-first):

Fixture	Warm single request	Four concurrent requests	Replacement-mixed
Clustered, 100K x 64	+5.7% / +8.5%	+8.4% / +5.5%	+4.3% / +2.6%
Scattered, 100K x 64	+5.9% / +3.7%	+2.7% / +3.8%	-7.3% / +2.5%
Scattered, 10K x 768	+4.7% / +6.4%	+5.2% / +6.6%	+4.8% / +7.6%

Calling-thread and process-wide CPU per warm/concurrent query decreased in all six pairs. Concurrent P99 decreased in all six, but P95 was not uniformly better. These are finite local comparisons, not significance or production-scaling claims. Four concurrent requests still used approximately 1.1-1.4 process CPU cores; reducing per-query work did not unlock four-core scaling.

Application-cold-LRU behavior did not improve consistently: scattered 100K throughput fell about 4.0% and 2.1%, with worse P95/P99 in both orders. The high-dimensional cold phase was mixed. One scattered replacement-mixed run also lost 7.3%, with higher latency tails and host activity. Those adverse results are retained; this comparison does not qualify a cold-cache or sustained-load improvement. Ordinary vector-search concurrency was mixed too.

All twelve reports passed both exact-recall references for 4,032 measured queries, with no fallback and unchanged physical probes, shards, search width and live policy. All 48 replacement gaps remained empty, with verified query overlap in every run. The report/source/native fingerprints and installed binary hashes were verified in the session end-to-end-audit.json.

One reverse-order attempt failed in host-only CPU measurement when the OS counter supplied no tick delta. Its log is retained. The completed reverse round uses the same retrieval driver with only the measurement summary corrected: zero-delta host activity is explicitly warned about and recorded as unavailable, not invented as zero utilization. Driver AST comparison verified that nothing outside that summary changed. Shared native binaries remain untouched and the native candidates remain unpromoted.

Shared-fixture cold-LRU attribution

The scattered cold-path losses were followed up on one immutable 100K x 64 fixture, so baseline and candidate used the same constructed segments and query vectors rather than separately built stores. Six pairs alternated execution order, explicitly clearing each segment LRU before each phase. Both variants used the original harness setup and query function, including live policy and both exact-recall references. The comparison called queries sequentially rather than through the original executor; it does not requalify concurrent performance.

Cold-LRU paired throughput changes were +6.60%, -0.37%, +0.84%, +0.39%, -0.99% and +0.53% (median +0.46%). This control did not reproduce a stable 2-4% loss attributable to the simplifications. It neither erases the original adverse runs nor proves their cause. Warm control pairs also varied, from -4.31% to +15.37% (median +2.11%); every pair, including outliers and initial page faults, is retained. No significance claim is made.

Only after all plain pairs, four diagnostic phases timed the existing segment loader. Each phase performed 1,152 sealed-index loads for 48 queries: 24 reloads per query, alongside 25 physical segment probes. Loader wall time accounted for 70.7-71.4% of those observed phases, and loader thread CPU was close to its elapsed time. This measures the complete loader, including file reads, mapping decoding and native index reconstruction; it is not a pure disk-wait or pure HNSW-compute measurement.

The single-entry application LRU intentionally churns the scattered candidate set. Raising its capacity would change the memory constraint, not demonstrate an algorithmic improvement. No cache size, search width, filter, production code or native binary was changed by this control. The 1,152 plain queries and 192 observed queries all passed both exact references with no fallback and unchanged shard/probe counts. Source/native hashes, baseline restoration and untouched installed binaries were verified; the session evidence is cold-paired.json and cold-paired-audit.json. Concurrent scaling, cold-cache acceptance and sustained load remain open.

Segment-loader component attribution

A follow-up separated the Python and native portions of the existing VectorIndex.load, without changing its implementation or rebuilding a native artifact. A bounded synthetic sealed segment contained 1,000 64-dimensional vectors, with M=16, construction width 100 and capacity 1,100. Six alternating plain/observed pairs performed 500 loads per phase. The OS file cache was warm; this is not a cold-disk or full-query comparison.

Plain phase median load time ranged from 1.520 to 1.577 ms. Five nonoverlapping call sites were timed in an AST copy of the exact loader:

Loader component	Median observed share
Native load_index call	88.25%
Python mapping JSON decode	4.96%
Integer-key ID-map restoration	3.00%
Python mapping-file read	0.23%
Native index-object construction	0.17%

The remaining time includes Python setup, file checks/open/close, logging, timer overhead and scheduling. The native load itself averaged about 1.40 ms per call across the observed phases. That call includes its own file reads, integrity validation, allocation and index reconstruction: the 88% is not a measurement of pure native compute or evidence that all disk I/O is negligible. Native subcomponents were not individually timed. Observed/plain phase-median changes ranged from -0.46% to +2.01%; instrumentation effects and drift are retained, not subtracted.

All 6,000 loads passed count/capacity checks. Full mappings, representative stored vectors and search results matched at both boundaries of every phase. The retained binding/algorithm source hashes match the isolated candidate's build audit. Installed binaries remain unchanged. The session artifacts are loader-breakdown.json and loader-breakdown-audit.json.

The retained HNSW binding does not release the Python GIL around load_index; its native method performs the load directly. The SDK also holds its hot-cache lock during resolution. These are potential cold-query concurrency constraints, not an explanation for warmed queries that do not load segments. No binding, synchronization, cache policy, file format or integrity validation was changed. A further native experiment would require separate approval and qualification.

Isolated native loader GIL-release qualification

After explicit approval, matching fresh HNSW baseline/candidate artifacts were built outside the shared installation. The candidate adds only py::call_guard<py::gil_scoped_release>() to the native load_index binding. A source audit verifies that no other binding/header changes; file format, integrity checks, SDK cache locks, search width and live policy are unchanged. This is not a Rust rebuild or a promoted package.

Each artifact passed forty concurrent independent load/data/search/mutation checks. A controlled Python observer progressed during native loading in 0/10 baseline trials and 10/10 candidate trials. That control temporarily increases the Python switch interval; throughput runs never do. Truncated and trailing-data integrity errors, missing-file errors and subsequent valid loads were preserved. The candidate passed 393 coupled tests, including concurrent cold removal/search and configuration/load coverage. Both builds retain the same upstream compiler warnings.

Twelve plain graph-first comparisons retained the original harness setup, query and replacement logic, adding a four-reader cold-LRU phase and omitting ordinary-vector phases. Each phase used 48 queries. Warm phases preloaded the segments; both cold phases retained the same one-entry LRU. No callback timers, segment timers or deterministic profilers were active.

Fixture	Cold four-reader QPS, baseline-first	Cold four-reader QPS, candidate-first
Clustered, 100K x 64	115.08 -> 134.72 (+17.1%)	114.96 -> 130.22 (+13.3%)
Scattered, 100K x 64	17.93 -> 22.30 (+24.4%)	18.27 -> 21.97 (+20.3%)
Scattered, 10K x 768	83.59 -> 110.49 (+32.2%)	82.90 -> 115.95 (+39.9%)

Cold-concurrent P95/P99 decreased in all six pairs. However, scattered process CPU per query rose from 59.63 to 94.70 ms and from 57.62 to 105.31 ms: approximately 59% and 83% more CPU per query, despite only 20-24% more throughput. Process CPU use rose from roughly 1.05-1.07 to 2.11-2.31 cores. Unlocking parallel work is not automatically an efficiency improvement. Other fixtures also retained their CPU costs rather than assuming a throughput gain meant less work.

Separate observed scattered comparisons checked the trade-off. Baseline queries averaged 23.29/22.94 sealed-index reloads versus 21.50/22.50 for the candidate: the CPU increase was not explained by more reloads. Calling-thread CPU in the native-load binding envelope rose from about 33 ms to 70/80 ms per query, accounting for most of the observed increase. That envelope includes binding/GIL reacquisition as well as native loading; the cause within it (for example, allocation, library contention or reacquisition) has not been isolated. Nested loader/native timers are not additive, and diagnostic timings do not replace the plain throughput evidence.

There is no demonstrated general warm-path or cold-single-request gain. Warm results varied, including a -14.4% clustered single-request sample with higher host activity and adverse concurrent samples; these are retained without claiming that host activity proves their cause. The finite comparisons do not establish significance, sustained load or stable four-core scaling.

All 2,880 plain queries passed both exact references, with no fallback, unchanged probes/shards and 48 empty replacement gaps with positive overlap. An additional 960 diagnostic queries passed, with sixteen further empty gaps. Source/artifact audits, installed binary hashes and evidence checks passed. Session evidence is retained under native-hnsw-loader-gil/, including qualification-audit.json. The candidate remains isolated and unpromoted: the scattered CPU-efficiency regression prevents treating this as an unconditional default improvement.

Trinity governance boundary

Trinity's dedicated native thread monitors slab allocation pressure, not Python GIL contention, query latency or available query cores. Heat is based on the sampled increase in allocated pages, not CPU temperature or measured lock waiting; reusing reclaimed slots need not increase that counter. Its independence from compaction protects monitoring responsiveness, but does not change scheduling at the Python/native lookup boundary.

The SDK enables Trinity when opening the cold store. However, the current production Python source has no consumer of trinity_signals() or poll_trinity_signals(). Native code exposes those channel-draining methods, but does not connect the signals to an automatic split/throttle executor. Signal detection must not be described as a fully wired feedback loop.

The current Mitosis.split() samples and classifies pages, writes WAL markers and returns daughter counts/statistics; it does not physically redistribute pages or publish a replacement cold index. A small isolated native probe confirmed that a manual split increased the split counter while leaving slab files, allocation counts and all allocated page bytes unchanged. All twenty test records remained readable after reopening. This was not a saturation, concurrent-writer or crash test.

Ordinary capacity growth is separate: PageSlab opens another slab when the current slab fills. That does not establish semantic page redistribution. Before enabling automatic Mitosis, physical relocation, writer exclusion, index/free-slot publication and crash recovery need a coherent implementation and qualification. The current split counter alone is not evidence of it.

Further signal-lifecycle gaps also remain: sends are nonblocking with their errors discarded, events are emitted on state transitions, and ThrottleClear is emitted for Heat-to-Normal but not Heat-to-Saturation. These are separate governance-hardening items, not fixes for the measured GIL handoff issue. No Trinity behavior or native binary was changed by this read-only investigation.

Pole 2 — Declared constraints, permissive unknown labels

from purple8_graph.validation import SchemaValidator, ValidatingGraphEngine, GraphSchema, NodeSchema, PropertySchema, PropertyType

schema = GraphSchema(name="my_schema")
schema.add_node_schema(NodeSchema(
    label="Person",
    properties=[
        PropertySchema(name="name", type=PropertyType.STRING, required=True),
        PropertySchema(name="age",  type=PropertyType.INTEGER),
    ],
))

validator = SchemaValidator(schema, strict_mode=False, allow_extra_properties=True)
validated_engine = ValidatingGraphEngine(engine, schema)

# Unknown labels are permitted; violations of declared required/type rules raise.
validated_engine.add_node("n4", labels=["UnknownLabel"], properties={"x": 1})

Pole 3 — Strict enforcement

validator_strict = SchemaValidator(schema, strict_mode=True, allow_extra_properties=False)
validated_engine = ValidatingGraphEngine(
    engine, schema, strict_mode=True, allow_extra_properties=False,
)

# This raises ValidationError — "UnknownLabel" is not in the schema
validated_engine.add_node("n5", labels=["UnknownLabel"], properties={"x": 1})

# This raises ValidationError — "age" is the wrong type
validated_engine.add_node("n6", labels=["Person"], properties={"name": "Bob", "age": "thirty"})

Discovering the schema from existing data

The most important feature of Purple8's schema model: schema can be an output of discovery, not an input.

from purple8_graph.validation import create_schema_from_graph

# Build a knowledge graph from 10,000 documents — no schema upfront
engine = GraphEngine("./data")

# ... add thousands of nodes and edges from LLM extraction ...

# NOW infer the schema from what was actually written
schema = create_schema_from_graph(engine)

print(schema.node_schemas)
# → [Person, Organization, Location, Event, Document, ...]

print(schema.edge_schemas)
# → [WORKS_FOR, LOCATED_IN, ATTENDED, AUTHORED_BY, ...]

# Now enforce it going forward
engine.set_schema(schema)

create_schema_from_graph() scans all nodes and edges, infers property types from observed values, and returns a GraphSchema you can inspect, modify, and enforce.

ValidatingGraphEngine uses the underlying engine's current canonical schema. Its optional strict/extra-property settings are additional caller-side checks, not a separate persisted schema or a way to bypass the tenant's engine rules.

LLM-inferred schema

For teams who want to define a schema from sample documents before any data exists:

from purple8_graph.genai import SchemaDetector, LiteLLMProvider

provider = LiteLLMProvider(model="openai/gpt-5.4-mini", api_key="...")
detector = SchemaDetector(provider)

# Feed sample documents — LLM infers entities and relationships
result = await detector.detect(sample_documents=sample_documents[:50])

# Returns a SchemaResult with node_types / edge_types inferred by the LLM
print(result.node_types)   # → [NodeTypeSchema(name='Person', ...), NodeTypeSchema(name='Document', ...)]

The schema is an output of AI inference — not a prerequisite.

Property types

PropertyType	Python equivalent	Example
STRING	str	"Alice"
INTEGER	int	42
FLOAT	float	3.14
BOOLEAN	bool	True
DATETIME	datetime	datetime(2026, 3, 25)
LIST	list	[1, 2, 3]
DICT	dict	{"key": "value"}

Comparison with schema-first systems

System	Schema model	First write requires schema?	Retroactive schema inference?
Purple8	Data-first, optional strict mode	❌ No	✅ create_schema_from_graph()
Neo4j	Schema-optional (explicit constraints)	❌ No	❌ No
Kùzu	Schemaless	❌ No	❌ No
Spanner Graph	Schema-first (DDL required)	✅ Yes	❌ No
TigerGraph	Schema-first (DDL required)	✅ Yes	❌ No

Why this matters for AI workloads

When you build a knowledge graph from LLM-extracted entities and relationships, the schema is emergent — it's a property of your data that you discover, not something you can define in advance. Spanner Graph and TigerGraph require DDL before the first byte of data. Purple8 lets you write first, understand your data, then optionally enforce a schema.

---

Referential Integrity

When an EdgeSchema registers source_labels and/or target_labels, Purple8 enforces these constraints at write time via engine.add_edge().

from purple8_graph.core.models import EdgeSchema, NodeSchema, GraphSchema, PropertySchema, PropertyType

schema = GraphSchema(name="hr_graph")

schema.add_node_schema(NodeSchema(label="Employee"))
schema.add_node_schema(NodeSchema(label="Department"))
schema.add_edge_schema(EdgeSchema(
    edge_type="WORKS_IN",
    source_labels=["Employee"],   # source must have this label
    target_labels=["Department"], # target must have this label
))

engine = GraphEngine("./hr", schema=schema)

engine.add_node("e1", labels=["Employee"],   properties={"name": "Alice"})
engine.add_node("d1", labels=["Department"], properties={"name": "Engineering"})

# ✅ Passes — labels match the schema constraints
engine.add_edge("w1", "WORKS_IN", "e1", "d1")

# ❌ Raises GraphEngineError — "d1" is a Department, not an Employee
engine.add_edge("w2", "WORKS_IN", "d1", "e1")
# → GraphEngineError: Schema validation failed for edge 'w2':
#     Edge 'w2' source node has invalid labels.
#     Expected one of ['Employee'], got ['Department']

Referential integrity is a no-op when no EdgeSchema is registered — existing schemaless graphs are unaffected.

---

Schema Migrations

Evolve your GraphSchema without full re-ingest using SchemaMigration and MigrationRunner.

Supported operations

Operation	Description
ADD_PROPERTY	Add a property (with optional default value) to all matching nodes/edges
REMOVE_PROPERTY	Remove a property key from all matching entities
RENAME_PROPERTY	Rename a property key while preserving its value
CHANGE_PROPERTY_TYPE	Coerce a property value via a converter callable
ADD_LABEL	Attach a new label to all nodes carrying the target label
REMOVE_LABEL	Strip a label from all matching nodes

Quick example

from purple8_graph.core.migration import (
    MigrationOperationType,
    MigrationOperation,
    SchemaMigration,
    MigrationRunner,
)

# Define a migration from schema v1.0.0 → v1.1.0
migration = SchemaMigration(
    migration_id="add_email_to_person",
    from_version="1.0.0",
    to_version="1.1.0",
    description="Add optional email property to all Person nodes",
    operations=[
        MigrationOperation(
            op_type=MigrationOperationType.ADD_PROPERTY,
            target_label="Person",
            property_name="email",
            default_value=None,          # existing nodes get None
        ),
        MigrationOperation(
            op_type=MigrationOperationType.RENAME_PROPERTY,
            target_label="Person",
            property_name="full_name",
            new_property_name="name",    # rename full_name → name
        ),
    ],
)

# Apply directly via the engine (version-gated)
result = engine.apply_migration(migration)
print(result)
# → MigrationResult('add_email_to_person': 42 affected, 0 skipped, 0 errors)

print(engine.schema.version)  # → "1.1.0"

Dry run (preview without writing)

result = engine.apply_migration(migration, dry_run=True)
print(f"Would affect {result.total_affected} entities")

Migration registry + multi-step paths

from purple8_graph.core.migration import MigrationRegistry

registry = MigrationRegistry()
registry.register(migration_1_0_to_1_1)
registry.register(migration_1_1_to_1_2)
registry.register(migration_1_2_to_1_3)

engine.register_migration(migration_1_0_to_1_1)
engine.register_migration(migration_1_1_to_1_2)
engine.register_migration(migration_1_2_to_1_3)

# Apply all migrations from current version to target in one call
results = engine.apply_migration_path("1.3.0")
for r in results:
    print(r)

---

Query Cost-Based Planner

Purple8's QueryPlanner analyses your query before execution and selects the cheapest physical strategy automatically across six strategies (ranked cheapest → most expensive):

How it works

Strategy	Complexity / Cost	When selected
PROPERTY_INDEX_SCAN	O(k) · ×1.0	Equality filter on an indexed property + label filter
LABEL_INDEX_SCAN	O(label_count) · ×2.0	Label filter present (no indexed property match)
VECTOR_SEARCH	O(log N · ef_search)	HNSW approximate nearest-neighbour search
TRAVERSAL	O(V + E)	BFS/DFS traversal from a start node
FULL_SCAN	O(N) · ×10.0	Fallback — no usable index
HYBRID	Combined	Vector search + label/property filter

The plan is chosen at get_nodes() call time, not at query compile time. Strategy selection is logged at DEBUG level.

Introspect without executing

from purple8_graph.core.query import Query

q = Query().labels("Person").where("email", "=", "alice@example.com")
plan = engine.get_query_plan(q)

print(plan)
# → QueryPlan(PROPERTY_INDEX_SCAN, cost=4.2, steps=[PROPERTY_INDEX_SCAN → LIMIT])

print(plan.to_dict())
# → {
#     "chosen_strategy": "PROPERTY_INDEX_SCAN",
#     "total_cost": 4.2,
#     "steps": [{"op": "PROPERTY_INDEX_SCAN", "estimated_rows": 4, ...}],
#     "warnings": []
#   }

Ensure an index is present for your hot query paths

# Build a property index so the planner can use PROPERTY_INDEX_SCAN
engine.build_property_index("Person", "email")

# Now repeated queries on email will use the index path
nodes = engine.get_nodes(query=Query().labels("Person").where("email", "=", "alice@example.com"))

TIP

Use engine.get_query_plan(query) in development to verify your query will use an index before deploying to production.