System Architecture & Internals

Purple8 Hyper Graph is built from the ground up to unify vector search, graph traversal, and AI workflows into a single cohesive engine. This guide explains the core architectural decisions that allow it to scale and remain highly available in production.

New to Purple8? Start with the Unified Data Store guide to understand how Purple8 replaces four separate systems (graph DB + vector DB + document store + search engine) with one embedded engine.

---

0. Unified Storage Layer

All data — graph topology (nodes, edges, adjacency indexes), document properties (schemaless JSON), vector embeddings (HNSW), and full-text indexes (BM25 inverted index) — lives in a single BrickCoreStorage (Purple8 Cortex) instance accessed via GraphEngine (the sole engine entry point in v0.72.0). A single write path ensures atomicity across all four storage modes, and the KMS envelope codec encrypts everything uniformly at rest. See Unified Data Store → for details and code examples.

BrickCoreStorage uses a Rust lock-free DashMap write buffer backed by BrickWAL (fdatasync before mutation) for durability. Every write is fsync'd to WAL before the mutation is applied — crash recovery replays pending WAL entries on restart. There is no RocksDB and no LSM compaction. (Raft consensus is available as a separate, opt-in Enterprise / Custom (Self-Hosted) component for multi-node HA/sharding — see §2a below — it is not part of the single-node durability path.)

0a. CortexAutoConfig & Bounded Working Set (auto_config=True)

BrickCoreStorage peak RSS does not grow monotonically with total corpus size $N$. When initialized with GraphEngine("./data", auto_config=True), CortexAutoConfig probes available host RAM and derives (M, ef_construction, segment_size, batch_size) so that peak process RSS remains bounded to alpha * available_ram (default alpha=0.75) via the brick seal-and-evict cycle:

1. The Rust DashMap write buffer fills for one batch (batch_size nodes).
2. BrickWAL.append_batch() seals the batch with a single os.write() + fdatasync().
3. Completed DashMap entries are evicted and pages are returned to the OS, returning RSS to baseline before the next batch begins.

In production benchmarks on an 8.84M MS MARCO corpus (24 GB dataset), peak RSS stayed bounded at ~9 GB across 90 seal cycles with 40,551 docs/s batch ingest, 0.33 ms Query P99 @ 100K, 23 ms HTTP P95 (1024d), 916 QPS (0.946 recall@10) on VectorDBBench 1M×768d, and 0% errors across a 1.18M-request soak test.

CortexAutoConfig also inspects hardware compute topology at startup:

• Multi-socket CPU & NUMA topology: Detects physical sockets, NUMA nodes, and computes recommended_shards = max(cpu_sockets, numa_nodes).
• CPU matrix accelerators: Detects Intel AMX tile hardware and Intel XPU (Xeon integrated GPU, Arc, Data Center GPU Max, Gaudi).
• NVIDIA Tensor Core GPUs: Detects Hopper (H100/H200), Blackwell (B200), and Blackwell Ultra B300 (288 GB HBM3e, FP4) (gpu_fp4_capable=True).

0b. Six-Sigma Operational Controls (v0.72.0)

Purple8 Hyper Graph v0.72.0 enforces four zero-data-loss and bounded-resource operational controls:

1. Disk high-watermark write admission gate (P8G_WRITE_ADMISSION_DISK_MAX_PCT=95.0): The write admission controller checks storage volume utilization before admitting mutations and rejects writes with HTTP 507 before the underlying disk reaches ENOSPC.
2. Non-destructive online storage integrity scrubber (engine.scrub_integrity() / POST /admin/storage/scrub): Verifies WAL frame CRCs and cold-slab records online without blocking reads or mutating storage state.
3. Durable Journey AI outbox (ai_dispatched flag + replay_pending_ai_dispatches() on crash recovery): Stage transitions (ADVANCED_TO) and SLA events (SLA_WARNED, SLA_BREACHED) that require an AI advisor step persist ai_dispatched=False on the audit edge before dispatch, flip it to True upon completion, and automatically replay any interrupted dispatches via replay_pending_ai_dispatches() after a crash (SIGKILL).
4. Streaming gzip/JSONL backups with 4-byte P8KM envelope header check: BackupManager streams compressed JSONL records and inspects only the 4-byte magic header (b"P8KM") before invoking KMS envelope decryption, avoiding full-archive buffering in memory when checking unencrypted or encrypted archives.

---

1. Zero-Downtime HNSW Lifecycle

Vector databases typically struggle with high write throughput because deleting or updating vectors requires either expensive re-indexing or complex locking. Purple8 solves this using a Tombstone & Auto-Compaction architecture.

Empty-tenant allocation and index compatibility

New default segmented vector indexes allocate their first native HNSW segment only on the first vector write, under the mutation lock. Empty searches, rebuilds, saves and reopens of these zero-segment indexes do not reserve native vector capacity. Previously persisted physical segments still load normally, including legacy empty segments. Initial segment capacity, growth headroom and rollover behavior are unchanged once vectors are written.

Segmented-index metadata now uses format version 2, which supports zero physical segments. The reader still accepts legacy formats 0 and 1. Older builds reject version 2; do not downgrade a datastore after saving its vector indexes with this build.

Soft Deletes (Tombstones)

When you delete or update a node that contains an embedding, the underlying HNSW (Hierarchical Navigable Small World) index does not physically remove the vector immediately. Instead, we use hnswlib.mark_deleted() to flag the vector as a tombstone.

• This operation is O(1) and completely non-blocking.
• Read queries automatically skip tombstoned vectors.

Lazy Auto-Compaction

As tombstones accumulate, traversal efficiency drops slightly. To prevent degradation, Purple8 monitors the ratio of deleted vectors.

• When the tombstone count reaches P8G_VECTOR_COMPACT_THRESHOLD (default is 500) or exceeds 10% of the total index size, an asynchronous auto-compaction task is triggered.
• Auto-compaction creates a fresh, optimized index in the background and hot-swaps it with a zero-downtime pointer update.
• Why this matters: You never have to schedule blocking cron jobs for full index rebuilds during off-peak hours.

---

2. Durability & Crash Recovery via WAL

Single-node durability is provided by the Write-Ahead Log (WAL), not by consensus. There is no ZooKeeper, and single-node writes never involve leader election.

Write path

Every mutation (add_node, add_edge, update_node) follows this sequence:

1. Append the mutation to the WAL with a single os.write() + fdatasync() — durable before the in-memory state is updated.
2. Apply the mutation to the DashMap write buffer (lock-free).
3. Return to the caller.

Crash recovery

The Cortex storage path retains brick.wal across both clean closes and crashes. It reconstructs hot state from CRC-checked operation frames, alongside durable cold slabs and their indexes. It does not use the separate Python WALWriter's pending/committed status records as its storage recovery mechanism.

flush() drains accepted writes and flushes cold-tier indexes; it does not create a native DashMap snapshot or delete the WAL. checkpoint() compacts the WAL to current live state. A checkpoint is not a historical archive or a point-in-time recovery boundary.

close() also releases the storage-owned cold slab and label-catalog references, rather than retaining native mappings/background governance workers until a closed engine is garbage-collected. Closed-store node reads, counts, pages and flushes raise an explicit error; reopen the engine to access data. The WAL and slab files remain on disk.

Recovery contract and operator checks

Cortex uses WAL and cold-slab persistence, not native snapshot/restore. An existing snapshot/ directory, damaged WAL frame, unsupported operation or failed record application blocks opening the store. It never silently serves a WAL prefix as a successfully restored database. Preserve the original directory; use a compatible build for legacy-data migration or restore a validated logical backup into a separate directory. Even an empty legacy snapshot directory is rejected for explicit operator inspection, not automatically removed.

The contract is visible through:

• Embedded SDK: engine.recovery_status(); sharded engines check every physical shard. TenantManager.recovery_status() checks opened stores and known recovery gates without opening more tenant engines.
• REST: GET /admin/storage/recovery, requiring ADMIN_SETTINGS. Unready storage returns HTTP 503, not success. No record contents or filesystem paths are returned.
• HTTP SDK: synchronous and asynchronous client.recovery_status().
• MCP: admin.recovery_status, requiring an admin role.
• Operator UI: the System administration → Health panel includes a storage recovery check and describes the persistence mode. The commercial LCNC → Storage page also shows readiness and unsupported capabilities, and returns 503 when readiness is unconfirmed.
• Probes: /health/ready returns 503 for opened stores requiring recovery or known tenant recovery gates; /health/live remains a process-liveness check. Public /health also reports aggregate storage readiness. These checks do not open or validate every inactive/on-disk tenant or verify backup archives.

The response reports persistence_mode="wal_and_slabs", native_snapshots=false, and historical_pitr=false. For online verification of WAL and slab checksums without restarting the engine, call engine.scrub_integrity() or POST /admin/storage/scrub.

HTTP authentication and operator diagnostics

Application API operations must declare an authentication boundary. The server validates its FastAPI routes during construction and again after optional routers are mounted at startup. An unclassified operation fails startup; new operations on the same application also fail closed at request time. Public and separately authenticated protocol exceptions use exact route templates and HTTP methods, never namespace-wide prefix exemptions.

The LCNC router requires a validated signed session and the live user's administration permission, including for newly added console handlers. The exact login, setup, recovery and stylesheet routes remain public. The anonymous-route regression exercises private routes on the running application, rather than accepting missing-route responses as authentication. This is an authentication-coverage gate, not proof of every operation's role, record-access policy or tenant isolation.

Public health/readiness responses omit tenant identifiers, detailed errors and check metadata. Public health does not enumerate or open tenant engines. GET /admin/health provides deployment-wide diagnostics, including tenant embedding queues and detailed readiness, and requires super-admin access. Ordinary tenant-admin credentials are insufficient. Production /welcome provides sign-in instructions without tenant counts, disk usage or generated demonstration keys; the unauthenticated demonstration ingest remains restricted to explicitly enabled development mode.

Ingestion formats, job reads/history and extraction previews require NODE_READ. Preview-only access does not grant graph writes: committing requires NODE_CREATE, job edits require NODE_UPDATE, and job deletion requires NODE_DELETE. Cold-storage archive creation requires NODE_CREATE, version metadata requires NODE_READ, raw archive download (POST /storage/restore) requires DATA_EXPORT, and deletion requires NODE_DELETE. Raw download does not restore records into the hot graph. Archive operations retain tenant scope.

Inbound connector HTTP webhooks require configured HMAC signing. An empty signing secret returns 503 before processing; a missing or invalid signature returns 401. Configure the sender and receiver before enabling delivery. Unsigned inbound HTTP is not a development fallback.

Quantum state and job access

All Quantum HTTP operations use explicit operation permissions. State/job reads, measurement, comparison and nearest-state lookup require NODE_READ. State encoding and solver operations that retain job results require NODE_CREATE; state edits/replacement require NODE_UPDATE, and deletions require NODE_DELETE. Reusing an existing solver job ID also requires NODE_UPDATE. These checks apply to restricted API keys, not just role names. REST viewers can still use read-only simulation and solver profiling. MCP keeps its declared per-tool roles and reaches the same protected HTTP handlers for persistence and tenant routing. The LCNC Quantum console dispatches through that router in-process using the live signed-in user's permissions and tenant, without a deployment API key or a localhost network request. Failed operations retain their HTTP error status. Console fields use the native request model names. Simulation is tensor-network simulation, and comparison evaluates stored-state fidelity; neither is a generic circuit editor or solver-comparison interface. Record denials use the main application's exception handlers, and escaped error fragments remain visible in the result pane. Inactive users, revoked permissions and stale tenant bindings are rejected rather than falling back to an installation identity.

Persisted states and jobs use the authenticated tenant's engine, including shared-key namespaces; no handler falls back to the deployment's default tenant or reads raw shared storage. Job persistence completes before returning success, and storage failures propagate instead of returning an empty list or a record that was not stored. State/job endpoints also verify the stored record's label, so supplying another graph record's ID does not grant access through the Quantum API.

New states use dimension-specific named embedding slots (quantum_state_<complex_dimension>), separate from document embeddings. Replacement updates the existing node rather than deleting/recreating it, preserving graph references, and must retain the state dimension; encode a new state for a different dimension. Empty, nonfinite and zero-norm vectors are rejected. Existing records are not automatically re-indexed by this change. Nearest-state lookup remains approximate ANN candidate selection; reported candidate fidelity is computed from the stored amplitudes, not inferred from the index's distance metric. It is not an exhaustive global fidelity ranking.

AEC jobs and permissions

AEC analysis and job reads require NODE_READ. Filesystem BIM ingestion requires NODE_CREATE; persisted topology and MORL jobs require both NODE_CREATE and NODE_UPDATE before computation starts. Job writes enable record enforcement. This does not establish universal record-write enforcement for AEC buildings.

Topology acceptance means the pending record has been stored in the authenticated tenant's engine. Completion requires persistence of the full result, including topology convergence/export inputs or MORL Pareto/history details. Storage errors are not converted into successful responses or empty histories. Completed results survive engine reopen; topology export reconstructs its mesh from stored inputs. MORL saliency currently reports a reward-history proxy, not GNN gradient attribution.

GET /aec/jobs accepts limit (1–1000, default 100) and nonnegative offset, returning jobs and next_offset. A scoped job label selects records before native tenant paging, avoiding the shared engine's unbounded filtered-query fallback. The LCNC AEC monitor reads bounded pages with the live session's tenant, permissions and record visibility, without deployment credentials or process-global result dictionaries. Sorting and active counts are page-local.

A different worker cannot determine whether pending/running work has stopped, so it reports unconfirmed with the recorded status. Verify the originating worker before resubmitting. Durable dispatch/restart resumption, admission control and workload-sized geometry/result capacity qualification remain open.

Pre-boundary records lacking _AECJobScoped are excluded from AEC result/list surfaces, even if a shared-storage wrapper supplied a tenant property. They are not deleted or automatically reassigned: review tenant provenance, ownership and result completeness before any explicit migration. These exclusions do not quarantine legacy data from raw graph APIs. Model-quality validation on real structural data remains a separate gate.

AEC stored-building boundaries

Stored-building operations use the authenticated tenant engine and check these compound permissions before resolving storage or parsing an IFC upload:

Operation	Permissions
List building summaries	NODE_READ
Read a building and its elements	NODE_READ, EDGE_READ
Create a demo or import IFC	NODE_CREATE, EDGE_CREATE
Run an analysis	NODE_READ, EDGE_READ, QUERY_TRAVERSE
Delete a building	NODE_READ, EDGE_READ, NODE_DELETE, EDGE_DELETE

Creation and deletion enable record enforcement. A contributor cannot delete another owner's building, and a denied member cannot cause earlier members to be deleted before authorization fails. Loaded model relationships are checked for delete access too. These checks do not qualify arbitrary external-link cascades or concurrent policy changes across a multi-record operation.

A building's metadata must be a Building record with the matching building_id. Element identities must match bld:<building_id>:<guid>. The visible model must match its declared element/relationship counts; inconsistent member identities or types are also rejected. If members are missing or blocked, detail, analysis and deletion return 409, rather than analyzing a silently truncated structural model. An absent or blocked metadata record returns 404. Summary counts are the metadata's declared counts, not a substitute for model completeness validation.

Import checks member-ID collisions and uses create-only native batches. It publishes the Building metadata only after all member writes acknowledge completion; short acknowledgements or storage errors cannot report a completed import. This is not a transactional import across nodes, edges and shards: failures can leave unpublished member records. Inspect these records before retrying; atomic rollback, orphan cleanup and concurrent-write qualification remain open. Deletion likewise rejects an incomplete acknowledgement rather than reporting a completed removal.

Regression coverage uses real shared, dedicated and sharded engines. IFC route tests substitute the optional parser while exercising native persistence, permissions, size/format validation and temporary-file cleanup. They do not qualify real IFC extraction fidelity or structural-model quality.

Backup and restore

Use the logical archive interfaces: IncrementalBackupManager.create_backup() and restore_backup(), or the permission-protected REST endpoints POST /backup/create and POST /backup/restore. These are separate from storage flush/checkpoint and do not invoke unsupported native snapshot methods.

GET /export and SDK export_json() expose readable graph records, not a portable original-file archive. POST /import and import_json() accept application graph JSON but reject managed File/version/catalog records and relationships before any import writes or requested clear. REST reports 422; the SDK raises ImportExportError. Use portable backup/restore for a graph containing managed files; generic CRUD cannot recreate their lifecycle state.

Quiesce affected application writes and document-processing jobs for backup and restore maintenance; do not treat a live export as an atomic online snapshot. Restore to an isolated destination first, verify graph/schema/document lineage and vector retrieval, close/reopen it, then measure recovery time at the intended workload size before cutover. Merge is the safe restore default; explicit replacement is destructive. Retain access to the required encryption keys. Crash/reopen and small archive round-trip tests do not establish a customer RTO, RPO, cross-zone failover target or large-corpus recovery qualification.

Repeatable local recovery drill

Run the native-engine drill from a source checkout using the existing virtual environment. Choose node/edge counts and vector dimensions matching the workload; the combined corpus and internal records must fit the installed licence.

PYTHONPATH=src .venv/bin/python benchmarks/recovery_qualification.py \
  --nodes 24000 --edges 24000 --dimension 32 --payload-bytes 256 \
  --output ./p8g-recovery-evidence.json

The drill uses an isolated temporary source and destination. It acknowledges batched writes, an update and a deletion, durably records the expected state, then sends SIGKILL to its own writer without closing the engine. Fresh workers validate crash recovery, create a checksummed logical archive, restore into a separate database, close it and reopen it in another process. Full-record fingerprints compare IDs, properties, labels, embedding slots, system timestamps and edge validity; the canonical schema, vector retrieval and adjacency are also checked. Fingerprinting streams records rather than keeping another corpus copy. Generated synthetic databases and archives are cleaned up after the run; only the requested JSON evidence file is retained. In v0.72.0, BackupManager also supports streaming gzip/JSONL backups with a 4-byte P8KM envelope header check (P8G_BACKUP_MAX_ARCHIVE_MB ceiling).

Optional --max-backup-seconds, --max-restore-to-verified-reopen-seconds and --max-peak-rss-mb enforce explicit local acceptance limits. Exceeding any supplied limit exits nonzero while retaining the failed JSON evidence. Worker errors and timeouts also produce failure evidence. An existing output file is never overwritten.

The JSON records phase durations, archive bytes, per-worker peak RSS, exact-state fingerprints and runtime/source/native-artifact hashes. Restore-to-verified-reopen includes worker startup, restore, close, fresh-process reopening and full verification; crash-open time excludes worker startup and reports verification separately. A pass without supplied limits means correctness only, not an agreed recovery objective. This synthetic local test does not qualify document originals, multi-tenant archives, encryption-key recovery, Azure Blob transfer, disk-cache-cold startup, replication or cross-zone failover. Use the documented maintenance procedure and representative customer data for those acceptance gates.

Single-node HA

For a single GraphEngine instance, high availability is achieved through Docker restart policies, Fly.io machine health checks, or any process supervisor.

Read replicas

Purple8 ships a read-replica subsystem (replica.py, replication.py) wired into secure_server.py. Set P8G_REPLICA_MODE=replica and P8G_PRIMARY_URL=http://primary-host:8100 to start a read-only mirror that pulls snapshots from the primary via ReplicationPoller, or use WAL streaming (replication.py) for low-RPO replication. The primary uses ReplicaRouter to round-robin reads across registered replicas.

2a. Raft consensus & self-hosted licensing tiers

Purple8 self-hosted deployments use a two-tier licensing model:

• Developer (Self-Hosted Free): Up to 50K nodes and 1 concurrent MCP agent on a single node.
• Enterprise / Custom (Self-Hosted): Unlimited nodes, unlimited MCP agents, full compliance controls, horizontal sharding, and multi-node high availability.

For deployments needing multi-node write availability beyond a single process, the Enterprise / Custom (Self-Hosted) tier includes a Raft consensus layer (raft.py, raft_cluster.py): leader election, heartbeats, append-entries replication, and a RaftLog backed by the same WAL infrastructure. This backs failover.py (automatic failover, sub-30s re-election) and coordination.py (distributed locking across shard groups) when sharding.py is used to horizontally partition a graph across multiple GraphEngine nodes. This is opt-in infrastructure for Enterprise-scale deployments — the single-node WAL path above requires none of it.

---

3. Lightweight CDC Streams

Change Data Capture (CDC) is built directly into the storage engine. Every mutation generates an event (e.g., NODE_CREATED, NODE_DELETED, EDGE_UPDATED).

• Pure Event Streams: The CDC payload contains the entity IDs, labels, and lightweight properties. It does not ship massive embeddings downstream.
• Why this matters: When wiring Purple8 into Apache Kafka, AWS Kinesis, or the internal Journey Engine, the event bus remains nimble. Downstream AI workflows or analytics listeners only pull embeddings on-demand via the API if they explicitly need them, preventing bandwidth bloat and OOM crashes in streaming pipelines.