System Architecture & Internals
Purple8 Hyper Graph is built from the ground up to unify vector search, graph traversal, and AI workflows into a single cohesive engine. This guide explains the core architectural decisions that allow it to scale and remain highly available in production.
New to Purple8? Start with the Unified Data Store guide to understand how Purple8 replaces four separate systems (graph DB + vector DB + document store + search engine) with one embedded engine.
0. Unified Storage Layer
All data — graph topology (nodes, edges, adjacency indexes), document properties (schemaless JSON), vector embeddings (HNSW), and full-text indexes (BM25 inverted index) — lives in a single BrickCoreStorage (Cortex) instance. A single write path ensures atomicity across all four storage modes, and the KMS envelope codec encrypts everything uniformly at rest. See Unified Data Store → for details and code examples.
BrickCoreStorage uses a lock-free DashMap write buffer backed by a Write-Ahead Log (WAL) for durability. Every write is fsync'd to WAL before the mutation is applied — crash recovery replays pending WAL entries on restart. There is no RocksDB, no LSM compaction, and no Raft consensus layer.
1. Zero-Downtime HNSW Lifecycle
Vector databases typically struggle with high write throughput because deleting or updating vectors requires either expensive re-indexing or complex locking. Purple8 solves this using a Tombstone & Auto-Compaction architecture.
Soft Deletes (Tombstones)
When you delete or update a node that contains an embedding, the underlying HNSW (Hierarchical Navigable Small World) index does not physically remove the vector immediately. Instead, we use hnswlib.mark_deleted() to flag the vector as a tombstone.
- This operation is O(1) and completely non-blocking.
- Read queries automatically skip tombstoned vectors.
Lazy Auto-Compaction
As tombstones accumulate, traversal efficiency drops slightly. To prevent degradation, Purple8 monitors the ratio of deleted vectors.
- When the tombstone count reaches
P8G_VECTOR_COMPACT_THRESHOLD(default is 500) or exceeds 10% of the total index size, an asynchronous auto-compaction task is triggered. - Auto-compaction creates a fresh, optimized index in the background and hot-swaps it with a zero-downtime pointer update.
- Why this matters: You never have to schedule blocking cron jobs for full index rebuilds during off-peak hours.
2. Durability & Crash Recovery via WAL
Durability is provided by the Write-Ahead Log (WAL), not by an external consensus protocol. There is no Raft, no ZooKeeper, no distributed leader election.
Write path
Every mutation (add_node, add_edge, update_node) follows this sequence:
- Append the mutation to the WAL with a single
os.write()+fdatasync()— durable before the in-memory state is updated. - Apply the mutation to the DashMap write buffer (lock-free).
- Return to the caller.
Crash recovery
On startup, Purple8 replays any WAL entries marked pending (i.e. written but not yet marked committed). This guarantees no mutation is lost across a crash or unclean shutdown.
Why no Raft?
Purple8 is an embedded, single-process engine. High availability is achieved through Docker restart policies, Fly.io machine health checks, or any process supervisor — not through a distributed consensus protocol.
Read replicas (implemented)
Purple8 ships a full read-replica subsystem (replica.py) wired into secure_server.py. Set P8G_REPLICA_MODE=replica and P8G_PRIMARY_URL=http://primary-host:8100 to start a read-only mirror that periodically pulls a snapshot from the primary via ReplicationPoller. The primary uses ReplicaRouter to round-robin reads across registered replicas. Incremental CDC-based replication (instead of full-snapshot) is the next planned iteration.
3. Lightweight CDC Streams
Change Data Capture (CDC) is built directly into the storage engine. Every mutation generates an event (e.g., NODE_CREATED, NODE_DELETED, EDGE_UPDATED).
- Pure Event Streams: The CDC payload contains the entity IDs, labels, and lightweight properties. It does not ship massive embeddings downstream.
- Why this matters: When wiring Purple8 into Apache Kafka, AWS Kinesis, or the internal Journey Engine, the event bus remains nimble. Downstream AI workflows or analytics listeners only pull embeddings on-demand via the API if they explicitly need them, preventing bandwidth bloat and OOM crashes in streaming pipelines.