Observability — Prometheus & OpenTelemetry

Purple8 Hyper Graph provides production-grade observability out of the box. This guide covers the built-in Prometheus metrics, OpenTelemetry distributed tracing, and the pre-configured Grafana stack.

Quick Start

bash

# Install the monitoring extras
pip install "purple8-hyper-graph[prod]"

# Start the full stack (API + Prometheus + Grafana)
docker compose up -d

Service	URL	Default Credentials
Purple8 API	`http://localhost:8000`	Set via `ADMIN_EMAIL` / `ADMIN_PASSWORD`
Prometheus	`http://localhost:9090`	—
Grafana	`http://localhost:3001`	`admin` / `admin`

Prometheus Metrics

How it works

Purple8 uses the standard prometheus-client library (≥0.19.0) as the backing implementation for all metrics. Each metric registered through MetricsRegistry is mirrored into an isolated prometheus_client.CollectorRegistry, so GET /metrics produces fully spec-compliant Prometheus text exposition:

Counter names carry the required _total suffix
Histogram output includes cumulative _bucket, _sum, and _count series
HELP and TYPE comment lines are always emitted

A built-in fallback formatter activates automatically if prometheus-client is somehow not importable, ensuring the endpoint never goes dark.

Scrape endpoint

GET /metrics
Content-Type: text/plain; version=0.0.4; charset=utf-8

Point Prometheus at your Purple8 server:

yaml

# prometheus.yml
scrape_configs:
  - job_name: 'purple8-hyper-graph'
    static_configs:
      - targets: ['purple8-hyper-graph:8000']
    metrics_path: '/metrics'
    scrape_interval: 10s
    scrape_timeout: 5s

Available Metrics

HTTP layer

Metric	Type	Labels	Description
`purple8_graph_http_requests_total`	Counter	`method`, `path`, `status`	Total HTTP requests processed
`purple8_graph_http_request_duration_seconds`	Histogram	`method`, `path`	Request latency distribution (default buckets: 5ms → 10s)
`purple8_graph_active_connections`	Gauge	—	Number of currently active connections

Graph operations

Metric	Type	Labels	Description
`purple8_graph_node_operations_total`	Counter	`operation`, `tenant`	Node CRUD operations (`add`, `update`, `delete`)
`purple8_graph_edge_operations_total`	Counter	`operation`, `tenant`	Edge CRUD operations (`add`, `update`, `delete`)
`purple8_graph_query_duration_seconds`	Histogram	`query_type`, `tenant`	Cypher / hybrid / vector query execution latency

Per-tenant

Metric	Type	Labels	Description
`purple8_graph_tenant_node_count`	Gauge	`tenant_id`	Current node count for each tenant
`purple8_graph_tenant_edge_count`	Gauge	`tenant_id`	Current edge count for each tenant
`purple8_graph_tenant_queries_total`	Counter	`tenant_id`, `query_type`	Total queries executed per tenant

Useful PromQL examples

promql

# Request rate (5-minute window)
rate(purple8_graph_http_requests_total[5m])

# P99 request latency
histogram_quantile(0.99, rate(purple8_graph_http_request_duration_seconds_bucket[5m]))

# Error rate (5xx responses)
sum(rate(purple8_graph_http_requests_total{status=~"5.."}[5m]))
  / sum(rate(purple8_graph_http_requests_total[5m]))

# Node operations per second by tenant
sum by (tenant) (rate(purple8_graph_node_operations_total[5m]))

# Slow queries (P95 > 100ms)
histogram_quantile(0.95, rate(purple8_graph_query_duration_seconds_bucket[5m])) > 0.1

OpenTelemetry Distributed Tracing

How it works

OTEL tracing is wired at two levels:

FastAPI auto-instrumentation — every HTTP request/response is automatically traced via opentelemetry-instrumentation-fastapi
Engine-level manual spans — core graph operations (add_node, traverse, vector_search, etc.) create child spans with attributes like node IDs, query types, and result counts

When the OTEL packages are not installed or the endpoint env var is unset, the engine uses lightweight no-op stubs — zero runtime overhead.

Activation

Set two environment variables:

bash

# Required — the gRPC endpoint of your OTEL Collector
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

# Optional — defaults to "purple8-hyper-graph"
export OTEL_SERVICE_NAME="purple8-hyper-graph"

That's it. On server startup you'll see:

otel_initialized  endpoint=http://localhost:4317

If the packages aren't installed, you'll see:

otel_unavailable  reason="No module named 'opentelemetry'"

Traced operations

Span Name	Layer	Description
`engine.add_node`	Core	Node creation with vector indexing
`engine.update_node`	Core	Node property / embedding update
`engine.delete_node`	Core	Node deletion (tombstone + HNSW mark)
`engine.add_edge`	Core	Edge creation
`engine.traverse`	Core	Graph traversal (variable-length paths)
`engine.vector_search`	Core	HNSW vector similarity search on `BrickCoreStorage` (Cortex)
`engine.hybrid_search`	Core	Combined BM25 + vector + graph reranking
`HTTP {method} {path}`	FastAPI	Auto-instrumented for every REST endpoint

Connecting to common backends

Jaeger

bash

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:1.54

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

Open http://localhost:16686 to view traces.

Grafana Tempo

Add Tempo as a datasource in Grafana and point the OTEL exporter at the Tempo distributor:

bash

export OTEL_EXPORTER_OTLP_ENDPOINT="http://tempo:4317"

AWS X-Ray / Azure Monitor / GCP Cloud Trace

Use the OTEL Collector as a relay and configure the appropriate exporter in the Collector config.

Docker Compose Stack

The production docker-compose.yml includes the full observability stack:

┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│  Purple8    │────▶│ Prometheus  │────▶│   Grafana    │
│  Graph API  │     │ :9090       │     │   :3001      │
│  :8000      │     └─────────────┘     └──────────────┘
│  /metrics   │
└─────────────┘

Key files

File	Purpose
`docker-compose.yml`	Full stack: API + Prometheus + Grafana
`docker-compose.dev.yml`	Dev stack: API only (no monitoring, fast startup)
`deploy/prometheus/prometheus.yml`	Prometheus scrape config (pre-configured for Purple8)
`deploy/grafana/provisioning/datasources/`	Auto-provisions Prometheus as a Grafana datasource
`deploy/grafana/provisioning/dashboards/`	Pre-built dashboard JSON files

Environment variables for the monitoring stack

Variable	Default	Description
`PROMETHEUS_PORT`	`9090`	Host port for Prometheus UI
`GRAFANA_PORT`	`3001`	Host port for Grafana UI
`GRAFANA_USER`	`admin`	Grafana admin username
`GRAFANA_PASSWORD`	`admin`	Grafana admin password

Adding OTEL to Docker

Add the endpoint to your API service environment in docker-compose.yml:

yaml

purple8-hyper-graph-api:
  environment:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
    OTEL_SERVICE_NAME: "purple8-hyper-graph"

SysAdmin Metrics Snapshot

For quick debugging without a Prometheus stack, Super Admins can call:

GET /sysadmin/metrics-snapshot
Authorization: Bearer <super-admin-token>

Returns all metrics as JSON:

json

{
  "metrics": [
    {"name": "purple8_graph_http_requests_total", "labels": {"method": "GET", "path": "/health"}, "value": 42},
    ...
  ]
}

TIP

This endpoint requires the prometheus-client package (pip install "purple8-hyper-graph[prod]") and Super Admin authentication.

Health Check Endpoints & Six-Sigma Operational Controls

Purple8 exposes health endpoints designed for Kubernetes probes and zero-spike multi-tenant operations:

Endpoint	Purpose	Healthy	Unhealthy
`GET /health`	Liveness / readiness / startup	`200` — `{"status": "healthy", "read_only": bool, "embedding_queue": {...}}`	`503`
`GET /admin/health`	Multi-tenant fleet health	Probes only active opened tenant engines in memory for zero-spike observability (never wakes cold tenant directories)	`503`
`GET /sysadmin/health-deep`	Deep diagnostic suite	Runs all registered `HealthChecker` checks (storage, memory, LLM connectivity)	`503`
`POST /admin/storage/scrub`	Online bit-rot scrubber	Executes `engine.scrub_integrity()` online across `BrickCoreStorage` (Cortex) and `BrickWAL` records	`500` / integrity report

Six-Sigma Storage & Admission Safeguards

Zero-spike /admin/health: In multi-tenant deployments with hundreds of dedicated tenants, /admin/health inspects only active, currently opened tenant engines, eliminating latency or RSS spikes from polling cold directories.
Online storage bit-rot scrubber (POST /admin/storage/scrub): Validates BrickCoreStorage and BrickWAL checksums plus P8KM AES-256-GCM tags online via engine.scrub_integrity() without downtime.
Disk high-watermark admission controller (P8G_WRITE_ADMISSION_DISK_MAX_PCT=95.0): Automatically throttles and rejects incoming writes when volume utilisation reaches 95.0%, protecting BrickWAL seal cycles from disk-full corruption.
Streaming gzip/JSONL backups with P8KM magic check: Backups stream as compressed JSONL with 4-byte P8KM header verification and P8G_BACKUP_MAX_ARCHIVE_MB archive size protection.

Configure Kubernetes probes in your deployment:

yaml

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 5

startupProbe:
  httpGet:
    path: /health
    port: 8000
  failureThreshold: 30
  periodSeconds: 2

Observability — Prometheus & OpenTelemetry ​

Quick Start ​

Prometheus Metrics ​

How it works ​

Scrape endpoint ​

Available Metrics ​

HTTP layer ​

Graph operations ​

Per-tenant ​

Useful PromQL examples ​

OpenTelemetry Distributed Tracing ​

How it works ​

Activation ​

Traced operations ​

Connecting to common backends ​

Jaeger ​

Grafana Tempo ​

AWS X-Ray / Azure Monitor / GCP Cloud Trace ​

Docker Compose Stack ​

Key files ​

Environment variables for the monitoring stack ​

Adding OTEL to Docker ​

SysAdmin Metrics Snapshot ​

Health Check Endpoints & Six-Sigma Operational Controls ​

Six-Sigma Storage & Admission Safeguards ​

Observability — Prometheus & OpenTelemetry

Quick Start

Prometheus Metrics

How it works

Scrape endpoint

Available Metrics

HTTP layer

Graph operations

Per-tenant

Useful PromQL examples

OpenTelemetry Distributed Tracing

How it works

Activation

Traced operations

Connecting to common backends

Jaeger

Grafana Tempo

AWS X-Ray / Azure Monitor / GCP Cloud Trace

Docker Compose Stack

Key files

Environment variables for the monitoring stack

Adding OTEL to Docker

SysAdmin Metrics Snapshot

Health Check Endpoints & Six-Sigma Operational Controls

Six-Sigma Storage & Admission Safeguards