Observability — Prometheus & OpenTelemetry

Purple8 Hyper Graph provides production-grade observability out of the box. This guide covers the built-in Prometheus metrics, OpenTelemetry distributed tracing, and the pre-configured Grafana stack.

---

Quick Start

# Install the monitoring extras
pip install "purple8-hyper-graph[prod]"

# Start the full stack (API + Prometheus + Grafana)
docker compose up -d

Service	URL	Default Credentials
Purple8 API	http://localhost:8000	Set via ADMIN_EMAIL / ADMIN_PASSWORD
Prometheus	http://localhost:9090	—
Grafana	http://localhost:3001	admin / admin

---

Prometheus Metrics

How it works

Purple8 uses the standard prometheus-client library (≥0.19.0) as the backing implementation for all metrics. Each metric registered through MetricsRegistry is mirrored into an isolated prometheus_client.CollectorRegistry, so GET /metrics produces fully spec-compliant Prometheus text exposition:

• Counter names carry the required _total suffix
• Histogram output includes cumulative _bucket, _sum, and _count series
• HELP and TYPE comment lines are always emitted

A built-in fallback formatter activates automatically if prometheus-client is somehow not importable, ensuring the endpoint never goes dark.

Scrape endpoint

GET /metrics
Content-Type: text/plain; version=0.0.4; charset=utf-8

Point Prometheus at your Purple8 server:

# prometheus.yml
scrape_configs:
  - job_name: 'purple8-hyper-graph'
    static_configs:
      - targets: ['purple8-hyper-graph:8000']
    metrics_path: '/metrics'
    scrape_interval: 10s
    scrape_timeout: 5s

Available Metrics

HTTP layer

Metric	Type	Labels	Description
purple8_graph_http_requests_total	Counter	method, path, status	Total HTTP requests processed
purple8_graph_http_request_duration_seconds	Histogram	method, path	Request latency distribution (default buckets: 5ms → 10s)
purple8_graph_active_connections	Gauge	—	Number of currently active connections

Graph operations

Metric	Type	Labels	Description
purple8_graph_node_operations_total	Counter	operation, tenant	Node CRUD operations (add, update, delete)
purple8_graph_edge_operations_total	Counter	operation, tenant	Edge CRUD operations (add, update, delete)
purple8_graph_query_duration_seconds	Histogram	query_type, tenant	Cypher / hybrid / vector query execution latency

Per-tenant

Metric	Type	Labels	Description
purple8_graph_tenant_node_count	Gauge	tenant_id	Current node count for each tenant
purple8_graph_tenant_edge_count	Gauge	tenant_id	Current edge count for each tenant
purple8_graph_tenant_queries_total	Counter	tenant_id, query_type	Total queries executed per tenant

Useful PromQL examples

# Request rate (5-minute window)
rate(purple8_graph_http_requests_total[5m])

# P99 request latency
histogram_quantile(0.99, rate(purple8_graph_http_request_duration_seconds_bucket[5m]))

# Error rate (5xx responses)
sum(rate(purple8_graph_http_requests_total{status=~"5.."}[5m]))
  / sum(rate(purple8_graph_http_requests_total[5m]))

# Node operations per second by tenant
sum by (tenant) (rate(purple8_graph_node_operations_total[5m]))

# Slow queries (P95 > 100ms)
histogram_quantile(0.95, rate(purple8_graph_query_duration_seconds_bucket[5m])) > 0.1

---

OpenTelemetry Distributed Tracing

How it works

OTEL tracing is wired at two levels:

1. FastAPI auto-instrumentation — every HTTP request/response is automatically traced via opentelemetry-instrumentation-fastapi
2. Engine-level manual spans — core graph operations (add_node, traverse, vector_search, etc.) create child spans with attributes like node IDs, query types, and result counts

When the OTEL packages are not installed or the endpoint env var is unset, the engine uses lightweight no-op stubs — zero runtime overhead.

Activation

Set two environment variables:

# Required — the gRPC endpoint of your OTEL Collector
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

# Optional — defaults to "purple8-hyper-graph"
export OTEL_SERVICE_NAME="purple8-hyper-graph"

That's it. On server startup you'll see:

otel_initialized  endpoint=http://localhost:4317

If the packages aren't installed, you'll see:

otel_unavailable  reason="No module named 'opentelemetry'"

Traced operations

Span Name	Layer	Description
engine.add_node	Core	Node creation with vector indexing
engine.update_node	Core	Node property / embedding update
engine.delete_node	Core	Node deletion (tombstone + HNSW mark)
engine.add_edge	Core	Edge creation
engine.traverse	Core	Graph traversal (variable-length paths)
engine.vector_search	Core	HNSW / DiskANN vector similarity search
engine.hybrid_search	Core	Combined BM25 + vector + graph reranking
HTTP {method} {path}	FastAPI	Auto-instrumented for every REST endpoint

Connecting to common backends

Jaeger

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:1.54

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

Open http://localhost:16686 to view traces.

Grafana Tempo

Add Tempo as a datasource in Grafana and point the OTEL exporter at the Tempo distributor:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://tempo:4317"

AWS X-Ray / Azure Monitor / GCP Cloud Trace

Use the OTEL Collector as a relay and configure the appropriate exporter in the Collector config.

---

Docker Compose Stack

The production docker-compose.yml includes the full observability stack:

┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│  Purple8    │────▶│ Prometheus  │────▶│   Grafana    │
│  Graph API  │     │ :9090       │     │   :3001      │
│  :8000      │     └─────────────┘     └──────────────┘
│  /metrics   │
└─────────────┘

Key files

File	Purpose
docker-compose.yml	Full stack: API + Prometheus + Grafana
docker-compose.dev.yml	Dev stack: API only (no monitoring, fast startup)
deploy/prometheus/prometheus.yml	Prometheus scrape config (pre-configured for Purple8)
deploy/grafana/provisioning/datasources/	Auto-provisions Prometheus as a Grafana datasource
deploy/grafana/provisioning/dashboards/	Pre-built dashboard JSON files

Environment variables for the monitoring stack

Variable	Default	Description
PROMETHEUS_PORT	9090	Host port for Prometheus UI
GRAFANA_PORT	3001	Host port for Grafana UI
GRAFANA_USER	admin	Grafana admin username
GRAFANA_PASSWORD	admin	Grafana admin password

Adding OTEL to Docker

Add the endpoint to your API service environment in docker-compose.yml:

purple8-hyper-graph-api:
  environment:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
    OTEL_SERVICE_NAME: "purple8-hyper-graph"

---

SysAdmin Metrics Snapshot

For quick debugging without a Prometheus stack, Super Admins can call:

GET /sysadmin/metrics-snapshot
Authorization: Bearer <super-admin-token>

Returns all metrics as JSON:

{
  "metrics": [
    {"name": "purple8_graph_http_requests_total", "labels": {"method": "GET", "path": "/health"}, "value": 42},
    ...
  ]
}

TIP

This endpoint requires the prometheus-client package (pip install "purple8-hyper-graph[prod]") and Super Admin authentication.

---

Health Check Endpoints

Purple8 exposes Kubernetes-compatible health probes that work alongside the metrics:

Endpoint	Purpose	Healthy	Unhealthy
GET /health	Liveness probe	200	503
GET /health/ready	Readiness probe (engine loaded)	200	503
GET /health/startup	Startup probe (initial load)	200	503

Configure in your Kubernetes deployment:

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  periodSeconds: 5

startupProbe:
  httpGet:
    path: /health/startup
    port: 8000
  failureThreshold: 30
  periodSeconds: 2