DocIntel ↔ Graph Integration Patterns

Purple8 Hyper Graph and Purple8 Document Intelligence (DocIntel) are two separate products, each with its own license. They are designed to work together through well-defined integration patterns that keep both systems independently deployable and operationally isolated.

DocIntel is optional. Bring an existing parser, extractor, ETL pipeline, or structured dataset directly to Graph. See Bring Your Own Ingestion for the commit contract, a tested payload, evidence mapping, and verification steps.

The Graph preview adapter preserves document Markdown, tables, page ranges, entity mention offsets, and extraction warnings. The ingestion wizard forwards this content with the reviewed entity/relationship selections. At commit time, it stores uploaded files (or pasted text) through Graph's file API and reuses the returned file reference. URL extraction does not imply that the original remote file has been stored. The commit result reports original-file, structured-content, chunk, and embedding outcomes separately.

Multipart preview and asynchronous jobs

Send file bytes as the multipart file field:

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@document.html;type=text/html" \
  "$GRAPH_URL/ingest/preview/file?async_mode=true"

The response contains a job_id; poll GET /ingest/jobs/{job_id} using the same Graph credentials. A completed job has status: "done" and a normalized preview. Failed jobs remain failed and do not receive a successful preview. Without async_mode=true, Graph waits for completion and returns the preview directly; DocIntel failures return HTTP 502. Missing DocIntel configuration returns HTTP 428 rather than silently selecting another extractor.

Both modes are preview-only (auto_commit=false): review the result, retain the original through the file API, then explicitly commit approved content. Extraction warnings and the extraction-model identifier remain visible. An empty extracted entity list is not evidence that source text or layout was lost, and neither preview nor ordinary commit automatically asserts native RDF.

Repeatable local HTTP acceptance proof

From a source checkout with the existing Graph and DocIntel virtual environments:

PYTHONPATH=src:. \
P8G_DOCINTEL_PYTHON=/path/to/Purple8-DocIntel/.venv/bin/python \
P8G_DOCINTEL_SOURCE=/path/to/Purple8-DocIntel/src \
.venv/bin/python -m pytest tests/e2e/test_docintel_semantic_http.py -q --no-cov

The POSIX proof starts the actual DocIntel HTTP job service on a private loopback port, checks authentication, uploads the public-domain excerpt through Graph's multipart route and exercises synchronous completion and asynchronous polling. DocIntel's parser, Markdown/page rendering, job processing and HTTP serialization are real. Its inference boundary is replaced by an explicitly labelled controlled transcript with no entities, not an extraction model. The checked-in review mapping separately selects the source dateline.

Authenticated Graph handlers then verify canonical-schema-backed ingestion, byte-identical original download, exact mention offsets and fingerprints, explicit RDF assertion, mixed RDF/LPG output, selected N-Quads and source-policy filtering. A controlled extraction failure must surface as an error without graph writes. Graph handlers use the test client; only the DocIntel leg crosses a real TCP connection in this proof. The separate Graph TCP restart proof qualifies Graph's network/restart behavior.

The service uses Uvicorn's portable h11/asyncio transport, isolated temporary storage and ephemeral test credentials. It does not read a deployment .env, inherit provider credentials or permit outbound service/model access. It is stopped after the test. No external LLM, OCR, model-download, automatic document-to-RDF mapping or model-quality claim follows from this proof.

Diversified transcript acceptance proof

tests/test_diverse_document_semantics.py accepts an operator-retained local qualification report rather than silently substituting a model response:

P8G_DIVERSE_DOCINTEL_REPORT=/path/to/qualification/report.json \
PYTHONPATH=src:. .venv/bin/python -m pytest \
  tests/test_diverse_document_semantics.py -q --no-cov

Each documents entry supplies source, source_sha256, transcript, mapping_literal, status and errors. The transcript is the actual DocIntel result JSON. Source paths and transcripts must remain available locally. Failed document rows fail the test; omitting the environment variable produces an explicit skip, not acceptance.

The proof retains original bytes, checks the exact reviewed Markdown quote, commits the transcript through authenticated Graph handlers and separately asserts the reviewed RDF statement. It checks source-closed LPG evidence, Unicode-codepoint mention spans, Markdown hashes, selected N-Quads and unauthenticated rejection. Extracted entities and relationships are retained; the reviewed observation is distinct from their predictions.

The local diversified parser corpus covers office, tabular, email, digital and scanned PDF inputs plus existing report pages. PDF text inside nested Form XObjects must survive into Markdown, not just top-level text boxes. This is not a guarantee of perfect visual reading order for arbitrary PDF layouts. Reports must distinguish parsing only from full model extraction. Passing a parser/transcript-to-Graph proof does not qualify NER/relationship accuracy, a production model/provider, HTTP load or deployment packaging.

Two-service TCP and abrupt-restart proof

The combined proof also runs Graph as a real HTTP service:

PYTHONPATH=src:. \
P8G_DOCINTEL_PYTHON=/path/to/Purple8-DocIntel/.venv/bin/python \
P8G_DOCINTEL_SOURCE=/path/to/Purple8-DocIntel/src \
.venv/bin/python -m pytest tests/e2e/test_document_dual_restart.py -q --no-cov

It runs shared and dedicated tenant storage, with two tenants defining the same reviewed-evidence label using different required marker enums. Each tenant saves its DocIntel connection through REST, uploads the excerpt through asynchronous multipart preview, polls the actual job, explicitly reviews and commits the evidence, retains the original, and separately asserts the RDF quad.

After successful responses, both services receive SIGKILL, without application shutdown. Fresh processes reopen their original stores. DocIntel keeps its loopback address and ephemeral service key; Graph reuses its saved tenant connection settings and the original bearer tokens. Recovery does not depend on re-uploading, re-extracting, re-committing or reconfiguring the document.

The checks compare completed DocIntel results/previews, tenant schemas, matched node/edge DTOs and timestamps, mixed evidence rows and selected N-Quads with their pre-crash values. They also verify original bytes, fingerprints and quote offsets, idempotent assertion retries, schema enum/immutable rejection after reopening, and actual BM25 results rather than scan fallback. A tenant's persisted File-row policy continues to hide its viewer's source download, mixed row and selected export; the other tenant's viewer remains unaffected.

This uses the same controlled empty inference transcript and separate reviewed mapping as the acceptance proof. The test shares an operator DocIntel service key between the tenant connections; it does not certify DocIntel job-level tenant isolation. Graph data/schema/policy isolation is tested. Licensing uses Graph development mode and DocIntel test enforcement settings, not a production license activation. No external model or service access is allowed.

The recovery boundary is acknowledged completed work followed by process crashes. It does not cover arbitrary mid-job/mid-commit interruption, machine power loss, cross-service atomicity, live inference quality, OCR, browser behavior, packaged deployments or production throughput.

For an interrupted ingestion or lost Graph response, keep the reviewed payload, obtain the user's retry confirmation and let Graph de-duplicate the identical commit. See Interrupted ingestion: review, confirm, retry. That retry also requires the same source: retain the extraction's original-byte checksum and any version pin. A source-version conflict requires review and confirmation of current-source extraction, not replay or relabelling of an old payload. This check belongs to Graph; it adds no DocIntel job state or recovery workflow. The separate Graph TCP proof tests interruption on either side of durable finalization; it does not add DocIntel job recovery states or automatically repeat extraction.

TL;DR — DocIntel parses documents and extracts entities/relationships. Graph stores them, runs RAG queries, and serves the API. They communicate over HTTP via a thin async client. Graph degrades gracefully when DocIntel is offline.

DocIntel's job_to_preview() now surfaces doc_type, chunking_hint, section_boundaries, and suggested_embedding_model. Pass these into /ingest/commit and ChunkingAgent selects the optimal (chunking_strategy, embedding_model) pair automatically — zero re-classification overhead.

API-first — headless by design

Purple8 Hyper Graph is a headless engine. Install it with pip install purple8-hyper-graph, call its REST APIs, and build any interface you want on top. No frontend ships with the core product — the optional web UI (port 3000) is a separate convenience layer that calls the exact same endpoints you would.

---

Developer Quick Reference

End-to-end: install → ingest → query — in 6 commands.

# 1. Install
pip install purple8-hyper-graph

# 2. Start the Graph server
purple8-hyper-graph serve --dev              # starts on :8000, dev-mode license bypass

# 3. (Optional) Start DocIntel v0.12.5 for 70+ format & 75+ language support
export PURPLE8_DOCINTEL_URL=http://localhost:8200
pip install purple8-docintel[all]
python -m purple8_docintel.server      # starts on :8200

# 4. Authenticate
TOKEN=$(curl -s -X POST http://localhost:8000/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@localhost","password":"admin"}' | jq -r .access_token)

# 5. Ingest a document (DocIntel parses → Graph stores)
curl -X POST http://localhost:8000/ingest/preview \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"text": "Alice Chen is a senior engineer at Acme Corp. She leads Project Alpha."}' \
  | jq .

# Review the preview, then commit:
curl -X POST http://localhost:8000/ingest/commit \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entities": [
      {"id":"alice","type":"Person","name":"Alice Chen","properties":{"role":"Senior Engineer"},"include":true},
      {"id":"acme","type":"Organization","name":"Acme Corp","properties":{},"include":true},
      {"id":"alpha","type":"Project","name":"Project Alpha","properties":{},"include":true}
    ],
    "relationships": [
      {"source":"alice","target":"acme","type":"WORKS_AT","properties":{},"include":true},
      {"source":"alice","target":"alpha","type":"LEADS","properties":{},"include":true}
    ],
    "doc_type": "document",
    "chunking_hint": "semantic",
    "suggested_embedding_model": "voyage-4-large"
  }'

# 6. Query via RAG
curl -X POST http://localhost:8000/rag/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"question": "Who leads Project Alpha?"}'

Or in Python:

import httpx, asyncio
from purple8_graph.docintel_client import get_docintel_client, is_docintel_configured

async def main():
    base = "http://localhost:8000"
    async with httpx.AsyncClient(base_url=base) as api:
        # Login
        r = await api.post("/auth/login", json={"email": "admin@localhost", "password": "admin"})
        token = r.json()["access_token"]
        headers = {"Authorization": f"Bearer {token}"}

        # Ingest text (Graph delegates to DocIntel if configured)
        preview = await api.post("/ingest/preview", headers=headers, json={
            "text": "Alice Chen leads Project Alpha at Acme Corp."
        })
        print(preview.json())

        # Commit approved entities
        await api.post("/ingest/commit", headers=headers, json=preview.json())

        # RAG query
        answer = await api.post("/rag/query", headers=headers, json={
            "question": "Who leads Project Alpha?"
        })
        print(answer.json()["answer"])

asyncio.run(main())

Headless by Design — Build Any Interface

Purple8 Hyper Graph is a headless API. There is no required UI — pip install purple8-hyper-graph gives you the full engine, every endpoint, and the complete RAG pipeline. You build whatever surface you need on top:

• Custom web app — React, Vue, Svelte, vanilla JS — anything that speaks HTTP
• MCP tool — Claude Desktop, Cursor, VS Code Copilot, or any MCP-compatible agent queries your graph natively via rag.query / rag.hybrid_query (part of Purple8's 125 MCP tools across 16 namespaces)
• Chatbot — wire POST /rag/query into Streamlit, Gradio, a Slack bot, a Teams integration, or an embedded chat widget
• Backend service — another microservice calls the REST API programmatically (Python, Node, Go, Java — anything with an HTTP client)
• CLI / scripts — curl, httpx, SDK — automate ingest, query, and evaluation in CI/CD or batch pipelines
• Any client — the Python SDK and TypeScript SDK wrap the same API surface for backend services and web/React apps

The optional web UI (purple8-hyper-graph-ui) is a low-code convenience layer for teams who prefer point-and-click. It calls the exact same REST endpoints. Developers can ignore it entirely.

---

Consuming the Generation Endpoint

The key output surface is POST /rag/query — it takes a natural-language question, retrieves context from the graph via hybrid search, and returns a grounded answer. Here's how to wire it into different interfaces:

MCP (AI Agents)

Any MCP-compatible agent (Claude Desktop, Cursor, VS Code Copilot) can call your graph without custom code:

// MCP tool definition (auto-registered when you run the MCP server)
{
  "name": "rag.query",
  "description": "Query the Purple8 knowledge graph with natural language",
  "input_schema": {
    "type": "object",
    "properties": {
      "question": { "type": "string" }
    },
    "required": ["question"]
  }
}

# Start the MCP server
pip install "purple8-hyper-graph[mcp]"
purple8-hyper-graph mcp serve

The MCP tool inherits your tenant's RAG Studio config — all retrieval tuning applies automatically.

Chatbot / UI Widget

Wire the endpoint into any conversational interface:

# Streamlit example
import streamlit as st
import httpx

st.title("Knowledge Assistant")
question = st.chat_input("Ask a question…")

if question:
    r = httpx.post("http://localhost:8000/rag/query",
        json={"question": question},
        headers={"Authorization": f"Bearer {TOKEN}"})
    answer = r.json()
    st.chat_message("assistant").write(answer["answer"])

    with st.expander("Sources"):
        for node in answer["context_nodes"]:
            st.write(f"• {node}")

// React chat component — fetch from any frontend
const ask = async (question: string) => {
  const res = await fetch('/api/rag/query', {
    method: 'POST',
    headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ question }),
  });
  return res.json(); // { answer, context_nodes, latency_ms }
};

Backend Service

Call from another microservice — no SDK required, just HTTP:

# Python backend calling Purple8 Hyper Graph
async def answer_customer_query(question: str) -> str:
    async with httpx.AsyncClient() as client:
        r = await client.post(
            "http://purple8-hyper-graph-api:8000/rag/query",
            json={"question": question},
            headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
        )
        return r.json()["answer"]

CLI / Automation

# One-liner for scripts and CI
curl -s -X POST http://localhost:8000/rag/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"question": "What are the open compliance findings?"}' \
  | jq -r .answer

---

Product Boundaries

Capability	Graph (PURPLE8_LICENSE_KEY)	DocIntel v0.12.5 (LICENSE__KEY)
Node / edge storage (BrickCoreStorage / Cortex)	✅	—
HNSW vector index	✅	—
RAG query & retrieval tuning	✅	—
RAG evaluation (DeepEval)	✅	—
Cypher-compatible queries	✅	—
Multi-tenant RBAC, JWT auth	✅	—
Basic text parsing (fallback)	✅	—
70+ document formats across 75+ languages	—	✅
Scanned OCR (5 OCR engines incl. self-hosted P8OCR & Tesseract)	—	✅
CAD/BIM (IFC, IFCZIP, DXF, DWG, STEP, STL, OBJ, GLTF, G-code) & SAP IDocs	—	✅
Sketch/whiteboard parsing	—	✅
Two-pass hybrid extraction (Pass 1 GLiNER NER → Pass 2 domain-tuned LLM across AEC, Contract, Financial, Scientific, General)	—	✅
100 MiB ODF zip-bomb protection & XML entity expansion guards	—	✅
Chunking strategies (semantic, parent-child, sentence-window)	—	✅
Table / image parsing	—	✅

---

Architecture

Pattern 1 — Sidecar Microservice (recommended)

Graph and DocIntel run as co-located microservices on the same host or Kubernetes pod. The Graph API proxies all ingest requests to DocIntel over localhost.

          ┌──────────────────────────────────────────────┐
          │  Your code / curl / SDK  (primary)           │
          │  — or optional Web UI    (port 3000)         │
          └─────────────────┬────────────────────────────┘
                            │  REST (JSON over HTTP)
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Purple8 Hyper Graph API  (port 8000)                 │
│                                                                 │
│   POST /ingest/preview       ─────┐                             │
│   POST /ingest/preview/file  ─────┤                             │
│   GET  /ingest/jobs/{id}     ─────┤  docintel_client.py         │
│   GET  /ingest/formats       ─────┘  (async HTTP client)        │
│                                       │                         │
│   POST /ingest/commit  ──── writes directly to graph engine     │
│   POST /rag/query      ──── reads from vector index + LLM      │
│   PUT  /rag/config      ──── per-tenant config (stored in graph)│
└───────────────────────────────┼─────────────────────────────────┘
                                │  HTTP (PURPLE8_DOCINTEL_URL)
                                │  Bearer token (PURPLE8_DOCINTEL_KEY)
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│              Purple8 DocIntel  (port 8200)                      │
│                                                                 │
│   POST /process        — file upload → async job                │
│   POST /process/url    — URL download → async job               │
│   GET  /jobs/{id}      — poll status + result                   │
│   GET  /jobs           — list all jobs                          │
│   DELETE /jobs/{id}    — cancel job                             │
│   GET  /formats        — supported format list                  │
│   GET  /health         — liveness probe                         │
│   GET  /ready          — readiness probe                        │
│                                                                 │
│   Format detection → Parse → Chunk → LLM extract               │
│   → Emit { entities[], relationships[] }                        │
└─────────────────────────────────────────────────────────────────┘

Pattern 2 — Remote Service

DocIntel runs on a dedicated host (or GPU instance for OCR-heavy workloads). Multiple Graph instances can share a single DocIntel deployment.

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Graph (US)  │     │  Graph (EU)  │     │  Graph (AP)  │
└──────┬───────┘     └──────┬───────┘     └──────┬───────┘
       │                    │                    │
       └─────────┬──────────┴──────────┬─────────┘
                 │   HTTPS / mTLS      │
                 ▼                     ▼
         ┌──────────────────────────────────┐
         │  DocIntel  (GPU instance)        │
         │  https://docintel.internal:8200  │
         └──────────────────────────────────┘

Pattern 3 — Embedded (No DocIntel)

Graph runs standalone with its built-in genai-based text extractor. Only plain text and URLs are supported — all other formats return an error.

POST /ingest/preview { text: "..." }
  ↓ DocIntel unavailable
  ↓ Fallback: KnowledgeExtractor.extract(text[:8000])
  ↓ Returns entities + relationships  (_fallback: true)

---

Data Flow

Synchronous Ingestion (small files)

1. Your code → POST /ingest/preview/file (file < 500 KB)
2. Graph API → DocIntelClient.process_file_sync(filename, bytes)
   2a. POST /process → { job_id, status: "queued" }
   2b. Poll GET /jobs/{id} every 1.5s until status = "done"
3. DocIntel returns { result: { entities[], relationships[], chunks_processed,
                               doc_type, chunking_hint, section_boundaries,
                               suggested_embedding_model, doc_type_confidence } }
4. Graph normalises via DocIntelClient.job_to_preview() → preview payload
   (includes doc_type, chunking_hint, section_boundaries, suggested_embedding_model,
   doc_type_confidence as top-level preview fields)
5. Your code reviews preview → POST /ingest/commit (selected entities/rels
   + doc_type / chunking_hint / suggested_embedding_model hint fields)
6. If chunking_agent_enabled=true, Graph runs ChunkingAgent.classify() with
   a DocIntelChunkingHint — zero re-classification, confidence 1.0
7. Graph writes nodes + edges to tenant graph, with chunking classification
   metadata stored as node properties

Asynchronous Ingestion (large files)

1. Your code → POST /ingest/preview/file?async_mode=true (IFC, large PDF)
2. Graph API → DocIntelClient.process_file(filename, bytes)
   2a. POST /process → { job_id, status: "queued" } — returns immediately
3. Graph returns { job_id, status: "queued" } to your code
4. Your code polls GET /ingest/jobs/{job_id}
   4a. Graph proxies to DocIntel GET /jobs/{job_id}
   4b. Returns { status: "processing", progress_pct: 45 }
5. When status = "done", Graph normalises result into preview shape
6. Your code reviews → POST /ingest/commit

Graceful Degradation

The Graph API never hard-fails if DocIntel is offline:

Endpoint	Fallback Behaviour
POST /ingest/preview (text)	Uses built-in KnowledgeExtractor via tenant LLM config. Response includes _fallback: true.
POST /ingest/preview (URL)	Returns HTTP 502 — DocIntel required for URL download.
POST /ingest/preview/file	Returns HTTP 502 — DocIntel required for file parsing.
GET /ingest/formats	Returns static fallback list with warning field.
GET /ingest/jobs/{id}	Returns HTTP 502.

---

Configuration

Environment Variables

Set these on the Graph service:

Variable	Default	Description
PURPLE8_DOCINTEL_URL	http://localhost:8200	Base URL of the DocIntel microservice
PURPLE8_DOCINTEL_KEY	"" (empty)	Bearer token for DocIntel API auth

Set these on the DocIntel service:

Variable	Description
OPENAI_API_KEY	LLM API key for entity extraction
LICENSE__KEY	DocIntel license key
HOST / PORT	Bind address (default 0.0.0.0:8200)
JOB_STORE	memory (default), redis, or sqlite
REDIS_URL	Required when JOB_STORE=redis

Docker Compose

services:
  purple8-hyper-graph-api:
    environment:
      - PURPLE8_DOCINTEL_URL=http://purple8-docintel:8200
      - PURPLE8_DOCINTEL_KEY=${DOCINTEL_API_KEY:-}
    depends_on:
      purple8-docintel:
        condition: service_healthy

  purple8-docintel:
    image: purple8/purple8-docintel:0.12.5
    container_name: purple8-docintel
    ports:
      - "8200:8200"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LICENSE__KEY=${DOCINTEL_LICENSE_KEY}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8200/health"]
      interval: 15s
      timeout: 5s
      retries: 3
    networks:
      - purple8-hyper-graph-net

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: purple8-docintel
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: docintel
          image: purple8/purple8-docintel:0.12.5
          ports:
            - containerPort: 8200
          env:
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: purple8-secrets
                  key: openai-api-key
            - name: LICENSE__KEY
              valueFrom:
                secretKeyRef:
                  name: purple8-secrets
                  key: docintel-license-key
          readinessProbe:
            httpGet:
              path: /ready
              port: 8200
            initialDelaySeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: purple8-docintel
spec:
  selector:
    app: purple8-docintel
  ports:
    - port: 8200
      targetPort: 8200

Then set PURPLE8_DOCINTEL_URL=http://purple8-docintel:8200 on the Graph deployment.

---

The Client Layer — docintel_client.py

The Graph communicates with DocIntel through a thin async HTTP client (src/purple8_graph/docintel_client.py). Key design decisions:

Connection Model

• No persistent connection — each call creates a short-lived httpx.AsyncClient. This keeps the client stateless and safe for multi-tenant use.
• Singleton pattern — get_docintel_client() returns a module-level instance to avoid re-reading env vars on every request.
• Per-tenant factory — _get_docintel_client_for_tenant() currently returns the singleton, but is designed for future per-tenant DocIntel routing.

Timeout Strategy

Operation	Timeout
GET requests (status, formats, health)	30 seconds
POST JSON (URL submission)	30 seconds
POST file upload	60 seconds
Job polling (total wait)	120 seconds
Poll interval	1.5 seconds

Result Normalisation

DocIntel returns entities and relationships in its own schema. The client includes static methods that normalise results into the shape expected by Graph's /ingest/commit:

# DocIntel shape → Graph shape
DocIntelClient.extract_entities(job)        # → [{ id, type, name, properties, include }]
DocIntelClient.extract_relationships(job)   # → [{ source, target, type, properties, include }]
DocIntelClient.job_to_preview(job)          # → full preview payload (see fields below)
DocIntelClient.extract_chunking_hint(job)   # → chunking hint dict or None

job_to_preview() now returns 5 additional top-level fields:

Field	Type	Description
doc_type	str	Document type classified by DocIntel (e.g. "legal_contract", "code")
chunking_hint	str	Recommended chunking strategy (e.g. "late", "syntactic")
section_boundaries	list[int]	Token offsets of major section breaks
suggested_embedding_model	str	Recommended embedding model (e.g. "voyage-4-large")
doc_type_confidence	float	DocIntel's classification confidence (0.0–1.0)

extract_chunking_hint(job) — convenience method for building a DocIntelChunkingHint directly:

hint_dict = DocIntelClient.extract_chunking_hint(job)
# Returns None if job result has no doc_type
# Otherwise: {
#   "doc_type": "legal_contract",
#   "chunking_hint": "late",
#   "suggested_embedding_model": "voyage-4-large",
#   "section_boundaries": [1024, 3072, 8192],
#   "confidence": 0.97
# }

This normalisation layer means DocIntel's internal schema can evolve without breaking the Graph API contract.

---

ChunkingAgent Integration

ChunkingAgent closes the loop between DocIntel's extraction intelligence and Graph's ingest pipeline. When chunking_agent_enabled: true is set in the RAG config, every /ingest/commit call automatically classifies the document and selects the optimal (chunking_strategy, embedding_model) pair.

The full pipeline

DocIntel job_to_preview()
        │
        │  doc_type, chunking_hint, suggested_embedding_model,
        │  section_boundaries, doc_type_confidence
        ▼
DocIntelClient.extract_chunking_hint(job)
        │
        │  → DocIntelChunkingHint(doc_type="legal_contract",
        │                         chunking_hint="late",
        │                         suggested_embedding_model="voyage-4-large",
        │                         confidence=0.97)
        ▼
POST /ingest/commit  {entities, relationships, doc_type, chunking_hint, ...}
        │
        │  chunking_agent_enabled=true
        ▼
ChunkingAgent.classify(text, filename, docintel_hint=hint)
        │  Fast-path: hint.doc_type → ChunkingStrategyRegistry lookup
        │  → confidence 1.0, no heuristic re-classification
        ▼
ChunkingDecision { doc_type, chunking_strategy, embedding_model, chunk_size,
                   chunk_overlap, classification_source, chunking_confidence }
        │
        ▼
Every ingested node gets chunking_props merged into its properties:
  { "chunking_strategy": "late", "embedding_model": "voyage-4-large",
    "doc_type": "legal_contract", "classification_source": "docintel",
    "chunking_confidence": 0.97 }

End-to-end example

Step 1: Enable ChunkingAgent in your RAG config:

curl -X PUT http://localhost:8000/rag/config \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"chunking_agent_enabled": true}'

Step 2: Submit a file to DocIntel and retrieve the preview:

import httpx, asyncio
from purple8_graph.docintel_client import DocIntelClient, get_docintel_client

async def ingest_with_chunking_agent(pdf_path: str, token: str):
    client = get_docintel_client()
    api = httpx.AsyncClient(base_url="http://localhost:8000")
    headers = {"Authorization": f"Bearer {token}"}

    # Submit file to DocIntel via Graph proxy
    with open(pdf_path, "rb") as f:
        r = await api.post("/ingest/preview/file",
            headers=headers,
            files={"file": (pdf_path, f, "application/pdf")})
    job = r.json()

    # Extract chunking hint from the job result
    hint = DocIntelClient.extract_chunking_hint(job)
    # hint = {"doc_type": "legal_contract", "chunking_hint": "late",
    #         "suggested_embedding_model": "voyage-4-large",
    #         "section_boundaries": [1024, 3072, 8192], "confidence": 0.97}

    # Commit — pass hint fields alongside approved entities
    commit_payload = {
        "entities": job["entities"],
        "relationships": job["relationships"],
        "source_name": pdf_path,
    }
    if hint:
        commit_payload.update({
            "doc_type": hint["doc_type"],
            "chunking_hint": hint["chunking_hint"],
            "suggested_embedding_model": hint["suggested_embedding_model"],
            "section_boundaries": hint.get("section_boundaries", []),
        })

    result = await api.post("/ingest/commit", headers=headers, json=commit_payload)
    print(result.json())
    # → {"committed": 12, "chunking": {"doc_type": "legal_contract",
    #     "chunking_strategy": "late", "embedding_model": "voyage-4-large",
    #     "classification_source": "docintel", "chunking_confidence": 0.97}}

Or via curl (pass hint fields straight from the preview response):

# 1. Get preview (which now includes doc_type, chunking_hint, etc.)
PREVIEW=$(curl -s -X POST http://localhost:8000/ingest/preview/file \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@contract_q1_2026.pdf")

# 2. Commit with the hint fields
curl -X POST http://localhost:8000/ingest/commit \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"entities\": $(echo $PREVIEW | jq '.entities'),
    \"relationships\": $(echo $PREVIEW | jq '.relationships'),
    \"source_name\": \"contract_q1_2026.pdf\",
    \"doc_type\": $(echo $PREVIEW | jq -r '.doc_type'),
    \"chunking_hint\": $(echo $PREVIEW | jq -r '.chunking_hint'),
    \"suggested_embedding_model\": $(echo $PREVIEW | jq -r '.suggested_embedding_model')
  }"

DocIntel vs heuristic classification

When chunking_agent_enabled is true, ChunkingAgent runs two paths:

Scenario	Path	Confidence	Behaviour
DocIntel job with doc_type	Fast-path	DocIntel's confidence (e.g. 0.97)	DocIntelChunkingHint passed → registry lookup; no heuristic re-run
Direct text ingest (no file)	Heuristic	0.40–0.95	Extension → MIME → content patterns → fallback "document"
DocIntel job, no doc_type field	Heuristic	0.40–0.95	extract_chunking_hint() returns None; agent falls back to heuristic

The fast-path is always preferred when available — it leverages DocIntel's purpose-built extraction intelligence at zero additional cost.

Customising the registry

The default ChunkingStrategyRegistry covers 8 doc types. Override it per-deployment:

from purple8_graph.chunking_agent import (
    ChunkingStrategyRegistry, ChunkingRule, save_chunking_registry
)

# Start from the default and add a custom doc type
registry = ChunkingStrategyRegistry.default()
registry.rules["engineering_spec"] = ChunkingRule(
    chunking_strategy="parent_child",
    embedding_model="voyage-4-large",
    chunk_size=2048,
    chunk_overlap=256,
)
save_chunking_registry("chunking_registry.json", registry)

Then load it at startup via the PURPLE8_CHUNKING_REGISTRY env var (or pass registry_path to ChunkingAgent).

---

Security

Authentication

DocIntel uses Bearer token authentication. The token is set via PURPLE8_DOCINTEL_KEY on the Graph side and validated by DocIntel's middleware.

Graph → DocIntel:
  Authorization: Bearer <PURPLE8_DOCINTEL_KEY>

Network Isolation

In production deployments:

1. DocIntel should not be exposed on public interfaces. Only the Graph API needs access.
2. Use Docker internal networks or Kubernetes ClusterIP services.
3. For multi-region deployments, secure the link with mTLS or a service mesh.

Data in Transit

• Sidecar pattern: Data stays on localhost — no encryption needed.
• Remote pattern: Use HTTPS (PURPLE8_DOCINTEL_URL=https://...). The httpx client respects TLS verification.

---

Monitoring

Health Checks

# DocIntel liveness
curl http://localhost:8200/health
# → { "status": "ok" }

# DocIntel readiness (checks LLM + job store connectivity)
curl http://localhost:8200/ready
# → { "status": "ready", "llm": "ok", "job_store": "ok" }

Graph-Side Detection

from purple8_graph.docintel_client import is_docintel_configured

if is_docintel_configured():
    # PURPLE8_DOCINTEL_URL is set — DocIntel integration active
    ...
else:
    # Running in standalone mode — fallback parser only
    ...

Prometheus Metrics

Both services expose /metrics endpoints. Key DocIntel metrics to monitor:

Metric	Description
docintel_jobs_total	Total jobs submitted (by format)
docintel_jobs_duration_seconds	Processing time per job
docintel_jobs_failed_total	Failed jobs (by error type)
docintel_parse_bytes_total	Total bytes parsed

---

Troubleshooting

DocIntel Not Connected

Symptom: GET /ingest/formats returns the static fallback list with "warning": "DocIntel service unavailable".

Fix:

1. Verify DocIntel is running: curl http://localhost:8200/health
2. Check PURPLE8_DOCINTEL_URL is set on the Graph service
3. Verify network connectivity between containers (same Docker network / K8s namespace)

File Upload Timeout

Symptom: POST /ingest/preview/file returns 502 after ~60 seconds.

Fix:

1. Use ?async_mode=true for files > 500 KB
2. Increase DocIntel's worker count for CPU-intensive formats (IFC, DWG)
3. Check DocIntel logs for OOM or parser crashes

Extraction Quality Issues

Symptom: Entities are missing or relationships are incorrect.

Fix:

1. Check the extraction model — gpt-4o produces better results than gpt-4o-mini
2. For domain-specific documents, consider DocIntel's custom prompt templates
3. Review chunk boundaries — semantic or sentence_window strategies may improve extraction on long documents

License Errors

Symptom: DocIntel returns 403 or startup logs show "invalid license".

Fix:

• Graph and DocIntel have separate licenses. Ensure both are set:
  • Graph: PURPLE8_LICENSE_KEY (or PURPLE8_DEV_MODE=1 for local dev)
  • DocIntel: LICENSE__KEY

---

Migration from Built-in Parser

If you're upgrading from Purple8 Hyper Graph < v0.25.0 (which used the built-in parser):

1. Install DocIntel alongside your Graph deployment
2. Set PURPLE8_DOCINTEL_URL — this is the only Graph-side config change needed
3. No data migration required — existing graph nodes are unaffected
4. Re-ingest documents if you want richer entity extraction from DocIntel's LLM pipeline

The Graph API surface (/ingest/preview, /ingest/commit) is unchanged. Client code that calls these endpoints will work without modification.