Document Intelligence

Purple8 Hyper Graph integrates with the Purple8 Document Intelligence microservice (DocIntel v0.12.5). All parsing, OCR, CAD/BIM support, format detection, language detection, LLM context windowing, and two-pass hybrid entity/relationship extraction run inside DocIntel — the Graph ingestion pipeline delegates directly to it.

API-first — no UI required

All DocIntel features are accessible via pip install purple8-hyper-graph and REST APIs (/ingest/preview, /ingest/preview/file, /ingest/commit). The optional web UI is a visual overlay for the same endpoints. See the Integration Patterns guide for the end-to-end developer workflow.

Highlights in DocIntel v0.12.5
70+ document formats across 75+ languages (PDF, DOCX, XLSX, PPTX, HTML, EML, EPUB, CAD/BIM IFC/IFCZIP/DXF/DWG/STEP/STL/OBJ/GLTF/G-code, SAP IDocs, and scanned images/documents)
Two-pass hybrid extraction: Pass 1 GLiNER NER → Pass 2 domain-tuned LLM relationship mapping across 5 domain profiles (AEC, Contract, Financial, Scientific, General)
5 OCR engines including self-hosted P8OCR and Tesseract, plus vision-model backends (openai-vision, azure-vision, google-vision)
Enterprise parser hardening: 100 MiB ODF zip-bomb protection and XML entity expansion guards
IFCParser 10-section structured output across IFC 2x3, IFC 4, IFC 4x3, and IFCZIP; SketchParser for hand-drawn diagrams, scanned blueprints, and whiteboards

How it works

┌──────────────────────────────────────────────────────────┐
│                   Purple8 Hyper Graph Server             │
│                                                          │
│   POST /ingest/preview          POST /ingest/preview/file│
│         │                               │                │
│         ▼                               ▼                │
│   docintel_client.py  ◄────────────────────────────────  │
│         │                                                │
└─────────┼────────────────────────────────────────────────┘
          │  HTTP (port 8200)
          ▼
┌──────────────────────────────────────────────────────────┐
│        Purple8 Document Intelligence (v0.12.5)           │
│                                                          │
│   Detect format & language (75+ languages)               │
│   → Parse (100 MiB ODF zip-bomb & XML entity guards)     │
│   → Pass 1: GLiNER NER entity extraction                 │
│   → Pass 2: Domain-tuned LLM relationship mapping        │
│     (AEC · Contract · Financial · Scientific · General)  │
│                                                          │
│   70+ formats: PDF · DOCX · XLSX · PPTX · HTML · EML     │
│   EPUB · IFC · IFCZIP · DXF · DWG · STEP · STL · OBJ     │
│   GLTF · G-code · SAP IDoc · ABAP · Scanned OCR          │
│   (5 OCR engines incl. P8OCR & Tesseract)                │
└──────────────────────────────────────────────────────────┘
          │
          ▼  ProcessingJob { status, progress_pct, result }
┌──────────────────────────────────────────────────────────┐
│       Graph Engine (BrickCoreStorage / Cortex)           │
│       ← normalised entities + relationships              │
└──────────────────────────────────────────────────────────┘

DocIntel handles 70+ document formats across 75+ languages with multi-engine OCR (5 OCR engines including self-hosted P8OCR and Tesseract), native CAD/BIM parsing, two-pass hybrid extraction, and no chunk limit.

Setup

1. Run Purple8 DocIntel

DocIntel is a separate microservice. The simplest way is Docker:

bash

docker run -d \
  --name purple8-docintel \
  -p 8200:8200 \
  -e OPENAI_API_KEY=sk-... \
  purple8/purple8-docintel:0.12.5

Or run from source with the extras you need:

bash

cd Purple8-DocIntel

# Core (txt, md, html, json, yaml, csv, rst, rtf, epub)
pip install -e "."

# Add PDF support
pip install -e ".[pdf]"

# Add Office formats (DOCX, XLSX, PPTX, ODF, EML, MSG)
pip install -e ".[office,email]"

# Add OCR (PNG, JPG, TIFF, BMP, WEBP — Tesseract)
pip install -e ".[ocr]"

# Add CAD (DXF, DWG via ezdxf)
pip install -e ".[cad]"

# Add BIM (IFC, IFCZIP via ifcopenshell)
pip install -e ".[bim]"

# Add Sketch / whiteboard (GPT-4o vision)
pip install -e ".[sketch]"

# Add Sketch via Azure AI Vision instead
pip install -e ".[sketch-azure]"

# Full install — all formats + all connectors
pip install -e ".[all]"

python -m purple8_docintel.server
# → listening on http://localhost:8200

2. Configure Purple8 Hyper Graph

Set environment variables before starting the Graph server:

bash

export PURPLE8_DOCINTEL_URL=http://localhost:8200   # required
export PURPLE8_DOCINTEL_KEY=your-api-key            # optional

If PURPLE8_DOCINTEL_URL is not set, the Graph server falls back to a lightweight genai-based plain-text extractor for .txt / .md content. All other formats require DocIntel.

3. Verify the connection

bash

curl http://localhost:8000/ingest/formats

json

{
  "formats": ["pdf", "docx", "xlsx", "pptx", "txt", "md", "html",
              "eml", "epub", "png", "jpg", "tiff", "ifc", "ifczip",
              "dxf", "dwg", "step", "stl", "obj", "gltf", "glb",
              "gcode", "idoc", "abap", "sketch", "json", "yaml",
              "csv", "rst", ...],
  "count": 72
}

REST API

`POST /ingest/preview`

Extract entities and relationships from a URL or raw text.

Request

json

{
  "url": "https://example.com/spec.pdf",
  "source_name": "project-spec"
}

Or with raw text:

json

{
  "text": "Alice Chen works at Acme Corp as a senior engineer…",
  "source_name": "notes"
}

Response

json

{
  "source_name": "project-spec",
  "chunks_processed": 14,
  "total_chars": 48200,
  "extraction_model": "gpt-4o",
  "extraction_duration_s": 3.2,
  "entities": [
    {
      "id": "entity_alice_chen",
      "type": "Person",
      "name": "Alice Chen",
      "properties": { "role": "Senior Engineer" },
      "confidence": 0.97,
      "include": true
    }
  ],
  "relationships": [
    {
      "source": "entity_alice_chen",
      "type": "WORKS_AT",
      "target": "entity_acme_corp",
      "properties": {},
      "confidence": 0.94,
      "include": true
    }
  ]
}

`POST /ingest/preview/file`

Upload a file for extraction. Supports ?async_mode=true for large files.

bash

# Synchronous (small files, < 500 KB recommended)
curl -X POST http://localhost:8000/ingest/preview/file \
  -F "file=@report.pdf"

# Async (large files — returns job_id immediately)
curl -X POST "http://localhost:8000/ingest/preview/file?async_mode=true" \
  -F "file=@large-bim-model.ifc"

Async response

json

{
  "job_id": "job_abc123",
  "status": "queued",
  "source_name": "large-bim-model.ifc",
  "message": "Job submitted to DocIntel. Poll GET /ingest/jobs/job_abc123 for progress."
}

`GET /ingest/jobs/{job_id}`

Poll job status and retrieve results when complete.

bash

curl http://localhost:8000/ingest/jobs/job_abc123

json

{
  "job_id": "job_abc123",
  "status": "extracting",
  "progress_pct": 62,
  "source_name": "large-bim-model.ifc",
  "created_at": "2026-03-27T10:00:00Z",
  "preview": null
}

When status is "done":

json

{
  "job_id": "job_abc123",
  "status": "done",
  "progress_pct": 100,
  "preview": {
    "entities": [...],
    "relationships": [...],
    "extraction_model": "gpt-4o",
    "extraction_duration_s": 18.4
  }
}

`GET /ingest/formats`

Returns the list of all document formats supported by the connected DocIntel instance.

bash

curl http://localhost:8000/ingest/formats

`GET /ingest/jobs`

List all jobs in the DocIntel job store.

bash

curl "http://localhost:8000/ingest/jobs?limit=20&offset=0"

Supported formats (70+ formats, 75+ languages)

Category	Formats	Extra
Documents	`pdf`, `docx`, `doc`, `odt`, `rtf`, `txt`, `md`, `rst`, `epub`, `tex`, `org`, `adoc`	`[pdf]`, `[office]`
Spreadsheets	`xlsx`, `xls`, `ods`, `csv`, `tsv`, `parquet`	`[office]`
Presentations	`pptx`, `ppt`, `odp`	`[office]`
Web / Markup	`html`, `htm`, `xml`, `json`, `jsonl`, `yaml`, `yml`, `toml`, `ini`	built-in
Email	`eml`, `msg`, `mbox`	`[email]`
Images & Scanned OCR	`png`, `jpg`, `jpeg`, `tiff`, `tif`, `gif`, `bmp`, `webp`, `heic` (via 5 OCR engines including self-hosted `P8OCR` and Tesseract)	`[ocr]`
CAD	`dxf`, `dwg`	`[cad]`
BIM	`ifc`, `ifczip`	`[bim]`
3D / Manufacturing	`step`, `stp`, `stl`, `obj`, `glb`, `gltf`, `gcode`, `iges`, `3mf`	`[cad]` / built-in
Enterprise / SAP	`idoc`, `abap`, `edifact`, `x12`	built-in
Sketch / Whiteboard	`sketch` (hand-drawn, scanned blueprints)	`[sketch]` or `[sketch-azure]`

Two-Pass Hybrid Extraction & Security Hardening

Two-pass hybrid extraction

DocIntel v0.12.5 uses a two-pass extraction pipeline to combine high-recall entity recognition with precision relationship mapping:

Pass 1 — GLiNER NER: Fast, zero-shot and fine-tuned span-level named entity recognition extracts candidate entities with character offsets and confidence scores across 75+ languages.
Pass 2 — Domain-Tuned LLM Relationship Mapping: Maps typed edges and normalises entity attributes using one of 5 built-in domain profiles:
- AEC — buildings, storeys, spaces, structural members, MEP components, IFC properties
- Contract — parties, clauses, obligations, effective dates, governing law, indemnities
- Financial — accounts, transactions, counterparties, instruments, regulatory filings
- Scientific — methods, datasets, citations, compounds, measurements, hypotheses
- General — people, organisations, locations, projects, events, concepts

Parser security hardening

All archive-backed and XML-based formats (ODF .odt/.ods/.odp, Office Open XML .docx/.xlsx/.pptx, .epub, .ifczip, .xml, and SAP IDoc XML) are protected by:

100 MiB ODF/archive zip-bomb protection — enforces strict decompressed byte budgets and compression-ratio limits before loading entries into memory.
XML entity expansion guards — rejects external DTDs, XXE payloads, and quadratic/billion-laughs entity expansion attacks at the parser boundary.

IFC / BIM parsing

The IFCParser produces a 10-section structured document for every .ifc or .ifczip file:

Section	Content
Project metadata	Name, description, site, building, address
Storeys	Name + elevation for every `IfcBuildingStorey`
Spaces	Room / zone name, long name, type, storey
Elements	Walls, slabs, beams, columns, doors, windows, MEP elements — grouped by type
Materials	Layer sets, constituent sets, profile sets fully described
Property sets	All `Pset_*` and user-defined property sets
Quantities	Area (m²), volume (m³), length, weight, count per element
Type objects	Classifications: Uniclass, Omniclass, NBS codes
Structural members	Curve / surface members, connections, actions
Miscellaneous	Groups, document refs, cost / work schedules, tasks, actors

Requires [bim] extra (ifcopenshell >= 0.8). Supports IFC 2x3, IFC 4, IFC 4x3 and IFCZIP.

Sketch & whiteboard parsing

The SketchParser analyses hand-drawn diagrams, scanned blueprints, and whiteboard photos using a vision or OCR backend. It produces an 8-section structured extraction including spaces, elements, connections, annotations, and materials.

OCR & Vision Backends (5 OCR engines supported across DocIntel):

Backend	Install	Config
`p8ocr` (self-hosted)	`[ocr]`	`OCR__ENGINE=p8ocr` (self-hosted neural OCR)
`tesseract`	`[ocr]`	`OCR__ENGINE=tesseract` / `SKETCH__ENGINE=local`
`openai-vision` (default for sketch)	`[sketch]`	`SKETCH__ENGINE=openai-vision`, `SKETCH__MODEL=gpt-4o`
`azure-vision`	`[sketch-azure]`	`SKETCH__ENGINE=azure-vision`
`google-vision`	`[sketch]`	`SKETCH__ENGINE=google-vision`

Key env vars:

Variable	Default	Description
`SKETCH__ENGINE`	`openai-vision`	Vision backend
`SKETCH__MODEL`	`gpt-4o`	OpenAI model (for `openai-vision`)
`SKETCH__DETAIL_LEVEL`	`high`	`low` \| `high` \| `auto`
`SKETCH__MAX_TOKENS`	`2048`	Max tokens for the vision model response
`SKETCH__CONFIDENCE_THRESHOLD`	`0.5`	Min confidence for Azure / Google labels (0–1)

Job store options

DocIntel supports two job stores. Configure via STORAGE__JOB_STORE:

Store	Config	Use case
`purple8` (default)	`STORAGE__JOB_STORE=purple8` + `STORAGE__PURPLE8_PATH=./docintel_jobs.p8l`	Embedded Purple8 Lite WAL store — no external process; survives restarts
`memory`	`STORAGE__JOB_STORE=memory`	Unit tests / ephemeral use only; jobs reset on restart

The purple8 store is an embedded, single-file WAL-backed job store — there is no separate Redis or SQLite backend to configure.

Deployment

DocIntel ships production-ready configs in deploy/. See deploy/README.md for the full runbook.

Docker Compose (recommended for teams)

bash

# Core service only
docker compose up -d

systemd (Linux VM / bare metal)

bash

sudo cp deploy/systemd/purple8-docintel.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now purple8-docintel

Kubernetes

bash

# Set the image tag in deploy/k8s/kustomization.yaml, then:
kubectl create namespace purple8
kubectl apply -k deploy/k8s/
kubectl rollout status deployment/purple8-docintel -n purple8

The K8s manifests include: Deployment (liveness + readiness probes), Service (ClusterIP, port 8200), ConfigMap (non-secret env vars), Secret template, PersistentVolumeClaim for the embedded purple8 job store.

HTTPS / reverse proxy

A production-hardened nginx.conf is in deploy/nginx/ — handles TLS termination, upstream keepalive, and connector header injection.

Health checks

bash

curl http://localhost:8200/health
# → {"status": "ok", "version": "0.12.5"}

curl http://localhost:8200/ready
# → {"status": "ready", "graph": "ok", "job_store": "ok"}

Ingestion Pipeline UI (optional)

Low-code convenience — not required

The web UI is an optional visual overlay for teams who prefer a point-and-click workflow. It calls exactly the same REST APIs documented above. Most developers integrate directly via curl, Python, or any HTTP client and can skip this section entirely.

The Ingestion Pipeline page in the Purple8 Hyper Graph web app (port 3000) reflects all DocIntel capabilities:

Step 1: Source

File dropzone — accept list is loaded dynamically from GET /ingest/formats on page load, so it always reflects what the connected DocIntel instance supports
URL mode — fetches and parses the URL through DocIntel (handles redirects, auth, and multi-page documents)
Text paste — sends raw text directly to DocIntel for entity extraction

Step 2: Extract

Async job progress bar — for files > 500 KB, the upload returns a job_id and the UI polls GET /ingest/jobs/{job_id} every 1.5 seconds; a live progress bar shows the current stage (queued → parsing → extracting → emitting → done)
Stats banner — shows chunks processed, total characters, entity count, relationship count, extraction model, and extraction duration
Fallback notice — if DocIntel was unavailable and the genai fallback was used, a yellow warning banner is shown
Entity cards — each entity shows its type badge, name (editable inline), up to 3 property key/value pairs, and a confidence percentage if < 100%
Relationship rows — each relationship shows source → type → target with a toggle for inclusion

Steps 3 & 4

Compare and Publish steps are unchanged — see the Ingest commit API for commit and publish details.

Python client usage

You can call docintel_client.py directly from your own Python code:

python

from purple8_graph.docintel_client import get_docintel_client, is_docintel_configured

if not is_docintel_configured():
    print("Set PURPLE8_DOCINTEL_URL to enable DocIntel")
else:
    client = get_docintel_client()

    # Process a file asynchronously
    import asyncio

    async def extract_from_file(path: str):
        with open(path, "rb") as f:
            data = f.read()

        job = await client.process_file(path.split("/")[-1], data)
        job = await client.wait_for_job(job["job_id"], timeout=120)

        entities = client.extract_entities(job)
        rels = client.extract_relationships(job)
        print(f"Extracted {len(entities)} entities, {len(rels)} relationships")
        return client.job_to_preview(job)

    preview = asyncio.run(extract_from_file("path/to/report.pdf"))

Or use the convenience "submit and wait" helpers (still async — call from a coroutine):

python

from purple8_graph.docintel_client import get_docintel_client
import asyncio

async def main():
    client = get_docintel_client()

    # From URL
    job = await client.process_url_sync("https://example.com/spec.pdf")

    # From file bytes
    with open("model.ifc", "rb") as f:
        job = await client.process_file_sync("model.ifc", f.read())

    preview = client.job_to_preview(job)
    print(preview["entities"])

asyncio.run(main())

Fallback behaviour

If PURPLE8_DOCINTEL_URL is not set or DocIntel returns an error, the Graph server falls back gracefully:

Source	Fallback behaviour
URL / text paste	Uses `genai.KnowledgeExtractor` with the configured LLM provider — works for plain prose, no structured format support
File upload	Returns HTTP 503 with a clear error message; no silent partial extraction
`/ingest/formats`	Returns a static list of 14 common formats with a `"warning"` field

The response includes "_fallback": true so the UI (and any downstream code) can surface the degraded-mode warning.

Environment variables

Variable	Default	Description
`PURPLE8_DOCINTEL_URL`	(not set)	Base URL of the DocIntel microservice, e.g. `http://localhost:8200`
`PURPLE8_DOCINTEL_KEY`	(not set)	Optional API key for DocIntel authentication

See Environment Variables for the full configuration reference.

Document Intelligence ​

How it works ​

Setup ​

1. Run Purple8 DocIntel ​

2. Configure Purple8 Hyper Graph ​

3. Verify the connection ​

REST API ​

POST /ingest/preview ​

POST /ingest/preview/file ​

GET /ingest/jobs/{job_id} ​

GET /ingest/formats ​

GET /ingest/jobs ​

Supported formats (70+ formats, 75+ languages) ​

Two-Pass Hybrid Extraction & Security Hardening ​

Two-pass hybrid extraction ​

Parser security hardening ​

IFC / BIM parsing ​

Sketch & whiteboard parsing ​

Job store options ​

Deployment ​

Docker Compose (recommended for teams) ​

systemd (Linux VM / bare metal) ​

Kubernetes ​

HTTPS / reverse proxy ​

Health checks ​

Ingestion Pipeline UI (optional) ​

Step 1: Source ​

Step 2: Extract ​

Steps 3 & 4 ​

Python client usage ​

Fallback behaviour ​

Environment variables ​

Document Intelligence

How it works

Setup

1. Run Purple8 DocIntel

2. Configure Purple8 Hyper Graph

3. Verify the connection

REST API

`POST /ingest/preview`

`POST /ingest/preview/file`

`GET /ingest/jobs/{job_id}`

`GET /ingest/formats`

`GET /ingest/jobs`

Supported formats (70+ formats, 75+ languages)

Two-Pass Hybrid Extraction & Security Hardening

Two-pass hybrid extraction

Parser security hardening

IFC / BIM parsing

Sketch & whiteboard parsing

Job store options

Deployment

Docker Compose (recommended for teams)

systemd (Linux VM / bare metal)

Kubernetes

HTTPS / reverse proxy

Health checks

Ingestion Pipeline UI (optional)

Step 1: Source

Step 2: Extract

Steps 3 & 4

Python client usage

Fallback behaviour

Environment variables