Document Intelligence

Purple8 Hyper Graph v0.25.0 replaces the previous hand-rolled document parser with a full integration of the Purple8 Document Intelligence microservice (DocIntel v0.4.2). All parsing, OCR, CAD/BIM support, format detection, LLM context windowing, and entity/relationship extraction now run inside DocIntel — the Graph ingestion pipeline simply delegates to it.

API-first — no UI required

All DocIntel features are accessible via pip install purple8-hyper-graph and REST APIs (/ingest/preview, /ingest/preview/file, /ingest/commit). The optional web UI is a visual overlay for the same endpoints. See the Integration Patterns guide for the end-to-end developer workflow.

What's new in DocIntel v0.4.2
48/48 tests green — all job stores (memory, redis, sqlite) have clear() and pop() for clean lifecycle management
/formats returns correct string values ("pdf") not enum reprs ("DocumentFormat.PDF")
Production deploy configs: systemd unit, nginx reverse proxy, Kubernetes manifests (deploy/k8s/)
.env.example fully rewritten with all production env vars documented
What's new in DocIntel v0.4.1
IFCParser fully rewritten — 10-section structured output for all IFC data domains
SketchParser — hand-drawn diagrams, scanned blueprints, and whiteboards via 4 vision backends (GPT-4o, Azure, Google, Tesseract)
DXF / JPEG magic-byte parser bug fixes

How it works

┌──────────────────────────────────────────────────────────┐
│                   Purple8 Hyper Graph Server                   │
│                                                          │
│   POST /ingest/preview          POST /ingest/preview/file│
│         │                               │                │
│         ▼                               ▼                │
│   docintel_client.py  ◄────────────────────────────────  │
│         │                                                │
└─────────┼────────────────────────────────────────────────┘
          │  HTTP (port 8200)
          ▼
┌──────────────────────────────────────────────────────────┐
│              Purple8 Document Intelligence               │
│                                                          │
│   Detect format → Parse → Split context windows         │
│   → LLM extract entities & rels → Emit to Graph         │
│                                                          │
│   40+ formats: PDF · DOCX · XLSX · PPTX · PNG/JPG/TIFF  │
│   IFC · DXF · DWG · STEP · SAP IDoc · ABAP · G-code     │
│   JSON · YAML · CSV · EPUB · RST · Sketch · GLB …       │
└──────────────────────────────────────────────────────────┘
          │
          ▼  ProcessingJob { status, progress_pct, result }
┌──────────────────────────────────────────────────────────┐
│       Graph Engine   ← normalised entities + rels        │
└──────────────────────────────────────────────────────────┘

The previous parser only handled 6 formats (txt, md, html, pdf, docx, doc) with a hard 20-chunk cap and no OCR. DocIntel handles 40+ formats with full OCR, native CAD/BIM parsing, and no chunk limit.

Setup

1. Run Purple8 DocIntel

DocIntel is a separate microservice. The simplest way is Docker:

bash

docker run -d \
  --name purple8-docintel \
  -p 8200:8200 \
  -e OPENAI_API_KEY=sk-... \
  purple8/purple8-docintel:0.4.2

Or run from source with the extras you need:

bash

cd Purple8-DocIntel

# Core (txt, md, html, json, yaml, csv, rst, rtf, epub)
pip install -e "."

# Add PDF support
pip install -e ".[pdf]"

# Add Office formats (DOCX, XLSX, PPTX, ODF, EML, MSG)
pip install -e ".[office,email]"

# Add OCR (PNG, JPG, TIFF, BMP, WEBP — Tesseract)
pip install -e ".[ocr]"

# Add CAD (DXF, DWG via ezdxf)
pip install -e ".[cad]"

# Add BIM (IFC, IFCZIP via ifcopenshell)
pip install -e ".[bim]"

# Add Sketch / whiteboard (GPT-4o vision)
pip install -e ".[sketch]"

# Add Sketch via Azure AI Vision instead
pip install -e ".[sketch-azure]"

# Full install — all formats + all connectors
pip install -e ".[all]"

python -m purple8_docintel.server
# → listening on http://localhost:8200

2. Configure Purple8 Hyper Graph

Set environment variables before starting the Graph server:

bash

export PURPLE8_DOCINTEL_URL=http://localhost:8200   # required
export PURPLE8_DOCINTEL_KEY=your-api-key            # optional

If PURPLE8_DOCINTEL_URL is not set, the Graph server falls back to a lightweight genai-based plain-text extractor for .txt / .md content. All other formats require DocIntel.

3. Verify the connection

bash

curl http://localhost:8000/ingest/formats

json

{
  "formats": ["pdf", "docx", "xlsx", "pptx", "txt", "md", "html",
              "png", "jpg", "tiff", "ifc", "dxf", "dwg", "step",
              "idoc", "abap", "gcode", "sketch", "json", "yaml",
              "csv", "epub", "rst", "glb", "stl", ...],
  "count": 42
}

REST API

`POST /ingest/preview`

Extract entities and relationships from a URL or raw text.

Request

json

{
  "url": "https://example.com/spec.pdf",
  "source_name": "project-spec"
}

Or with raw text:

json

{
  "text": "Alice Chen works at Acme Corp as a senior engineer…",
  "source_name": "notes"
}

Response

json

{
  "source_name": "project-spec",
  "chunks_processed": 14,
  "total_chars": 48200,
  "extraction_model": "gpt-4o",
  "extraction_duration_s": 3.2,
  "entities": [
    {
      "id": "entity_alice_chen",
      "type": "Person",
      "name": "Alice Chen",
      "properties": { "role": "Senior Engineer" },
      "confidence": 0.97,
      "include": true
    }
  ],
  "relationships": [
    {
      "source": "entity_alice_chen",
      "type": "WORKS_AT",
      "target": "entity_acme_corp",
      "properties": {},
      "confidence": 0.94,
      "include": true
    }
  ]
}

`POST /ingest/preview/file`

Upload a file for extraction. Supports ?async_mode=true for large files.

bash

# Synchronous (small files, < 500 KB recommended)
curl -X POST http://localhost:8000/ingest/preview/file \
  -F "file=@report.pdf"

# Async (large files — returns job_id immediately)
curl -X POST "http://localhost:8000/ingest/preview/file?async_mode=true" \
  -F "file=@large-bim-model.ifc"

Async response

json

{
  "job_id": "job_abc123",
  "status": "queued",
  "source_name": "large-bim-model.ifc",
  "message": "Job submitted to DocIntel. Poll GET /ingest/jobs/job_abc123 for progress."
}

`GET /ingest/jobs/{job_id}`

Poll job status and retrieve results when complete.

bash

curl http://localhost:8000/ingest/jobs/job_abc123

json

{
  "job_id": "job_abc123",
  "status": "extracting",
  "progress_pct": 62,
  "source_name": "large-bim-model.ifc",
  "created_at": "2026-03-27T10:00:00Z",
  "preview": null
}

When status is "done":

json

{
  "job_id": "job_abc123",
  "status": "done",
  "progress_pct": 100,
  "preview": {
    "entities": [...],
    "relationships": [...],
    "extraction_model": "gpt-4o",
    "extraction_duration_s": 18.4
  }
}

`GET /ingest/formats`

Returns the list of all document formats supported by the connected DocIntel instance.

bash

curl http://localhost:8000/ingest/formats

`GET /ingest/jobs`

List all jobs in the DocIntel job store.

bash

curl "http://localhost:8000/ingest/jobs?limit=20&offset=0"

Supported formats

Category	Formats	Extra
Documents	`pdf`, `docx`, `doc`, `odt`, `rtf`, `txt`, `md`, `rst`, `epub`	`[pdf]`, `[office]`
Spreadsheets	`xlsx`, `xls`, `ods`, `csv`, `tsv`	`[office]`
Presentations	`pptx`, `ppt`, `odp`	`[office]`
Web / Markup	`html`, `htm`, `xml`, `json`, `jsonl`, `yaml`, `yml`, `toml`	built-in
Email	`eml`, `msg`	`[email]`
Images (OCR)	`png`, `jpg`, `jpeg`, `tiff`, `tif`, `gif`, `bmp`, `webp`	`[ocr]`
CAD	`dxf`, `dwg`	`[cad]`
BIM	`ifc`, `ifczip`	`[bim]`
3D / Manufacturing	`step`, `stp`, `stl`, `obj`, `glb`, `gltf`, `gcode`	`[cad]` / built-in
Enterprise / SAP	`idoc`, `abap`	built-in
Sketch / Whiteboard	`sketch` (hand-drawn, scanned blueprints)	`[sketch]` or `[sketch-azure]`

IFC / BIM parsing (v0.4.1+)

The IFCParser produces a 10-section structured document for every .ifc or .ifczip file:

Section	Content
Project metadata	Name, description, site, building, address
Storeys	Name + elevation for every `IfcBuildingStorey`
Spaces	Room / zone name, long name, type, storey
Elements	Walls, slabs, beams, columns, doors, windows, MEP elements — grouped by type
Materials	Layer sets, constituent sets, profile sets fully described
Property sets	All `Pset_*` and user-defined property sets
Quantities	Area (m²), volume (m³), length, weight, count per element
Type objects	Classifications: Uniclass, Omniclass, NBS codes
Structural members	Curve / surface members, connections, actions
Miscellaneous	Groups, document refs, cost / work schedules, tasks, actors

Requires [bim] extra (ifcopenshell >= 0.8). Supports IFC 2x3, IFC 4, IFC 4x3 and IFCZIP.

Sketch & whiteboard parsing (v0.4.1+)

The SketchParser analyses hand-drawn diagrams, scanned blueprints, and whiteboard photos using a vision model. It produces an 8-section structured extraction including spaces, elements, connections, annotations, and materials.

Backends:

Backend	Install	Config
`openai-vision` (default)	`[sketch]`	`SKETCH__ENGINE=openai-vision`, `SKETCH__MODEL=gpt-4o`
`azure-vision`	`[sketch-azure]`	`SKETCH__ENGINE=azure-vision`
`google-vision`	`[sketch]`	`SKETCH__ENGINE=google-vision`
`local`	`[ocr]`	`SKETCH__ENGINE=local` (Tesseract, lower accuracy)

Key env vars:

Variable	Default	Description
`SKETCH__ENGINE`	`openai-vision`	Vision backend
`SKETCH__MODEL`	`gpt-4o`	OpenAI model (for `openai-vision`)
`SKETCH__DETAIL_LEVEL`	`high`	`low` \| `high` \| `auto`
`SKETCH__MAX_TOKENS`	`2048`	Max tokens for the vision model response
`SKETCH__CONFIDENCE_THRESHOLD`	`0.5`	Min confidence for Azure / Google labels (0–1)

Job store options

DocIntel supports three job stores. Configure via STORAGE__JOB_STORE:

Store	Config	Use case
`memory` (default)	`STORAGE__JOB_STORE=memory`	Development / single-process; jobs reset on restart
`sqlite`	`STORAGE__JOB_STORE=sqlite` + `STORAGE__SQLITE_PATH=./data/jobs.db`	Single-server persistence; survives restarts
`redis`	`STORAGE__JOB_STORE=redis` + `STORAGE__REDIS_URL=redis://localhost:6379/0`	Multi-worker / horizontal scaling

Use redis when running multiple DocIntel workers behind a load balancer so all workers share the same job state.

Deployment

DocIntel ships production-ready configs in deploy/. See deploy/README.md for the full runbook.

Docker Compose (recommended for teams)

bash

# Core service only
docker compose up -d

# With Redis job store
docker compose --profile redis up -d

systemd (Linux VM / bare metal)

bash

sudo cp deploy/systemd/purple8-docintel.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now purple8-docintel

Kubernetes

bash

# Set the image tag in deploy/k8s/kustomization.yaml, then:
kubectl create namespace purple8
kubectl apply -k deploy/k8s/
kubectl rollout status deployment/purple8-docintel -n purple8

The K8s manifests include: Deployment (liveness + readiness probes), Service (ClusterIP, port 8200), ConfigMap (non-secret env vars), Secret template, 5 Gi PersistentVolumeClaim for the SQLite job store.

HTTPS / reverse proxy

A production-hardened nginx.conf is in deploy/nginx/ — handles TLS termination, upstream keepalive, and connector header injection.

Health checks

bash

curl http://localhost:8200/health
# → {"status": "ok", "version": "0.4.2"}

curl http://localhost:8200/ready
# → {"status": "ready", "graph": "ok", "job_store": "ok"}

Ingestion Pipeline UI (optional)

Low-code convenience — not required

The web UI is an optional visual overlay for teams who prefer a point-and-click workflow. It calls exactly the same REST APIs documented above. Most developers integrate directly via curl, Python, or any HTTP client and can skip this section entirely.

The Ingestion Pipeline page in the Purple8 Hyper Graph web app (port 3000) reflects all DocIntel capabilities:

Step 1: Source

File dropzone — accept list is loaded dynamically from GET /ingest/formats on page load, so it always reflects what the connected DocIntel instance supports
URL mode — fetches and parses the URL through DocIntel (handles redirects, auth, and multi-page documents)
Text paste — sends raw text directly to DocIntel for entity extraction

Step 2: Extract

Async job progress bar — for files > 500 KB, the upload returns a job_id and the UI polls GET /ingest/jobs/{job_id} every 1.5 seconds; a live progress bar shows the current stage (queued → parsing → extracting → emitting → done)
Stats banner — shows chunks processed, total characters, entity count, relationship count, extraction model, and extraction duration
Fallback notice — if DocIntel was unavailable and the genai fallback was used, a yellow warning banner is shown
Entity cards — each entity shows its type badge, name (editable inline), up to 3 property key/value pairs, and a confidence percentage if < 100%
Relationship rows — each relationship shows source → type → target with a toggle for inclusion

Steps 3 & 4

Compare and Publish steps are unchanged — see the Ingest commit API for commit and publish details.

Python client usage

You can call docintel_client.py directly from your own Python code:

python

from purple8_graph.docintel_client import get_docintel_client, is_docintel_configured

if not is_docintel_configured():
    print("Set PURPLE8_DOCINTEL_URL to enable DocIntel")
else:
    client = get_docintel_client()

    # Process a file asynchronously
    import asyncio

    async def extract_from_file(path: str):
        with open(path, "rb") as f:
            data = f.read()

        job = await client.process_file(data, name=path.split("/")[-1])
        job = await client.wait_for_job(job.job_id, timeout_s=120)

        entities = client.extract_entities(job)
        rels = client.extract_relationships(job)
        print(f"Extracted {len(entities)} entities, {len(rels)} relationships")
        return client.job_to_preview(job)

    preview = asyncio.run(extract_from_file("path/to/report.pdf"))

Or use the synchronous helpers (blocks the calling thread):

python

from purple8_graph.docintel_client import get_docintel_client

client = get_docintel_client()

# From URL
preview = client.process_url_sync("https://example.com/spec.pdf")

# From file bytes
with open("model.ifc", "rb") as f:
    preview = client.process_file_sync(f.read(), name="model.ifc")

print(preview.entities)

Fallback behaviour

If PURPLE8_DOCINTEL_URL is not set or DocIntel returns an error, the Graph server falls back gracefully:

Source	Fallback behaviour
URL / text paste	Uses `genai.KnowledgeExtractor` with the configured LLM provider — works for plain prose, no structured format support
File upload	Returns HTTP 503 with a clear error message; no silent partial extraction
`/ingest/formats`	Returns a static list of 14 common formats with a `"warning"` field

The response includes "_fallback": true so the UI (and any downstream code) can surface the degraded-mode warning.

Environment variables

Variable	Default	Description
`PURPLE8_DOCINTEL_URL`	(not set)	Base URL of the DocIntel microservice, e.g. `http://localhost:8200`
`PURPLE8_DOCINTEL_KEY`	(not set)	Optional API key for DocIntel authentication

See Environment Variables for the full configuration reference.

Document Intelligence ​

How it works ​

Setup ​

1. Run Purple8 DocIntel ​

2. Configure Purple8 Hyper Graph ​

3. Verify the connection ​

REST API ​

POST /ingest/preview ​

POST /ingest/preview/file ​

GET /ingest/jobs/{job_id} ​

GET /ingest/formats ​

GET /ingest/jobs ​

Supported formats ​

IFC / BIM parsing (v0.4.1+) ​

Sketch & whiteboard parsing (v0.4.1+) ​

Job store options ​

Deployment ​

Docker Compose (recommended for teams) ​

systemd (Linux VM / bare metal) ​

Kubernetes ​

HTTPS / reverse proxy ​

Health checks ​

Ingestion Pipeline UI (optional) ​

Step 1: Source ​

Step 2: Extract ​

Steps 3 & 4 ​

Python client usage ​

Fallback behaviour ​

Environment variables ​

Document Intelligence

How it works

Setup

1. Run Purple8 DocIntel

2. Configure Purple8 Hyper Graph

3. Verify the connection

REST API

`POST /ingest/preview`

`POST /ingest/preview/file`

`GET /ingest/jobs/{job_id}`

`GET /ingest/formats`

`GET /ingest/jobs`

Supported formats

IFC / BIM parsing (v0.4.1+)

Sketch & whiteboard parsing (v0.4.1+)

Job store options

Deployment

Docker Compose (recommended for teams)

systemd (Linux VM / bare metal)

Kubernetes

HTTPS / reverse proxy

Health checks

Ingestion Pipeline UI (optional)

Step 1: Source

Step 2: Extract

Steps 3 & 4

Python client usage

Fallback behaviour

Environment variables