Bring Your Own Ingestion

DocIntel is optional. Use your own parser, OCR, extraction model, ETL pipeline, or structured-data export. Graph accepts the resulting data directly; the ingestion contract is not restricted to a particular extractor.

DocIntel provides an integrated parsing/extraction path. It does not make an unsupported fact true, and choosing another extractor does not prevent you from supplying document structure and source evidence.

Choose the entry point

Input you already have	Entry point	Responsibility
Entities, relationships, document text, tables	POST /ingest/commit or MCP data.commit	Map your output to the contract below. No DocIntel call is required.
Individual graph records	POST /nodes, POST /edges, or the embedded GraphEngine API	Build the desired topology yourself. Direct writes do not run the document commit workflow.
RDF 1.1 N-Quads	POST /semantic/import, MCP graph.import_nquads, or semantic_graph().import_nquads()	Supply a stable blank_node_scope, respect bounded input limits and obtain existing import/read/create permissions. No DocIntel call is required.
Your own embeddings	The top-level embedding field of the node API, or the engine's embedding argument	Match the configured index dimension and use compatible embedding models for ingestion and queries. Do not treat an arbitrary property named embedding as an indexed vector.
Raw documents you want DocIntel to extract	/ingest/preview, /ingest/preview/file, then /ingest/commit	Review the extracted content and warnings before committing.

Use the server's /docs or /openapi.json for the deployed request schema. The examples below use the REST server, not an external database.

For RDF input, follow the bounded N-Quads contract. It preserves literal lexical forms and graph membership, with explicit authorization preflight and non-atomic per-quad writes. On interruption, retry the entire original document with the same scope and tenant. This path does not run document extraction, create source-file anchors or invent RDF from ordinary node properties.

What to supply and what it enables

Field	Meaning and capability	If omitted
entities	Stable IDs, types, names, and application properties	Use an empty array for document/table-only ingestion.
relationships	Directed assertions with endpoints identified by entity ID	Use an empty array. Graph does not reconstruct missing factual relationships from similarity alone.
markdown	Full parsed document in reading order, including information not selected as entities	No whole-document chunks are produced by this commit path. Entity summaries are not a replacement for the source text.
markdown_sections	Section-start character offsets into exactly that Markdown	Non-empty Markdown is treated as one section, with long sections split as needed.
Entity spans	[start, end) mention offsets into exactly that Markdown	Graph falls back to case-insensitive entity-name matching. That is approximate evidence linkage.
tables	Original cell strings, header-row count, table index, optional page and caption	Markdown tables remain searchable text but are not automatically converted into queryable Table/Row nodes.
pages	Page numbers and character ranges within Markdown	Original page locations are not available through Page nodes.
file_ref	ID returned by POST /files/upload in the same tenant	Original bytes are not retained unless supplied separately. Structured tables/pages require a document anchor; see below.
context	Optional content-derived time, time span, geography, author, or classification confidence	Missing signals remain absent; ingestion time is not substituted for an event date.
doc_type, chunking_hint, section_boundaries, suggested_embedding_model	Advisory classification/chunking hints	The configured pipeline uses its own classification/configuration. Hints do not guarantee a particular provider or model is available.

Offsets are zero-based character positions in the submitted Markdown, not byte offsets in the original PDF or offsets in an earlier normalized string. Compute them after finalizing the text. JavaScript producers must account for Unicode code points rather than UTF-16 code units when non-BMP characters are present. Keep spans tied to source mentions even when a reviewer changes the canonical entity name.

Newly written MENTIONED_IN edges with match="span" retain the usable spans, span_coordinate_space="markdown_unicode_codepoints", and markdown_sha256 (SHA-256 of the exact submitted Markdown encoded as UTF-8). Each edge holds only the mentions whose start lies in its target chunk; a mention crossing a chunk boundary retains its full document-relative range. These are not offsets into HTML/PDF bytes, and the Markdown fingerprint is not the original-file fingerprint. Name-search fallback does not invent offsets. Existing evidence edges are not retroactively migrated.

Offsets identify locations; they are not a truth or authenticity guarantee. Keep the original and your parser output when reproducibility is required. For the narrow, reproducible document-to-native-RDF example, see the reviewed document proof.

Document anchors and originals

Upload the original through POST /files/upload, then pass its returned file_id as file_ref. Ordinary upload without a lineage intent only stores the file by default; it does not call DocIntel. For custom or retained reviewed ingestion, explicitly pass re_extract=false, including when resolving a lineage choice. Use the same tenant/authentication context for upload and commit.

The commit also supports bounded file_bytes_b64 for inline originals. Large files should use upload plus file_ref; inline limits are controlled by P8G_COMMIT_INLINE_FILE_MB.

For inline originals, Graph validates and carries forward the actual byte checksum. If those bytes identify exactly one accessible stored File, identity review uses that File's opaque ID, so repeating an unchanged reviewed commit does not require linking the same entities again. This also supports existing entities whose provenance records only the File ID. Ambiguous byte matches, stale revisions and cross-document identity conflicts still fail before writes; choose an explicit source or resolve the reported conflict rather than disabling identity review.

Logical file identity and version ancestry

New uploads receive unique opaque file_id values. Existing IDs remain valid. The SHA-256 content_hash identifies bytes, not a logical File: two deliberately independent Files can contain identical bytes without sharing identity. Always retain the returned file_id; do not reconstruct it from a digest. An unchanged, unambiguous current duplicate returns the existing File without resetting its version or overwriting its description and tags.

Each stored revision has an immutable tenant-scoped FileVersion record. Sequential predecessor and chosen base are separate relationships: creating v3 from v1 keeps v2 in history and records v2 as the predecessor, v1 as the base. Use either of these explicit choices when uploading:

Intent	Request	Result
Revise the same logical File	POST /files/upload?intent=new_version&file_id=FILE_ID&base_version=1	Next sequential version of that File, based on v1.
Keep an independent alternative	POST /files/upload?intent=linked_file&file_id=FILE_ID&base_version=1	New File with its own v1, linked to the exact source version.

Both requests use the usual multipart file body. Existing PUT /files/FILE_ID remains a same-File replacement; optional base_version selects an older base instead of the current revision. MCP data.replace_file accepts the same intents and base version. Replacement and both lineage intents automatically queue DocIntel processing by default. Set re_extract=false on REST/LCNC requests or the MCP tool to store bytes and commit your own extraction instead. The reviewed-ingestion wizard does this explicitly: choosing lineage does not re-extract its retained review. A fresh ordinary upload can opt into processing with re_extract=true; an unchanged duplicate remains a no-op, not an extraction retry.

Historical or multiple byte matches without an explicit choice return 409 with detail.reason="file_lineage_required", retryable=false, authorized conflicts and exact resubmission actions. No File/version is published for that rejected upload. Present those choices to the caller rather than guessing intent or automatically retrying. Hidden matches do not grant access to another File's identity or original.

Discover accessible history through GET /files/FILE_ID/versions and download an exact original through GET /files/FILE_ID/versions/VERSION. Superseded versions are logically archived, not moved to cold storage. Replacement also retires source-owned extraction records and their embeddings, preserving historical topology with HAD_* links instead of current HAS_* links. This does not retract independent RDF assertions or shared entities. Current collection pages and property lookups exclude archived records before applying their result windows. Vector eligibility is checked inside ANN selection; full-text results are checked against live graph records even when background postings have not caught up. Traversal does not walk through retired records, and history-mode traversal caches are separate from current caches. These rules apply to shared and dedicated tenant views and to Graph/Hybrid RAG retrieval built on those paths. Replacement durably records status="processing" and retires the old extraction before dispatching the new extraction. During processing, current content is unavailable: retrieval does not silently fall back to the old revision. Missing DocIntel configuration rejects automatic processing with 428 before publication; a provider failure after publication leaves the new revision in error, with history and originals retained. Retry the current revision with POST /files/FILE_ID/extract; retrying does not create another FileVersion. An extraction-attempt identity prevents an obsolete retry of the same bytes and version from overwriting newer output or progress.

With re_extract=false, the revision instead enters awaiting_ingestion until a source-bound custom/reviewed commit succeeds. A custom commit cannot take over a managed extraction in progress. Successful ingestion records extracted inside the ingestion transaction; this is not a readiness guarantee for every optional asynchronous index/embedding worker. Interrupted managed work is marked error on writable tenant admission after restart, not silently replayed. There is no fixed processing-time SLA or globally atomic retrieval snapshot. SDK FileVersionStore users own their processing orchestration; the ledger does not require an external DocIntel service.

FullTextEngine.load_index() restores derived postings, not graph records: reattach the index to the corresponding live/restored graph. Missing or retired records are not returned merely because an old index contains them. FullTextIndex remains available for standalone text-index usage without a graph. This does not promise a multi-resource snapshot across an in-flight retrieval, a concurrent replacement and an LLM response; applications needing a specific source revision must retain and validate its version binding.

Legacy byte history is adopted without inventing a known base: lineage_status="legacy_unrecorded"; filesystem timestamps and an unknown creator are not evidence of the original author's identity or creation time.

Document directories now preserve the validated tenant ID exactly; distinct IDs such as tenant__a and tenant_a must never share a physical namespace. Legacy originals outside their canonical tenant directory are rejected, not silently copied or reassigned. Operators must verify ownership and perform an explicit offline migration before admitting affected legacy data.

Tables and pages attach to a document node. A stored file is the recommended anchor for custom ingestion. DocIntel's direct emit can also provide a DocIntelRecord anchor, but custom extractors do not need to fabricate one. Without an anchor, the response reports structured content as skipped; it is not proof that the supplied tables/pages were stored.

The browser ingestion wizard stores an uploaded file or pasted source text when you commit, not during preview. URL extraction alone does not retain the remote original. Supply a stored original if you need original-file downloads and document-anchored tables/pages.

Source-file read visibility

File reads now apply the existing role-based row/field policies, classification ceiling and PII masking, not just the policies on semantic query output. This covers REST file inventory, metadata, version history, current/historical downloads and their LCNC counterparts. Extraction and cold archiving also check source-read access before reading or forwarding original bytes. Their existing write permission remains required in addition to source read.

A hidden File returns 404 on direct reads, including historical downloads; it is absent from file listings and visible metadata summaries. Redacting or masking the canonical blob_ref makes content reads return 403, rather than using an unfiltered locator to bypass the policy. Historical file_ref fields are filtered using that same canonical blob_ref policy. Filename and other historical metadata are field-filtered and PII-masked too.

Inventory totals and summaries describe the visible metadata within the bounded scan, before page slicing. stats.truncated signals an incomplete inventory when the scan budget is exceeded. If any visible file's size is unavailable or masked, stats.total_bytes is null, not a false zero or a partial sum presented as a complete total. A hidden stored version counter produces current_version: null; internal live-version selection still uses the authoritative stored counter. React and LCNC display unavailable metadata without fabricating sizes or offering an undefined version download.

These are source-access and metadata-filtering rules, not content redaction: hiding a metadata property does not rewrite a PDF/HTML file or find PII inside its original bytes. A caller still authorized for content may download those original bytes. Use a File row/classification rule or restrict the content locator when the original itself must not be accessible.

Example: your extractor, no DocIntel

Assume your extractor produced this Markdown:

Acme supplies Steel.

| Item | GBP |
| --- | --- |
| Steel | 1,200 |

The Markdown string below includes its final newline. Its length is 69 characters. The canonical entity name Acme Ltd deliberately differs from the source mention Acme: offsets preserve the evidence without renaming the source.

First upload your actual original. Set GRAPH_URL to your running server URL and GRAPH_AUTH_HEADER to your authorized header, for example the header name and value required by your deployment.

curl --fail-with-body "$GRAPH_URL/files/upload" \
  -H "$GRAPH_AUTH_HEADER" \
  -F "file=@invoice.md"

Save this JSON as commit.json, replacing REPLACE_WITH_UPLOADED_FILE_ID with the returned file_id:

{
  "source_name": "invoice.md",
  "file_ref": "REPLACE_WITH_UPLOADED_FILE_ID",
  "entities": [
    {
      "id": "company:acme",
      "type": "Company",
      "name": "Acme Ltd",
      "properties": {},
      "spans": [[0, 4]]
    },
    {
      "id": "material:steel",
      "type": "Material",
      "name": "Steel",
      "properties": {},
      "spans": [[14, 19]]
    }
  ],
  "relationships": [
    {
      "source": "company:acme",
      "target": "material:steel",
      "type": "SUPPLIES",
      "properties": {"amount": "1,200", "currency": "GBP"},
      "edge_dedup_key": "invoice-line-1"
    }
  ],
  "markdown": "Acme supplies Steel.\n\n| Item | GBP |\n| --- | --- |\n| Steel | 1,200 |\n",
  "markdown_sections": [0],
  "pages": [{"page_number": 1, "char_start": 0, "char_end": 69}],
  "tables": [
    {
      "page_number": 1,
      "table_index": 0,
      "header_rows": 1,
      "rows": [["Item", "GBP"], ["Steel", "1,200"]]
    }
  ],
  "upsert": true,
  "update_entities": false,
  "build_similarity_edges": false
}

curl --fail-with-body "$GRAPH_URL/ingest/commit" \
  -H "$GRAPH_AUTH_HEADER" \
  -H "Content-Type: application/json" \
  --data-binary @commit.json

No LLM is required to write this supplied content. Automatic embeddings require a configured provider; inspect the embedding statuses rather than assuming a successful write means vector retrieval is ready.

This example disables automatic similarity edges to keep supplied factual relationships distinct. The default is enabled: SIMILAR_TO edges describe embedding similarity, not verified facts such as SUPPLIES.

The same contract over MCP

Call data.commit with the JSON commit object itself as the tool arguments, not wrapped in a payload field. Its input schema is generated from the active REST CommitRequest, including entity spans, relationship deduplication keys, Markdown, tables/pages, context, inline originals, and update options. It does not invoke DocIntel. It requires an editor-capable role and permission for data.commit if the API key uses an explicit tool allowlist.

For DocIntel-assisted extraction:

1. Call data.ingest_text with text and source_name, or data.ingest_file with a public url and source_name. Local filesystem paths are not supported by the file tool. Both accept profile_hint, auto_commit, wait_seconds, and an optional previously uploaded file_ref.
2. Set auto_commit=false to review without graph writes. Completed calls return the complete preview, not an entity sample. Review warnings and inclusion flags, then pass the reviewed content to data.commit.
3. Extraction uses the asynchronous preview/job API. wait_seconds defaults to 30 and is bounded to 600. If still running, the response contains job_id and committed=false. Repeat the same tool and original arguments with that job_id to resume; do not resubmit without it.
4. With the default auto_commit=true, only successful complete previews are committed. Failed/cancelled jobs and malformed previews return errors without committing. Commit responses retain the capability outcomes and include extraction diagnostics under extraction.

These extraction tools do not automatically retain original bytes in Graph. Provide file_ref, or use data.commit with bounded file_bytes_b64, when you need an original and a document anchor. URL extraction alone is not proof that tables/pages have a stored anchor; inspect structured_content.

DocIntel agents can retrieve complete extraction evidence using docintel.get_result: page through entities, relationships, pages, merged_entities, extraction_warnings, section_boundaries, and markdown_sections with offset/limit until next_offset is null. The metadata section supplies the remaining result fields and counts. Use docintel.get_markdown for text, and docintel.get_tables with table, row_offset, and max_rows until next_row_offset is null. docintel.get_job remains a compact status/sample view, not a full export. Extraction confidence and source_chunk_ids are available for review, but are not first-class persisted Graph commit evidence fields.

Identity review and correction

An entity ID is an identity decision, not just a slug. Two documents mentioning the same name do not establish that they refer to the same person or company. Graph does not automatically rename colliding entities or search the corpus for a similar name and merge it.

Choose the contract appropriate to your producer:

• identity_policy="trust_ids" is the REST/MCP default for customer-managed IDs. Reusing an ID intentionally refers to that tenant's existing entity. You own the ID assignment and disambiguation strategy.
• identity_policy="review_conflicts" is supplied by DocIntel previews and direct emissions, and by the browser ingestion wizard when the preview omits a policy. It checks existing IDs before commit writes. Different names/types, inconsistent explicit identifiers, and cross-document name-only matches require review. A normalized name match alone is not identity proof. Normalization preserves meaningful punctuation, currency, numeric signs and code symbols.

In strict mode, properties.name or metadata.name cannot contradict the entity's top-level name. Correct that ambiguity before retrying; store aliases under distinct property keys rather than letting a property silently replace the canonical name.

For stable domain identifiers, supply an entity-level map such as "identity": {"company_registration:gb": "00123"}. Use a namespace with a well-defined uniqueness scope; preserve leading zeros. Keys and values must be nonblank strings and comparisons are exact. Claims are stored on newly created entities and checked against the existing entity's stored claims. An identifier absent from that existing record is not assumed to match. These are producer assertions, not external verification or a global identifier lookup; matching identifiers on different IDs do not automatically merge those IDs. Do not write the reserved _p8g_identity property yourself.

Strict mode also recognizes same-source provenance (file_ref, metadata file_id/sha256, or the hash of bounded inline original bytes). DocIntel forwards its original-byte hash. This permits a compatible re-extraction of the same source without repeatedly confirming every existing entity. Source references supplied by other extractors remain their assertions; a filename alone is not sufficient.

Recover from a rejected commit

An HTTP 409 with detail.reason="entity_identity_conflict" contains written=false, retryable=false, conflicts and total_conflicts. Each displayed conflict identifies the original entity index/ID, reasons, and only existing name/type information the caller may read. Identifier values and inaccessible records are not disclosed. Responses contain a bounded conflict list; resolve displayed items and retry to check any remaining conflicts.

Choose a deliberate correction:

1. Different entity: correct its incoming ID, name, type, properties or identifier claims. Update relationship source/target references to the corrected ID. The browser's entity editor remaps unambiguous direct endpoints; arbitrary strings inside properties are not rewritten.
2. Exclude it: set include=false. Relationships touching an excluded entity are skipped.
3. Intentionally the same entity: set identity_resolution="link_existing" on that incoming entity. The target must exist and be readable. This is an explicit reviewer assertion, not automated proof. It links without changing the existing entity's facts, even if the request also enables update_entities.

Resubmit the complete corrected payload through REST or MCP data.commit. Blindly retrying an unchanged identity conflict will not resolve it. Keep the source Markdown, spans, tables/pages and stored original reference. Use update_entities=true separately only for an intentional authorized edit; it does not turn a conflicting explicit identifier into verified evidence. Existing entities without identifier claims require explicit linking rather than an assumption that a newly supplied identifier matches.

DocIntel's workspace also offers a JSON correction editor before sending or retrying. It requires relationship endpoints to reference entities in the reviewed arrays, so changing an ID without fixing its references fails validation. Saved corrections are a separate overlay: the original extraction and document text remain unchanged. MCP/A2A agents can pass the overlay to docintel.retry_emit as review; use docintel.get_result(view="reviewed", section="entities") or section="relationships" to inspect it. The default view="extracted" retains the original model output. Graph previews use a saved overlay when one is present.

Upload is a separate transaction: an already uploaded original remains after a rejected commit and should be reused. DocIntel's failed-job retry copy does not retain inline original bytes; use a previously supplied file_ref, or upload the original and use Graph's complete commit contract if original-file retention is required after such a failure.

These checks do not repair historical merges or provide serializable isolation against independent direct node writes. Document commits serialize their identity check and writes through the tenant commit lock.

Preserve meaning, not just record counts

• Keep negation, conditions, exceptions, units, currency, and temporal scope in your extraction schema and properties. Graph stores your assertions; this endpoint does not verify their factual correctness.
• Use stable IDs appropriate to your domain. Identical strings are not always the same real-world entity; a name-only identity strategy can merge unrelated people or organizations.
• Keep original table strings. Typed copies are additional fields for indexed columns, not replacements for the original cells. Ambiguous headers can yield an unstructured table without queryable rows.
• Supply context only when supported by the content. Its fields are claims, distinct from engine-owned tenant, actor, and ingestion-time metadata.
• Extraction confidence and source_chunk_ids survive the DocIntel preview for review, but are not first-class persisted evidence fields in the commit model. Graph chunks may differ from extraction windows. Use Markdown spans for entity evidence links; put application-specific relationship evidence in relationship properties if you need it persisted.
• Preview warnings and merge diagnostics are review information, not a durable extraction-quality report. Retain the original extraction output separately if your audit process requires it.

Verify the result

Do not use HTTP success or entity counts alone as a retrieval-quality check.

1. Inspect original_file: stored/existing includes the document ID; not_provided or skipped does not establish that originals were retained.
2. Inspect document_chunks: chunk count, mentioned_in_edges, embedding status, and mentions_by_span versus mentions_by_name. With the example above, one chunk and two span-based entity mentions should be produced.
3. Inspect structured_content: tables, rows, pages, and unstructured_tables. The example should produce one table, one data row, and one page.
4. Read the entities and relationships back. Check important identifiers, amounts, units, and conditions against the source.
5. Discover the table ID from stored Table nodes, then use GET /tables/{table_id} and POST /tables/{table_id}/rows/query to verify original cells. Download originals through GET /files/{file_id}/download.
6. Run representative retrieval questions and inspect supporting chunks. Compare expected facts, not just whether an answer was generated.

entity_embed_status or chunk embed_status equal to not_configured means the supplied data was stored without automatic embeddings. Resolve provider configuration before expecting vector-backed retrieval.

Retries, updates, and lifecycle

Interrupted ingestion: review, confirm, retry

Graph owns graph identity and de-duplication; DocIntel remains the extractor. Retain the reviewed payload and original file_ref. After an interruption or lost response, explain that the outcome may be unknown and ask the user to confirm the retry. Re-submit the same payload rather than automatically starting extraction again. New entities are created; existing entity IDs are linked without overwriting their facts unless an authorized edit was explicitly requested. Stable assertion keys prevent duplicate relationships.

The ingestion wizard retains the reviewed preview and uploaded original after a failed commit. It does not retry automatically: the user presses Commit again. A failed comparison is shown as an error, not as an empty successful comparison; the user must retry comparison before reaching the commit action. Identity conflicts still require correction, exclusion, or explicit confirmation of link_existing. MCP ingestion tools instruct agents to obtain the same retry confirmation and reuse data.commit with the retained review.

This confirmation is a caller/UI interaction, not a new server-side approval token or a claim that Graph can verify a human is present. REST, SDK and custom integrations must provide the appropriate confirmation experience themselves. Graph continues to enforce authentication, authorization and identity checks.

Identical retry means identical stable IDs and source references, not merely similar text or matching names. Re-extraction may generate different IDs or facts; review those changes rather than assuming automatic semantic de-duplication. If extraction never produced a reviewable payload, obtain and review a new preview before confirming ingestion.

The POSIX tests/e2e/test_ingest_retry_socket.py proof crashes an actual Graph HTTP service after document writes, both before durable finalization and after finalization but before any response reaches the caller. Shared and dedicated tenants reopen the original store and explicitly retry the retained payload. Checks cover complete rollback versus retained committed work, non-duplicated entities/chunks/pages/relationships and claims, unchanged pre-existing entity facts, byte-identical original download, quote evidence, RDF/LPG output, BM25 and tenant isolation. This uses a recorded reviewed extraction, not a new DocIntel recovery workflow or live inference. It does not certify arbitrary instruction-level crashes, power loss or distributed exactly-once delivery.

Source changes invalidate a retained review

Keep the preview's metadata.sha256: it identifies the original bytes, not Markdown, extracted text or the stable File ID. A File retains its ID across replacement, but its content_hash and version change. For custom extraction, calculate SHA-256 from the bytes actually extracted and supply it in metadata.sha256. Optionally pin the captured positive integer version with metadata.file_version; an initial File without an explicit version is version

1. The version pin also detects replacement with identical bytes.

file_ref, top-level file_id and metadata.file_id must name the same stored File when supplied together. Preserve the binding during review and retry:

{
  "file_ref": "stored-file-id",
  "metadata": {
    "sha256": "<64 hexadecimal digits calculated from the original bytes>",
    "file_version": 2
  }
}

This is the source-binding portion of a commit; include the normal reviewed entities, relationships, Markdown and layout fields as appropriate. Inline originals are also checked against their decoded bytes and any existing File with that original content ID.

A changed or missing bound source returns HTTP 409 with reason=source_version_conflict and retryable=false before writes. Malformed binding values or disagreeing file references return 422. Do not change the old review's checksum/version to bypass a conflict, automatically replay it, or automatically start extraction. Obtain extraction of the current source, review it, then obtain confirmation for that new commit. Historical assertions and stored versions remain intact; this is not a historical-ingest mode or automatic RDF retraction.

The source is checked before preparation and again inside the existing tenant-ingestion lock before journal creation. File upload, replacement, deletion, archive/metadata updates and extraction-status publication use that same in-process tenant lock. Graph's stored-file extraction retains preview metadata and binds its commit to the bytes and version actually submitted. A delayed old job cannot replace newer content or overwrite its progress. This is not serialization against arbitrary raw SDK writes, other processes or external filesystem modification.

REST and MCP share the commit check. Standalone SDK IngestOrchestrator applies the same binding validation before its own chunk/link writes; it does not roll back entity writes performed earlier by its caller. Legacy unpinned payloads remain accepted for compatibility, but cannot prove that a review describes the current bytes: use a checksum and, where needed, a version pin for this guarantee.

Strict chunk identity now distinguishes File versions even when Markdown is unchanged. Version-1 and unversioned ID generation is retained. A confirmed commit on an older installation's version-2-or-later content can replace its old content-only chunk IDs; follow the File's current HAS_CHUNK edges rather than caching generation-dependent chunk IDs. File/entity/RDF assertion IDs are not changed by this correction.

Existing write and recovery contract

For migration or disaster recovery, use the existing portable source-evidence backup. Graph-only record export is not an original-file backup. The portable archive includes referenced originals and stored versions and remaps storage references into the target store; its restore checks do not depend on the source machine remaining available.

• An unresolved included relationship endpoint is rejected before writes. Endpoints may refer to included entities or existing tenant nodes. Relationships touching explicitly excluded entities are skipped.
• Existing entity IDs are linked without overwriting their facts by default. Set update_entities=true only for an intentional, authorized property edit. upsert=true controls deterministic relationship IDs; it does not request entity edits.
• Use edge_dedup_key when multiple assertions of the same type connect the same endpoints. Choose stable assertion keys; unrelated documents sharing a key do not automatically gain separate provenance records.
• Inspect commit failure details, including stage, retryability, and rollback outcome. commit_outcome distinguishes not_committed, unconfirmed and committed; a lost response or failed finalization is not proof of rollback. Known storage failures and the disk high-watermark write admission gate (P8G_WRITE_ADMISSION_DISK_MAX_PCT=95.0) return 507 before the underlying volume hits ENOSPC. For unconfirmed or incomplete undo, repair storage and restart before retrying the identical payload. New tenant-engine admission returns 503 with reason=commit_recovery_required until recovery can finish; this also applies to cached engines. Existing trusted references are not revoked. Do not blindly retry an incomplete rollback.
• A terminal journal record, durable rename to .complete, and directory sync confirm completion. Journal removal is only cleanup. A committed response with journal_cleanup="pending" still means data is committed: check the storage warning rather than treating it as rolled back. The UI displays this warning; MCP preserves it in the returned result.
• Directory synchronization must be supported by the filesystem/platform. Sync failures are explicit, not silently downgraded. Complete pending recovery and stop cleanly before downgrading: old readers may ignore terminal records in unfinished .jsonl files. See the completion and I/O contract.
• Tenant creation and reopening must finish interrupted-commit recovery before publishing the engine, including shared and sharded admission paths. Failed undo, complete malformed journal records, and pending recovery on read-only storage block access instead of silently serving partial content. The journal is retained for recovery; legacy .failed journals are retried too.
• Only an unterminated final journal record is treated as a torn tail. A newline-terminated invalid record requires operator investigation. Stop the affected service, preserve the journal and storage for diagnosis, and repair the underlying fault or restore a consistent backup before reopening. Do not delete recovery journals just to make startup succeed.
• A lost response does not prove rollback. Reuse the same entity IDs, source name, file_ref, Markdown, structure, and assertion keys when confirming the outcome. Distinct assertion keys produce distinct relationships. Local regression checks cover full REST/MCP replay and abrupt process exit before mutation, mid-mutation, and after journal completion for dedicated and shared storage; this is not a power-loss or full sharded-workflow certification.
• Document-anchored commits write deterministic MENTIONS claim edges for newly created as well as existing entities. Source claims are present on the first commit, rather than appearing only when the same payload is retried.
• System created_at/updated_at persist through storage reads, checkpoint/ WAL replay and exact-snapshot recovery. Existing records retain creation time; ordinary updates advance update time only when stored content changes. Identical ingestion retries do not rewrite unchanged structured nodes merely because they carry system or tenant metadata. A create-or-replace call may still change ingestion context. Do not treat system times as authoritative source/event dates; supply those explicitly as properties. Previously lost timestamp history is not reconstructed. See the system-timestamp contract.
• For homogeneous SDK ingestion, add_nodes_batch() acknowledges only new IDs and now persists both graph records and supplied default vectors before native mutation. Existing IDs are skipped without changing facts, timestamps, ownership or vectors; the first repeated new ID wins. A full retry returns zero inserted records. Use update/upsert for replacement. Recovery no longer requires a clean close or an explicit checkpoint. An exception can still leave recoverable data: stop, verify recovery and replay the original payload rather than inventing new IDs. A damaged WAL blocks new writes/checkpoints until repaired or restored. Recovery adds storage/startup work, and this hot-tier path is not a newly qualified bounded-RSS ingest path. See the batch acknowledgement and recovery contract for tested boundaries, read-only recovery and resource costs.
• Do not transfer that SDK retry contract to POST /nodes/bulk: existing IDs produce a conflict, not a successful zero-insert result. Verify stored records against the original payload before deciding whether a lost response needs another write. Single-node POST /nodes now resolves the same native tenant schema used by SDK and MCP writes, including shared isolation. The real-HTTP restart proof uses single-node creation in both modes; the previous unscoped-schema blocker is closed without exposing the base engine's schema. This is not complete REST-surface qualification; see the HTTP proof and resource limits.
• Batch-created entities, chunks and structured records now publish to the integrated BM25 queue after storage succeeds. BM25 indexes name, title, description, body, text, bio and content, not every custom property. Visibility is asynchronous: a successful write is not proof that a keyword query already used BM25, and the text-search API can fall back to a bounded scan. Check its reported method. Queue overflow and indexing failures are logged. On writable reopening with BM25 enabled, the engine rebuilds the index from durable graph records before readiness, repairing missing background work and stale snapshots. This also adds a content-sized scan to clean restarts; disabled/read-only engines do not rebuild BM25. See BM25 publication boundaries.
• Upload and commit are separate operations. If upload succeeds but commit fails, the original remains stored and can be reused on retry. Do not upload another copy solely because extraction/commit failed.
• Re-extraction may retire prior chunks and structured content for the document anchor. Use file versioning deliberately; a fresh commit is not an implicit audit archive of every prior extraction.
• File deletion, extraction replacement, and removal of shared entities are distinct lifecycle operations. Do not assume deleting an original removes all derived or shared graph records.

See DocIntel integration for the integrated extraction path and Unified Data Store for the storage model.