Skip to content

Knowledge

A workflow with a knowledge node ({type: knowledge, index: fts}) keeps one corpus per workflow (not per version: entries from v1 and v2 coexist, each with its workflow_version_id). When a run finishes successfully its DocIR is compiled to anchored markdown (the same p{n}_l{m} anchors the citation verifier uses), a YAML header with the metadata and the extracted fields (path: value [anchor]), and chunked (~400 tokens) into Postgres full-text search (ADR-013). The node itself is a marker: the interpreter treats it as a no-op.

API

All routes live under /v1/workflows/{id}/. Rows marked mcp carry the mcp OpenAPI tag, which T47 turns into agent tools; the same operations are available in-process as extractit.knowledge.tools.KnowledgeTools (list_documents, search, read, query_extractions; no side effects, no network).

Route mcp Purpose
GET knowledge status: indexed, pending, failed, stale, last_indexed_at
GET knowledge/documents mcp keyset list; filters by date, category, name, metadata
GET knowledge/documents/{entry_id}?pages=&anchors= mcp read only what is asked
GET knowledge:search?q=&k= mcp hits with entry, page, anchors, score, fragment ≤ 300 chars
GET knowledge:snapshot deterministic tar.gz (one markdown per entry + index.json), ETag = sha256
POST knowledge:reindex {scope: all\|failed} background reindex of the history / of failed entries
POST knowledge:ask mcp question with verified citations, threads
POST knowledge:query {sql} mcp restricted read-only SQL over extractions

Query syntax

q is parsed by websearch_to_tsquery: words are ANDed, "quoted phrases", -excluded, and or. Quotes or operators never raise a syntax error. Case and diacritics are ignored; amounts and invoice numbers are kept literally (INV-2026-9001, 1.210,00). If the AND query returns fewer than k hits, an OR over the terms completes the list.

Limits

  • Reading: 64 KB per call (truncated, next_anchor).
  • Search fragments: 300 characters; k ≤ 50.
  • More than 100 000 entries: search and list need a filter, otherwise 400 FILTER_REQUIRED.
  • ask: 8 steps and 60 000 input tokens by default (incomplete: true with reason), threads keep 6 turns on the server (the client's history is ignored; another workflow's thread is 404).
  • query: one SELECT over kb_documents / kb_extractions, allowed functions count sum avg min max lower upper coalesce date_trunc cast, no subqueries, CTEs, set operations, comments or catalogs; LIMIT ≤ 500; runs as role dx_knowledge_ro in a READ ONLY transaction with 2 s timeout.

Errors (RFC 9457)

KNOWLEDGE_NOT_ENABLED (409), KNOWLEDGE_NOT_READY (409), FILTER_REQUIRED (400), QUERY_NOT_ALLOWED (400). An invalid body is 400 validation-error.

Answers with citations

The answering loop uses only the read-only tools. Document text enters the prompt in delimited data blocks the model is told to treat as data. Every citation goes through the grounding verifier (verified | partial | block_only | unverified | contradicted, with normalized regions). With strict=true (default) claims without a verified or partial citation are dropped and, if none remain, abstained=true; with strict=false they are returned with supported=false.

Events

The index is refreshed by run.completed; extraction.corrected marks the entry stale and packet.deleted removes it (emitted by T32 and T50; tests use synthetic events).

Evaluation

bench/knowledge_qa/ (30 questions) and bench/reports/knowledge-qa.md: exact match 1.00 and verified citations 1.00 on literal questions (thresholds 0.8 and 0.9), with a deterministic oracle policy standing in for the model. Search: top-5 hit rate 30/30 on 100 documents, p95 45 ms over 10 000 entries, 1 000 documents × 10 pages indexed in 50 s.