Workflow interpreter
The interpreter (T26) runs a published workflow version durably on the runtime of T14: parse, extract, validate, branch and review nodes, fan-out by page and by scope, a per-node cache, routing by verdict, resume from the failed node, re-run with another version and human review inside the run. Design notes: architecture/interpreter.md.
Running a workflow
POST /v1/runs
{"workflow": "facturas@prod", "packet_id": "pkt_…", "priority": 5, "metadata": {…}}
workflow is name@alias, [email protected], a wfv_… id or {"version_id": "wfv_…"}. Multipart
(workflow, files[], metadata, priority) uploads the documents as a new packet first. The
alias is resolved once, when the run is created: the run keeps that version even if the alias
moves while it runs, waits for a review or is resumed. Errors: unknown alias → 404
ALIAS_NOT_FOUND; a draft or archived version → 422 VERSION_NOT_PUBLISHED; a bare name → 422
ALIAS_REQUIRED. The answer is 202 with the run and Location: /v1/runs/{id}.
| Route | What it does |
|---|---|
GET /v1/runs/{id} (T17) |
Run status, steps, usage; ?wait=N |
GET /v1/runs/{id}/result (T17) |
Result: outputs (extractions[node_id], validations, classifications, segments, artifacts) and reproducibility (graph_hash, plan_hash, schema_hashes, model_snapshots, engine_versions) |
GET /v1/runs/{id}/nodes |
One entry per (node, scope): status, attempts, cache key and hit, timings, usage, output references |
POST /v1/runs/{id}:cancel (T17) |
Cancels: the node in flight stops at its next check, pending nodes are skipped |
POST /v1/runs/{id}:resume (T17) |
A failed/cancelled run again, same id: only the failed node and its descendants run |
POST /v1/runs/{id}:rerun |
New run (trigger.kind=rerun, parent_run_id) with another version; unchanged nodes come from the cache |
POST /v1/runs/{id}/nodes/{node_id}:complete |
Delivers a human review to a waiting review node; a second call is a no-op |
GET /v1/runs/{id}/artifacts/{artifact_id} |
A node's file, ETag = sha256, 304 on If-None-Match |
Node statuses: pending, running, waiting (review), succeeded, failed, skipped (with
skip_reason: branch_not_taken, upstream_failed, upstream_skipped, cancelled) and
cancelled. Run statuses: succeeded when every active node finished; partially_succeeded when
only some scopes (segments, categories, files of a packet) or parallel branches failed; failed
when a node without alternative failed (parse, or the extract of a linear workflow);
waiting_review while a review waits; cancelled.
The five nodes of T26
| Node | Runs as | What it does |
|---|---|---|
parse |
child workflow, queue dx_wf_cpu |
Prepares the file (T17, conversion with T16), fans the pages out in tasks of DX_PARSE_PAGES_PER_TASK pages, at most DX_PARSE_PAGE_CONCURRENCY in flight, and merges the raw pages into one DocIR identical to a single-pass parse. Engines without page ranges (Docling today) degrade to one task per file (warning page_ranges_unsupported). |
extract |
child workflow, queue dx_wf_llm |
T06 harness + T08 finalize (T17's steps) on the DocIR of the scope (docir_slice), with the version's schema. |
validate |
in the run (inline) or dx_wf_llm/dx_wf_io with AI/external rules |
T09 rules on the extraction; stores the verdicts, says whether a blocking rule failed. |
branch |
in the run | T09 Check on the extraction: pass activates true; fail, inconclusive and a condition that cannot be evaluated activate false. |
review |
in the run | policy.when (always, on_blocking_failure, on_low_confidence, on_unclassified); when it applies the node waits (waiting, run waiting_review, event run.waiting_review) until :complete, or fails with REVIEW_TIMEOUT after policy.timeout_hours (168 h by default). The completion (decision, corrections by field path or a whole extraction) becomes the effective extraction downstream; reject fails the node (REVIEW_REJECTED). |
parse, extract and validate are cacheable. Classify and split (T28), notify (T27), report and
fill_form (T29), transform and plugins (T30), export (T31) add their handlers with the contract
below; a run whose plan has a node type without handler fails before running anything.
Handler contract
A handler implements NodeHandler and is registered for its node type:
from typing import Any
from extractit.workflows.contracts import NodeOutcome
from extractit.workflows.interpreter.executor import NodeContext, register_handler
class UppercaseInvoiceNumber:
"""Example: an extraction with the invoice number in upper case."""
node_type = "transform"
cacheable = True # same config + same inputs -> reuse a succeeded node_run
orchestrates = False # the whole run() is one durable step with retries
def run(self, ctx: NodeContext, node: Any) -> NodeOutcome:
output = ctx.load_extraction()
number = output.get("data", {}).get("invoice_number")
if number is None:
return NodeOutcome.fail("NO_INVOICE_NUMBER", "the extraction has no invoice number")
output["data"]["invoice_number"] = str(number).upper()
return NodeOutcome.ok(output_refs={"extraction": ctx.put_json(output)})
register_handler("transform", UppercaseInvoiceNumber(), replace=True)
run(ctx, node) -> NodeOutcomegets the typed node of the version (node.config) and aNodeContext:run(ids, version,graph_hash),node_id,scope(key,kind,pages,file_id),inputs(port kind → reference of the nearest upstream instance:parsed,extraction,verdicts…), loadersload_parsed()(the DocIR cut to the scope),load_extraction(),load_verdicts(),put_json()/read_json()(immutable blobs),artifacts.put(bytes | path, kind, media_type, name),usage,loggerandcancelled()/check_cancelled()(check it between pages and model calls).NodeOutcome:status(succeeded/failed),output_refs(references, never document content),activated_branches(branching nodes),scopes(fan-out: oneScopeSpec{key, kind file|segment|category, branch?, pages?}per descendant execution),artifacts,warnings,error{code, message},internal_error. An exception becomes a failed outcome (<CLASS>_ERROR) after the retries; a cancellation propagates.orchestrates = Truerunsrun()in the workflow itself, for handlers that make their own durable calls (ctx.step, page fan-out,ctx.wait_for): code between those calls must be deterministic.- Optional
cache_material(upstream, node, scope, info)adds key material (inputscontent hashes,engine_versions,model_snapshots,parse_profile).
JSON Schemas of Classification, DocumentSegment, ScopeSpec and NodeOutcome (versioned):
schemas/interpreter/ (python -m extractit.workflows.contracts --out schemas/interpreter).
Settings
| Variable | Default | Meaning |
|---|---|---|
DX_PARSE_PAGE_CONCURRENCY |
8 | Page tasks in flight per file |
DX_PARSE_PAGES_PER_TASK |
1 | Pages per task |
DX_WORKFLOW_MAX_FANOUT |
8 | Instances in flight per wave (per run: options.max_fanout) |
DX_INTERPRETER_{CPU,LLM,IO}_CONCURRENCY |
4/16/16 | Child workflows of each class per worker |
DX_INTERPRETER_PAGE_QUEUE_CONCURRENCY |
16 | Page tasks per worker |
DX_INTERPRETER_QUEUE_POLLING_SEC / DX_INTERPRETER_RESULT_POLLING_SEC |
0.1 / 0.05 | Polling of the interpreter's queues and waits |
DX_REVIEW_TIMEOUT_HOURS |
168 | Review timeout when the policy sets none |
Extension point for usage metering
interpreter.activity_closed: on_activity_closed(hook) registers hook(session, event), called
once per finished (node, scope) — succeeded, failed or reused from the cache — inside the
transaction that closes its node_run, with tenant, project, run, node, scope, status,
cache_hit, usage (model calls, tokens, cost, pages) and duration. T51 (usage, credits, budgets)
plugs in here; the interpreter knows nothing about billing.