Workflow interpreter on the durable runtime
How T26 executes a workflow version on DBOS (T14). User-facing contract and routes: workflows/interpreter.md.
Layers
POST /v1/runs ──► InterpreterRunner.create ──► RunService.submit(kind="workflow") (API process)
resolve alias once, pin version, packet (T17)
dx_run (T14, tier queue) ──► interpret(ctx) (worker)
load plan (immutable version, compile_plan of T25)
loop: core.next_ready(plan, state)
skip requests ──► node_runs skipped (+ node.skipped)
inline nodes ──► begin ► run handler in the workflow ► finish
cpu / llm / io nodes ──► begin ► child dx_wf_node on dx_wf_<class> ► finish
(bounded window of max_fanout, awaited in order)
core.apply(state, outcome)
core.run_status ► result (run_results, kind workflow) ► dx_run sets the run status
- Pure core (
interpreter/core.py):next_ready(plan, state)andapply(state, request, outcome); no clock, randomness, I/O, database, HTTP or gateway (import-linter contractinterpreter-core-is-pure). Requests come in(node_id, scope_key)order; any completion order of independent steps reaches the same final state (Hypothesis test). - Bookkeeping as checkpointed steps:
begin(node_runs→running, attempt + 1, cache lookup,node.started) andfinish(final status,node.completed/node.failed,interpreter.activity_closed) are DBOS steps of the parent; events go to T14's outbox in the same transaction as the state change, sosequencehas no gaps. - Children: one
dx_wf_node(run, node, scope, attempt)per instance on the queue of itsresource_class, with the run's priority; the child id carries the attempt. A plain handler is one T14 step (execute_step, 3 attempts with exponential backoff forcpu/llm, 1 forio); an orchestrating handler (parse, review) makes its own durable calls. - Page fan-out: the parse child starts one
dx_wf_pagesper task on queuedx_wf_pages, at mostDX_PARSE_PAGE_CONCURRENCYin flight (fanout.bounded_map), each a step whose raw pages are a content-addressed blob. Waits use DBOSwait_firstwith a short polling interval (DBOS'sget_resultpolls once a second, too slow for hundreds of 200 ms tasks).
Crash, resume, rerun
- SIGKILL: DBOS recovers the parent with the same executor id and replays its checkpoints:
begin/finishare not repeated and children re-attach by id; a finished child returns its recorded result. Only work in flight runs again (at most the window). - Resume (
:resume, T17 →RunService.resume): runs of kindworkfloware registered withset_resume_from_start, so the failed run is rewound to its first step. The interpreter re-plans;beginfinds the succeedednode_runof every finished node and reuses its outputs without starting a child; the failed node gets attempt + 1 (a new child id) and its descendants run. The run id does not change. (A rewind to the failed DBOS step would not do: the failure happened in a child, whose id DBOS would re-attach to.) - Rerun (
:rerun): a new run withparent_run_idandtrigger.kind = rerun. Not a DBOS fork — a fork reuses the original workflow arguments, the pinned version among them. The savings come from the per-node cache:cache_key = sha256(JCS{node_type, config_hash, input content hashes, scope pages, schema_hashes, model snapshot, engine_versions, parse profile}); a succeedednode_runof the project with the same key is reused (cache_hit).
Human review
The review node waits with T14's ctx.wait_for: the run records waiting_review (event
run.waiting_review) and cancels its own workflow, which frees the queue slot and the thread (a
recv would hold both for days); RunService.deliver — called by :complete and, later, T33's
inbox — sends the payload and resumes it. Nothing else is in flight while a review waits: a wave
runs its inline nodes before starting children, and children are awaited before the next wave.
Timeouts are enforced by T14's reconcile (the node fails with REVIEW_TIMEOUT).
Cancellation
RunService.cancel marks the run and cancels its workflow with its children
(cancel_children). Handlers check ctx.cancelled() between pages and model calls; the parent
marks the unfinished node_runs cancelled and the nodes never reached skipped
(skip_reason = cancelled), and T14 emits run.cancelled once.
Extension point interpreter.activity_closed
Declared for T51 (usage, credits, plans and budgets; ee/): on_activity_closed(hook); called
once per closed billable activity — a (node, scope) that finished, also when reused from the cache
— inside the transaction that closes the node_run, with run, node, scope, status, usage (model
calls, tokens, cost, pages) and duration. Exactly once also across crashes: the closing step is
checkpointed. No billing logic lives in the interpreter.
Additive changes to the runtime (T14)
dbos_app:start_child/child_result(child workflows by explicit id, fast result wait),register_plain_queue(the interpreter's class queues),EngineClient.cancel(..., cancel_children=).handlers.set_resume_from_start(kind)and its use inRunService.resume.RunService.submit(parent_run_id=)andcancelpropagating to child workflows.
Deviations from the ticket
- The parent is T14's synchronous
dx_runworkflow, not an async one: the long waits (reviews) leave the slot through T14's park mechanism, which already survives restarts. - Node artifacts reuse T17's
run_artifacts(metadatanames node and scope) instead of a second table; the migration0007_t26createsnode_runsonly. - The review wait runs in the parent (T14
wait_for), not in a child started withstart_workflow: same effect (no slot held), one wait mechanism in the codebase.