Durable runtime: runs, queues, cancellation, resume and human review
Server-side execution of runs (T14) on DBOS Transact (MIT, pinned dbos==3.4.0) over the
application Postgres: no new service. The hypothesis H1 (the DBOS API does what this design needs)
was checked first with ten executable checks against the real API — see
docs/adr/h1-dbos-checklist.md, regenerated by pytest -m h1.
The local library and the extractit CLI never import the runtime (.importlinter).
| Piece | Module |
|---|---|
Settings and RuntimeLimits (injectable provider) |
extractit.runtime.config |
| The only module that imports DBOS | extractit.runtime.dbos_app |
| Run model, state machine, tables | extractit.runtime.models |
RunService (submit, cancel, resume, fork, deliver, timeline…) |
extractit.runtime.service |
Handler registry and RunContext |
extractit.runtime.handlers |
Generic workflow dx_run, human-review wait |
extractit.runtime.workflows |
Measured steps, @durable_step |
extractit.runtime.steps |
| Tier queues and priorities | extractit.runtime.queues |
| Cooperative cancellation | extractit.runtime.cancel |
Transactional outbox, transitions, OutboxReader |
extractit.runtime.outbox |
reconcile_orphan_runs, purge_run_events |
extractit.runtime.schedule |
| Error taxonomy | extractit.runtime.errors |
| Spans, histograms, log context | extractit.runtime.telemetry |
| Worker process | python -m extractit.runtime.worker (T46: extractit worker) |
Database layout and roles
DBOS keeps its system tables (workflow status, inputs, step outputs, queues, notifications,
schedules) in schema dbos of the application database, owned by role dx_dbos
(NOBYPASSRLS, not superuser; created by deploy/compose/postgres/init-roles.sql, password
DX_DBOS_DB_PASSWORD). DBOS migrates that schema itself on launch; Alembic never touches it and the
RLS audit of T11 lists dbos as its explicit exempt schema: it holds no tenant content, only
ids, hashes and small JSON (see Small ids only).
The runtime's own tables live in public, under RLS by tenant and project (migration 0003_t14):
| Table | Holds |
|---|---|
runs |
id (= DBOS workflow id), tenant, project, kind, status, priority 0–9, idempotency key, input_ref/output_ref/error (small JSON), parent_run_id, batch_id, queue_name, event_seq, cancel_requested_at, wait fields, timestamps, cost_usd, tokens, pages |
run_steps |
one row per step (key <function id>.<name>): status, attempt, start/end, duration_ms, cost, tokens, error_class, error_message, cache_hit, output_ref, dbos_function_id |
run_events |
the outbox: (run_id, seq) 1..n without gaps, type, payload |
batches |
a lot of N runs: total, priority, open/completed |
Workers write through tenant_session (role dx_app): no worker uses dx_system (a test
enforces it). The cross-tenant reads of the maintenance jobs use dx_dbos with two narrow policies:
runtime_dispatch (SELECT of the routing columns of active runs) and runtime_purge_*
(SELECT/DELETE of outbox rows older than a day). dx_dbos cannot read output_ref, error,
run_steps, batches or any T11 table.
Connections: DX_DATABASE_URL (dx_app) and DX_DBOS_DATABASE_URL (dx_dbos, same database).
Life cycle of a run
submit handler returns
┌──────► queued ──► running ─────────────────────────► succeeded | partially_succeeded
│ │ │ ▲ └─► failed (step error)
│ resume │ cancel │ │ delivery / deadline
│ ▼ ▼ │
│ cancelled ◄── waiting_review cancel ──► cancelled
└──── failed | cancelled
The full table is TRANSITIONS in models.py; outbox.transition() refuses anything else.
- Submit.
RunService.submitinserts the run (queued) and itsrun.queuedevent in one transaction, commits, then enqueuesdx_runwithrun_idas the DBOS workflow id, the DBOS priority10 - pand the project id as partition key. If the process dies between the commit and the enqueue,reconcile_orphan_runsenqueues it later (enqueueing twice is harmless: the workflow id is the run id).submit_manychecksmax_queued_runswith one grouped query before inserting anything (QuotaExceededwithretry_after), inserts the batch and its N runs, and enqueues them in one transaction of the DBOS system database (1 000 runs in < 5 s). - Start.
dx_run(run_id, tenant_id, project_id, kind, input_ref, traceparent)moves the runqueued → running(a plain idempotent transaction, so that resume, rewind and fork still go through the states) and calls the handler registered forkind. - Steps. Every side effect is a step (
ctx.step(...)or@durable_step): a DBOS step whose body also writes therun_stepsrow and the run aggregates. A recovered or resumed run replays checkpointed steps without executing them again. - End. The workflow writes
succeeded/partially_succeeded(withoutput_ref) orfailed(witherror = {class, message, step}). A batch whose last active run ends is closed and emitsbatch.completedexactly once (the batch row is lockedFOR NO KEY UPDATE).
Queues, priority and per-project limits
DBOS partitioned queues give one partition_concurrency per queue (H1, C3), so each per-project
limit n maps to a tier queue dx_runs_c<n>, partitioned by project id; projects with the same
limit share the queue and are visited in random order every poll, so a small project is not starved
by a big backlog (WF-10: project B finishes within 3× its isolated time while A has 200 queued).
global_concurrency and worker_concurrency (DX_RUNTIME_GLOBAL_CONCURRENCY,
DX_RUNTIME_WORKER_CONCURRENCY) apply per tier queue. Priorities are 0–9 (9 highest, default 5).
RuntimeLimits(max_concurrent_runs, max_queued_runs) come from a LimitsProvider. The default,
EnvLimitsProvider, reads DX_MAX_CONCURRENT_RUNS (10) and DX_MAX_QUEUED_RUNS (10 000); T17
plugs project_limits of T12 in its place.
Cancellation
RunService.cancel stamps cancel_requested_at and cancels the workflow. A queued run never
starts (the service writes cancelled); a running one stops at its next step boundary, or sooner
when the handler calls ctx.check_cancelled() between sub-steps (≤ 5 s in the tests). The single
run.cancelled event is written by whoever gets there first through a conditional transition; a
second cancel returns the same state. A cancel that lands while a step is in flight surfaces in DBOS
3.4 as an ownership conflict: the step finishes, its output cannot be checkpointed, and a later
resume runs that one step again.
Resume and fork
resume(onlyfailedorcancelled; anything else is aRunConflict, HTTP 409): a cancelled workflow is resumed withresume_workflow; a failed one is rewound withrewind_workflowfrom thedbos_function_idof the failed step (H1, C5: DBOS does not resumeERRORworkflows). Steps before it keep their single execution.fork(run_id, from_step)creates a child run (parent_run_id) and callsfork_workflowfrom that step's function id with the child id as workflow id: the child reuses the outputs of the earlier steps (copied into itsrun_stepsascache_hit) and re-executes fromfrom_step.dx_runtakes its effective run id from the workflow id, because a fork copies the parent's inputs.
Human review (ctx.wait_for)
A parked recv would keep its executor thread and queue slot (H1, C7). wait_for(topic,
timeout_s) therefore:
- probes
recv("review:<topic>", 0); if the payload is there, returns it; - checks the deadline (a checkpointed clock); if passed, returns
None; - records
waiting_review(a step), probes once more, and then cancels its own workflow, which frees the slot (onlyPENDINGworkflows count). That self-cancel is not checkpointed by DBOS, so it is guarded by the run row: on replay it is skipped.
RunService.deliver(run_id, topic, payload) sends the payload, stamps wait_released_at and
resumes the workflow; the replay finds the message (≤ 2 s in the tests, also after a worker
restart). reconcile_orphan_runs resumes runs whose deadline passed (the handler gets None) and
any parked run whose resume was lost in a race.
Outbox (DEV-02)
Each transition locks the run, updates it, reserves the next seq with
UPDATE runs SET event_seq = event_seq + 1 … RETURNING, inserts the run_events row and calls
pg_notify('dx_run_events', run_id) in the same transaction: a rolled-back transition leaves
no event and no notification. OutboxReader.poll(run_id, after_seq, limit) serves T27. LISTEN
needs a direct connection (not PgBouncer in transaction mode).
Scheduled maintenance
DBOS cron schedules (one execution per tick with several workers, H1 C8):
dx_reconcile_orphan_runsevery 30 s (DX_RUNTIME_RECONCILE_GRACE_SEC, default 15 s, keeps it away from runs that just changed): enqueues orphans (≤ 45 s), finishes runs whose workflow was cancelled or failed while the worker was dead, restarts resumes interrupted between commit and DBOS call, and resumes expired or lost review waits;dx_purge_run_eventsdaily: deletes outbox rows older thanDX_RUN_EVENTS_RETENTION_DAYS(30).
Registering a handler and a step
from extractit.runtime.handlers import HandlerResult, RunContext, register_handler
from extractit.runtime.steps import durable_step
@durable_step("parse", retries=3, backoff=2.0)
def parse(ctx: RunContext, blob_id: str) -> dict[str, str]:
... # read the blob through ctx.session(), do the work
ctx.add_usage(cost_usd=0.002, tokens_in=1200, tokens_out=80, pages=3)
return {"docir_blob_id": "...", "sha256": "..."} # ids and hashes only
@register_handler("extract")
def extract(ctx: RunContext) -> HandlerResult:
parsed = parse(ctx, ctx.input_ref["blob_id"])
ctx.check_cancelled()
review = ctx.wait_for("fields", timeout_s=3600) # None on timeout
return HandlerResult(output_ref={"result_id": "..."}, partial=review is None)
Workers import handler modules from DX_RUNTIME_HANDLERS (comma-separated) and from the
extractit.run_handlers entry points. Handler code between steps must be deterministic; side
effects go through steps.
Errors: raise InputError, ValidationFailed (never retried), ParseError, ModelError
(retried), or NonRetryableError(message, error_class=...); anything else is classified by
errors.classify (input|parse|model|validation|internal). DBOS records only a sanitized
StepFailed (class and step name).
Small ids only
Workflow inputs, step outputs and messages are stored by DBOS without RLS or per-tenant encryption. The runtime therefore uses DBOS' portable JSON serializer (no pickle) and passes only ids, hashes and small JSON:
- a step output, a run
input_ref/output_refand a review payload above 64 KiB raiseStepPayloadTooLarge— store the content as a blob and pass its id; - failures recorded by DBOS carry the error class and step, never the message;
- spans and metrics carry ids, names, classes and numbers.
A test runs a handler over a sentinel "document" and checks that the sentinel appears in no log, no
span and no table of schema dbos.
Observability
structlog context variables run_id and step; spans dx.run (parented on the traceparent
propagated from the request) and dx.step.<name> with run_id and tenant_id; histograms
dx_run_duration_seconds and dx_step_duration_seconds and the up-down counter dx_queue_depth
through the OpenTelemetry API. The worker creates OTLP exporters only when
OTEL_EXPORTER_OTLP_ENDPOINT is set (T53 ships the exporter package).
Operations
docker compose -f deploy/compose/docker-compose.dev.yml -p dx-dev up -d --wait
DX_MIGRATOR_DATABASE_URL=postgresql://dx_migrator:[email protected]:55432/extractit \
uv run alembic upgrade head
DX_DATABASE_URL=postgresql://dx_app:[email protected]:55432/extractit \
DX_DBOS_DATABASE_URL=postgresql://dx_dbos:[email protected]:55432/extractit \
DX_RUNTIME_HANDLERS=my_handlers \
uv run python -m extractit.runtime.worker
Restart a worker with the same DX_RUNTIME_EXECUTOR_ID to recover the runs it left in flight
(H1, C9; a kill -9 repeats at most the step in flight). Run pytest -m h1 after any DBOS
upgrade; the version is pinned exactly in pyproject.toml. Tests:
pytest tests/runtime (Postgres 16 via DX_TEST_DATABASE_URL or Docker).
Plan B (not activated)
If a critical H1 check (C2, C3, C5, C6 or C7) ever fails on a DBOS upgrade without an alternative in
its API, replace dbos_app.py only (estimate +1–2 d):
- Queue: a
run_queuetable polled withSELECT … FOR UPDATE SKIP LOCKED, ordered by priority andqueued_at; - Per-project concurrency: a semaphore table (
project_id,running) updated in the claim transaction, checked againstRuntimeLimits.max_concurrent_runs; - Steps, resume and fork: a step-output log (
run_step_outputs(run_id, function_id, output)) replayed by the workflow driver; resume and fork re-run the handler and skip logged steps; - Human review: the run leaves its slot and is re-queued by
deliver; - Schedules: a leader lock (
pg_try_advisory_lock) in the worker.
RunService, handlers, steps, the outbox and the tables stay as they are.