Skip to content

Durable runtime: runs, queues, cancellation, resume and human review

Server-side execution of runs (T14) on DBOS Transact (MIT, pinned dbos==3.4.0) over the application Postgres: no new service. The hypothesis H1 (the DBOS API does what this design needs) was checked first with ten executable checks against the real API — see docs/adr/h1-dbos-checklist.md, regenerated by pytest -m h1. The local library and the extractit CLI never import the runtime (.importlinter).

Piece Module
Settings and RuntimeLimits (injectable provider) extractit.runtime.config
The only module that imports DBOS extractit.runtime.dbos_app
Run model, state machine, tables extractit.runtime.models
RunService (submit, cancel, resume, fork, deliver, timeline…) extractit.runtime.service
Handler registry and RunContext extractit.runtime.handlers
Generic workflow dx_run, human-review wait extractit.runtime.workflows
Measured steps, @durable_step extractit.runtime.steps
Tier queues and priorities extractit.runtime.queues
Cooperative cancellation extractit.runtime.cancel
Transactional outbox, transitions, OutboxReader extractit.runtime.outbox
reconcile_orphan_runs, purge_run_events extractit.runtime.schedule
Error taxonomy extractit.runtime.errors
Spans, histograms, log context extractit.runtime.telemetry
Worker process python -m extractit.runtime.worker (T46: extractit worker)

Database layout and roles

DBOS keeps its system tables (workflow status, inputs, step outputs, queues, notifications, schedules) in schema dbos of the application database, owned by role dx_dbos (NOBYPASSRLS, not superuser; created by deploy/compose/postgres/init-roles.sql, password DX_DBOS_DB_PASSWORD). DBOS migrates that schema itself on launch; Alembic never touches it and the RLS audit of T11 lists dbos as its explicit exempt schema: it holds no tenant content, only ids, hashes and small JSON (see Small ids only).

The runtime's own tables live in public, under RLS by tenant and project (migration 0003_t14):

Table Holds
runs id (= DBOS workflow id), tenant, project, kind, status, priority 0–9, idempotency key, input_ref/output_ref/error (small JSON), parent_run_id, batch_id, queue_name, event_seq, cancel_requested_at, wait fields, timestamps, cost_usd, tokens, pages
run_steps one row per step (key <function id>.<name>): status, attempt, start/end, duration_ms, cost, tokens, error_class, error_message, cache_hit, output_ref, dbos_function_id
run_events the outbox: (run_id, seq) 1..n without gaps, type, payload
batches a lot of N runs: total, priority, open/completed

Workers write through tenant_session (role dx_app): no worker uses dx_system (a test enforces it). The cross-tenant reads of the maintenance jobs use dx_dbos with two narrow policies: runtime_dispatch (SELECT of the routing columns of active runs) and runtime_purge_* (SELECT/DELETE of outbox rows older than a day). dx_dbos cannot read output_ref, error, run_steps, batches or any T11 table.

Connections: DX_DATABASE_URL (dx_app) and DX_DBOS_DATABASE_URL (dx_dbos, same database).

Life cycle of a run

           submit                      handler returns
  ┌──────► queued ──► running ─────────────────────────► succeeded | partially_succeeded
  │          │          │  ▲                          └─► failed   (step error)
  │ resume   │ cancel   │  │ delivery / deadline
  │          ▼          ▼  │
  │      cancelled ◄── waiting_review          cancel ──► cancelled
  └──── failed | cancelled

The full table is TRANSITIONS in models.py; outbox.transition() refuses anything else.

  1. Submit. RunService.submit inserts the run (queued) and its run.queued event in one transaction, commits, then enqueues dx_run with run_id as the DBOS workflow id, the DBOS priority 10 - p and the project id as partition key. If the process dies between the commit and the enqueue, reconcile_orphan_runs enqueues it later (enqueueing twice is harmless: the workflow id is the run id). submit_many checks max_queued_runs with one grouped query before inserting anything (QuotaExceeded with retry_after), inserts the batch and its N runs, and enqueues them in one transaction of the DBOS system database (1 000 runs in < 5 s).
  2. Start. dx_run(run_id, tenant_id, project_id, kind, input_ref, traceparent) moves the run queued → running (a plain idempotent transaction, so that resume, rewind and fork still go through the states) and calls the handler registered for kind.
  3. Steps. Every side effect is a step (ctx.step(...) or @durable_step): a DBOS step whose body also writes the run_steps row and the run aggregates. A recovered or resumed run replays checkpointed steps without executing them again.
  4. End. The workflow writes succeeded/partially_succeeded (with output_ref) or failed (with error = {class, message, step}). A batch whose last active run ends is closed and emits batch.completed exactly once (the batch row is locked FOR NO KEY UPDATE).

Queues, priority and per-project limits

DBOS partitioned queues give one partition_concurrency per queue (H1, C3), so each per-project limit n maps to a tier queue dx_runs_c<n>, partitioned by project id; projects with the same limit share the queue and are visited in random order every poll, so a small project is not starved by a big backlog (WF-10: project B finishes within 3× its isolated time while A has 200 queued). global_concurrency and worker_concurrency (DX_RUNTIME_GLOBAL_CONCURRENCY, DX_RUNTIME_WORKER_CONCURRENCY) apply per tier queue. Priorities are 0–9 (9 highest, default 5).

RuntimeLimits(max_concurrent_runs, max_queued_runs) come from a LimitsProvider. The default, EnvLimitsProvider, reads DX_MAX_CONCURRENT_RUNS (10) and DX_MAX_QUEUED_RUNS (10 000); T17 plugs project_limits of T12 in its place.

Cancellation

RunService.cancel stamps cancel_requested_at and cancels the workflow. A queued run never starts (the service writes cancelled); a running one stops at its next step boundary, or sooner when the handler calls ctx.check_cancelled() between sub-steps (≤ 5 s in the tests). The single run.cancelled event is written by whoever gets there first through a conditional transition; a second cancel returns the same state. A cancel that lands while a step is in flight surfaces in DBOS 3.4 as an ownership conflict: the step finishes, its output cannot be checkpointed, and a later resume runs that one step again.

Resume and fork

  • resume (only failed or cancelled; anything else is a RunConflict, HTTP 409): a cancelled workflow is resumed with resume_workflow; a failed one is rewound with rewind_workflow from the dbos_function_id of the failed step (H1, C5: DBOS does not resume ERROR workflows). Steps before it keep their single execution.
  • fork(run_id, from_step) creates a child run (parent_run_id) and calls fork_workflow from that step's function id with the child id as workflow id: the child reuses the outputs of the earlier steps (copied into its run_steps as cache_hit) and re-executes from from_step. dx_run takes its effective run id from the workflow id, because a fork copies the parent's inputs.

Human review (ctx.wait_for)

A parked recv would keep its executor thread and queue slot (H1, C7). wait_for(topic, timeout_s) therefore:

  1. probes recv("review:<topic>", 0); if the payload is there, returns it;
  2. checks the deadline (a checkpointed clock); if passed, returns None;
  3. records waiting_review (a step), probes once more, and then cancels its own workflow, which frees the slot (only PENDING workflows count). That self-cancel is not checkpointed by DBOS, so it is guarded by the run row: on replay it is skipped.

RunService.deliver(run_id, topic, payload) sends the payload, stamps wait_released_at and resumes the workflow; the replay finds the message (≤ 2 s in the tests, also after a worker restart). reconcile_orphan_runs resumes runs whose deadline passed (the handler gets None) and any parked run whose resume was lost in a race.

Outbox (DEV-02)

Each transition locks the run, updates it, reserves the next seq with UPDATE runs SET event_seq = event_seq + 1 … RETURNING, inserts the run_events row and calls pg_notify('dx_run_events', run_id) in the same transaction: a rolled-back transition leaves no event and no notification. OutboxReader.poll(run_id, after_seq, limit) serves T27. LISTEN needs a direct connection (not PgBouncer in transaction mode).

Scheduled maintenance

DBOS cron schedules (one execution per tick with several workers, H1 C8):

  • dx_reconcile_orphan_runs every 30 s (DX_RUNTIME_RECONCILE_GRACE_SEC, default 15 s, keeps it away from runs that just changed): enqueues orphans (≤ 45 s), finishes runs whose workflow was cancelled or failed while the worker was dead, restarts resumes interrupted between commit and DBOS call, and resumes expired or lost review waits;
  • dx_purge_run_events daily: deletes outbox rows older than DX_RUN_EVENTS_RETENTION_DAYS (30).

Registering a handler and a step

from extractit.runtime.handlers import HandlerResult, RunContext, register_handler
from extractit.runtime.steps import durable_step


@durable_step("parse", retries=3, backoff=2.0)
def parse(ctx: RunContext, blob_id: str) -> dict[str, str]:
    ...  # read the blob through ctx.session(), do the work
    ctx.add_usage(cost_usd=0.002, tokens_in=1200, tokens_out=80, pages=3)
    return {"docir_blob_id": "...", "sha256": "..."}  # ids and hashes only


@register_handler("extract")
def extract(ctx: RunContext) -> HandlerResult:
    parsed = parse(ctx, ctx.input_ref["blob_id"])
    ctx.check_cancelled()
    review = ctx.wait_for("fields", timeout_s=3600)  # None on timeout
    return HandlerResult(output_ref={"result_id": "..."}, partial=review is None)

Workers import handler modules from DX_RUNTIME_HANDLERS (comma-separated) and from the extractit.run_handlers entry points. Handler code between steps must be deterministic; side effects go through steps.

Errors: raise InputError, ValidationFailed (never retried), ParseError, ModelError (retried), or NonRetryableError(message, error_class=...); anything else is classified by errors.classify (input|parse|model|validation|internal). DBOS records only a sanitized StepFailed (class and step name).

Small ids only

Workflow inputs, step outputs and messages are stored by DBOS without RLS or per-tenant encryption. The runtime therefore uses DBOS' portable JSON serializer (no pickle) and passes only ids, hashes and small JSON:

  • a step output, a run input_ref/output_ref and a review payload above 64 KiB raise StepPayloadTooLarge — store the content as a blob and pass its id;
  • failures recorded by DBOS carry the error class and step, never the message;
  • spans and metrics carry ids, names, classes and numbers.

A test runs a handler over a sentinel "document" and checks that the sentinel appears in no log, no span and no table of schema dbos.

Observability

structlog context variables run_id and step; spans dx.run (parented on the traceparent propagated from the request) and dx.step.<name> with run_id and tenant_id; histograms dx_run_duration_seconds and dx_step_duration_seconds and the up-down counter dx_queue_depth through the OpenTelemetry API. The worker creates OTLP exporters only when OTEL_EXPORTER_OTLP_ENDPOINT is set (T53 ships the exporter package).

Operations

docker compose -f deploy/compose/docker-compose.dev.yml -p dx-dev up -d --wait
DX_MIGRATOR_DATABASE_URL=postgresql://dx_migrator:[email protected]:55432/extractit \
  uv run alembic upgrade head
DX_DATABASE_URL=postgresql://dx_app:[email protected]:55432/extractit \
DX_DBOS_DATABASE_URL=postgresql://dx_dbos:[email protected]:55432/extractit \
DX_RUNTIME_HANDLERS=my_handlers \
  uv run python -m extractit.runtime.worker

Restart a worker with the same DX_RUNTIME_EXECUTOR_ID to recover the runs it left in flight (H1, C9; a kill -9 repeats at most the step in flight). Run pytest -m h1 after any DBOS upgrade; the version is pinned exactly in pyproject.toml. Tests: pytest tests/runtime (Postgres 16 via DX_TEST_DATABASE_URL or Docker).

Plan B (not activated)

If a critical H1 check (C2, C3, C5, C6 or C7) ever fails on a DBOS upgrade without an alternative in its API, replace dbos_app.py only (estimate +1–2 d):

  1. Queue: a run_queue table polled with SELECT … FOR UPDATE SKIP LOCKED, ordered by priority and queued_at;
  2. Per-project concurrency: a semaphore table (project_id, running) updated in the claim transaction, checked against RuntimeLimits.max_concurrent_runs;
  3. Steps, resume and fork: a step-output log (run_step_outputs(run_id, function_id, output)) replayed by the workflow driver; resume and fork re-run the handler and skip logged steps;
  4. Human review: the run leaves its slot and is re-queued by deliver;
  5. Schedules: a leader lock (pg_try_advisory_lock) in the worker.

RunService, handlers, steps, the outbox and the tables stay as they are.