Skip to content

H1 — DBOS Transact checklist

  • Date: 2026-11-17 22:12 UTC
  • Pinned in pyproject.toml: dbos==3.4.0 (installed: 3.4.0)
  • Database: PostgreSQL 16, DBOS system tables in schema dbos (role dx_dbos) of the application database
Check Result What it proves
C1 PASS Workflows and steps with retries (retries_allowed, max_attempts, interval_seconds, backoff_rate); outputs recorded in Postgres
C2 (critical) PASS Queues with global_concurrency, worker_concurrency, limiter and priority (lower value runs first)
C3 (critical) PASS Per-project fairness: partitioned queue keyed by project
C4 PASS cancel_workflow on a running and on an enqueued workflow
C5 (critical) PASS Resume a cancelled or failed workflow without repeating completed steps
C6 (critical) PASS fork_workflow from a step (function id from list_workflow_steps)
C7 (critical) PASS send/recv: 200 parked waits survive kill -9 of the worker
C8 PASS Cron schedule runs once per tick with two processes
C9 PASS Recovery after kill -9 in the middle of a fan-out of steps
C10 PASS DBOSClient enqueue without registered workflows; JSON serialization; dbos schema next to Alembic and RLS

Findings

  • C1: @DBOS.step(retries_allowed=True, max_attempts, interval_seconds, backoff_rate) retries in-process; step outputs are rows of dbos.operation_outputs (JSON with the portable serializer) and a re-run of the same workflow id replays them without executing.
  • C2: DBOS.register_queue(name, global_concurrency=, worker_concurrency=): queues are database-backed (table dbos.queues), discovered by workers within ~1 s; peak observed = 2 with worker_concurrency=2.
  • C2: limiter={'limit': 2, 'period': 1.0}: at most 2 starts per rolling second.
  • C2: Priority is always on for every queue in 3.4 (no priority_enabled flag): SetEnqueueOptions(priority=n), 1 = highest, ties in FIFO order; ExtractIt maps its 0-9 scale (9 highest) to 10 - p.
  • C3: Partitioned queues exist: partition_concurrency (global per partition) plus SetEnqueueOptions(queue_partition_key=project_id); partitions are visited in random order each poll, so a small project is not starved by a large backlog (B: 6 jobs in 2.1 s while A had 31 queued). The partition limit is one value per queue: per-project limits use one queue per limit value (tier).
  • C4: cancel_workflow flips the status to CANCELLED at once (enqueued workflows never start); a running workflow stops at its next step boundary (DBOSWorkflowCancelledError, a BaseException), here 1.0 s incl. a 1 s observation window.
  • C5: resume_workflow(id, queue_name=...) re-enqueues a CANCELLED workflow on the given queue (default: DBOS' internal queue) keeping its partition key; recorded steps replay.
  • C5: A FAILED (ERROR) workflow is not resumable with resume_workflow; rewind_workflow(id, start_step=<function_id of the failed step>, queue_name=...) re-runs it under the same id from that step (steps 1-2 keep one execution). The run service uses resume for cancelled runs and rewind for failed ones. With the portable JSON serializer a workflow error comes back as PortableWorkflowError (message kept, class lost): the runtime records its own error class in run_steps/runs.
  • C6: fork_workflow(id, start_step, queue_name=...) copies the outputs of steps before start_step and re-runs from it under a new id; SetWorkflowID fixes that id (the child run id); list_workflow_steps gives each step's function_id; the fork records forked_from.
  • C7: 200 workflows parked in DBOS.recv(topic, timeout_seconds=600) survive kill -9: after a restart with the same executor_id they are recovered, replay their recorded steps and receive the messages sent with DBOSClient.send (none lost, none doubled). A parked recv holds an executor thread and, inside a queue, a concurrency slot: the runtime parks HITL waits outside the slot (see runtime.md).
  • C7: recv(topic, timeout_seconds=0) returns at once; send needs an existing destination workflow (a CANCELLED one is fine). A workflow that cancels itself frees its queue slot (only PENDING counts); send + resume_workflow(id, queue_name=...) brings it back, recorded steps replay and the next recv gets the message (1.04 s). A cancel issued by the workflow on itself is NOT checkpointed (replay would cancel again), so the self-cancel is guarded by state outside DBOS. The runtime builds wait_for on this.
  • C8: DBOS.create_schedule(schedule_name, workflow_fn, schedule='* * * * * *', context=) persists the schedule in dbos.workflow_schedules; two processes of the same application ran 8 ticks exactly once each (workflow id derived from the schedule and tick; executions per executor: {'c8-a': 8}).
  • C9: kill -9 after step 5 of 10: on restart (same executor_id) DBOS recovers the PENDING workflow, replays recorded steps and finishes; re-executed steps: ['dx-h1-c9-2a707c:step:5'] (at most the one in flight).
  • C10: DBOSClient(system_database_url, application_name, serializer) enqueues by workflow name from a process that registers no workflow and never launches DBOS (the API process); the worker application executes it.
  • C10: A JSON serializer exists (DBOSPortableJSONSerializer, set in DBOSConfig.serializer and on every DBOSClient): workflow inputs, step outputs and messages are JSON; bytes and arbitrary objects raise TypeError, datetimes become ISO strings. The default is pickle; the runtime always uses the JSON one and passes only ids, hashes and small JSON.
  • C10: DBOS runs its own migrations inside schema dbos owned by dx_dbos (NOBYPASSRLS, not superuser, needs no CREATE on the database); nothing lands in public, Alembic's alembic_version is untouched, dx_app has no USAGE on dbos, and the RLS audit of T11 passes with dbos as its explicit exempt schema.

Decision

H1 holds: the runtime uses DBOS Transact on the application Postgres (no new service). Plan B stays documented in docs/architecture/runtime.md and is not activated.