Skip to content

Review inbox (S8) and human review inside a run

A workflow with a review node pauses the run when a document needs a person, hands the work out through an inbox (/review) and wakes the run up again when the last task is done — without repeating any step. This page covers the policy of the node, the tasks, assignment, the SLA, double review, the permission matrix and the contract with the interpreter (T26). The review screen itself (fields, decisions, verification) is Review in the viewer.

The review node policy

ReviewPolicy is part of the node configuration; its JSON Schema is schemas/review/review_policy.schema.json (and the node-type registry serves it to the node panel of the canvas). Every field is optional.

Field Meaning Default
when always, on_blocking_failure, on_low_confidence, on_unclassified, on_sampling on_blocking_failure
threshold confidence below which a field (or the document) is reviewed (on_low_confidence) 0.8
sampling_rate share of runs the QA sample takes (on_sampling), 0–1 0
auto_approve.threshold_policy_id thp_…: a certified threshold policy of calibrated confidence decides instead of when none
queue queue the tasks go to default
assignment manual or round_robin manual
reviewers usr_… ids of the round-robin; empty: the members with Reviewer role or above []
sla_minutes minutes from creation to due_at 1440
escalate_to usr_… that receives a task when its SLA expires none
double_review two reviewers work in isolation, a third adjudicates disagreements false
audit_sla_minutes SLA of the random-audit tasks 2880
timeout_hours how long the run waits before the node fails with REVIEW_TIMEOUT 168

Canonical names and aliases. The stored (and hashed) spelling is the one T26 introduced: when and threshold. The ticket's spellings are accepted on input and normalized to the canonical ones, so a workflow written either way has the same graph hash:

You may write Stored as
trigger: always when: always
trigger: low_confidence when: on_low_confidence
trigger: rule_failure when: on_blocking_failure
trigger: sampling when: on_sampling
confidence_below threshold

Which documents need a person

when Opens a task when… Reason recorded
always always always
on_blocking_failure a blocking rule failed upstream blocking_failure + the failed rule ids
on_low_confidence a field, or the document, has confidence < threshold low_confidence, and fields_to_review lists those fields
on_unclassified a classification left the document unclassified unclassified
on_sampling the QA sample takes the run qa_sampling

The QA sample is deterministic: sha256(workflow_id:run_id) mod 10 000 < sampling_rate × 10 000, so the same run always gets the same decision and the rate holds over a population (0.1 over 1 000 fixed runs gives 100 ± 15 tasks). It is a quality check with a public seed; the secret audit sample of T39 is separate (below). When there is nothing to review (no field below the threshold, every field pre-approved) no task is created and the node continues.

Auto-approval and random audit (CNF-04, CNF-05)

With auto_approve.threshold_policy_id the node asks ThresholdService.decide(result, validations, policy_id) of T39:

  • auto-approved (every required field above its λ and no blocking rule failed): no task, the run continues. The document is then offered to AuditSampler.consider; when the sample takes it a task of kind audit enters the queue audit with reason random_audit and due_at = created_at + audit_sla_minutes. The run does not wait for it; verifying it in the review screen is what labels the sample for T40. It escalates with the same sweep as any task.
  • not auto-approved: a normal task whose reasons are the decision's (required_below_threshold, blocking_rule_failed, …) and whose fields_to_review are the fields that did not clear their λ.
  • no decision possible (no such policy, an output that does not parse): the safe answer is a person, with reason auto_approval_unavailable. Nothing is ever auto-approved without a valid policy.

Tasks

review_task (rt_…) is one unit of work: a run, a node, a role and a due date.

Field
state open → in_progress (claimed) → completed; escalated after the SLA; expired when the run ended without it
kind review (the run waits for it) or audit (it does not)
queue, reason[], fields_to_review[] where it sits, why, and what to look at
assignee usr_…; with an API key the actor is the key (key_…), not a person
task_role, group_id primary, secondary (double review) or adjudicator
due_at, priority the keyset listing orders by (due_at, id): most urgent first

A run is idempotent per (run, node, scope, kind, role): opening the same task twice returns the first one. Cancelling a run — or one that fails — moves its pending tasks to expired; completing one of them then answers 409.

API

Route Needs What it does
GET /v1/review-tasks?queue=&state=&assignee=&reason=&workflow_id=&kind=&cursor=&limit= result:read the inbox (state=pending is open + in progress + escalated; assignee=me\|unassigned\|usr_…)
GET /v1/review-tasks/summary result:read counters: open (the badge of the sidebar), waiting, in_progress, escalated, overdue, mine, unassigned, by_queue
GET /v1/review-tasks/{id} result:read one task; an adjudicator task also carries both reviewers' values
GET /v1/review-tasks/{id}/decisions result:read the decisions of that task (a reviewer of a double review asking for the other's gets 404)
POST /v1/review-tasks/{id}:claim / :release review:act + assignee take / give back a task
POST /v1/review-tasks/{id}:complete review:act + assignee verify (or reject) the document for this task; idempotent
POST /v1/review-tasks:assign project:manage {task_ids \| filter + exclude, assignee \| strategy: round_robin, reviewers?}; up to 1 000 per call
GET /v1/review-reviewers project:manage who a task can be assigned to
GET /v1/review-metrics?since=&until=&reviewer= result:read per reviewer: documents/hour, % of corrected fields, double-review agreement

Errors are RFC 9457 problems: not-assigned-verifier (403), insufficient-scope (403, an API key without the review scope), conflict (409), not-found (404). Everything of T32 (PATCH /fields, :approve, :reject, rows) goes through the same guard.

Assignment

  • Manual: assignee is set on every task of the call.
  • Round-robin: a counter per (tenant, project, queue) moves with one INSERT … ON CONFLICT DO UPDATE … RETURNING, so concurrent batches never read the same position and a batch over k reviewers gives each ⌊n/k⌋ or ⌈n/k⌉ tasks (difference at most one, whatever the counter was). The next batch continues where the last stopped. Without reviewers nobody is assigned: the task waits in the queue.
  • Closed tasks in a selection are skipped and reported in skipped. 500 tasks assign in under 2 s.
  • An Admin can reassign a task in progress; the previous assignee's claim is cleared.

SLA and escalation

due_at = created_at + sla_minutes (audit tasks: audit_sla_minutes). The counter of the inbox is green, amber in the last fifth of the SLA and red from due_at on; the colour is never the only signal (the state is written next to it). The counter refreshes every second in the browser; the list refetches every 10 s.

A scheduled DBOS workflow, dx_review_sweep_sla (every 60 s, idempotent), does three things:

  1. moves open or in-progress tasks past due_at to escalated and assigns them to escalate_to (a task that was already escalated is not touched, so running the sweep twice changes nothing);
  2. expires the overdue tasks of runs that were cancelled or failed;
  3. redelivers the wake-up signal of runs that are still waiting_review although every task is done (a crash between the verdict and the signal).

Double review

With double_review: true the node opens two independent tasks (primary, secondary) with the same group_id. Each reviewer works in isolation: their decisions are stored with the task and applied = false, so they do not change the effective value, and GET …/decisions of the other task is 404. One person cannot complete both (409).

When both are done the decisions are compared field by field:

  • agree → the agreement is applied as system:double_review, the document is verified (one review.completed) and the run resumes;
  • disagree (in several fields at once is fine) → what they agree on is applied now and an adjudicator task opens with both reviewers' values for each disputed field (assigned to escalate_to when set). Its decision is applied as an override and verifies the document.

The agreement metric is matching fields / compared fields; a field only one reviewer touched is compared against the original value.

Permission matrix

Viewer / read key Reviewer Admin / Owner
List tasks, summary, metrics, decisions of a task yes yes yes
Claim, release, complete 403 their own tasks any task
PATCH /fields, rows, :approve, :reject on a run with a pending task 403 only if assigned (else 403 not-assigned-verifier) yes
Same on a run without a pending task 403 yes yes
Assign in batch, reassign, list reviewers 403 403 yes

A task nobody holds must be claimed first (claiming assigns it); completing one assigns it to the caller. Scopes are checked before roles: an API key without the review scope gets insufficient-scope. With an API key the actor of a task is the key, so metrics and completed_by name key_…, not a person. Another tenant's task or run is always 404.

The screen (S8)

/review (and /workflows/{id}/review, which filters by workflow):

  • the table is windowed with @tanstack/react-virtual over @tanstack/react-table: a few dozen rows are in the DOM whatever the size of the queue (at most 60), more pages load as you scroll;
  • filters (queue, reason, assignee, state) live in the URL;
  • assign: tick tasks (or «select every task that matches the filters» and untick the exceptions), then choose a reviewer or round-robin (Admin only);
  • focus mode (button, or the Review action of a row) opens the first task in the review screen; the review's «verify and next» (Shift+Enter) verifies the document and opens the next task in queue order, and the last one ends with «Cola vacía». A task nobody holds is claimed before it opens. Shift+← goes back;
  • the metrics tab shows each reviewer's figures with the counts they come from;
  • the badge next to «Revisión» in the sidebar is summary.open, refreshed every 10 s.

Contract with the interpreter (T26)

The node is T26's review handler, extended — not duplicated. GateReviewTasks (src/extractit/review/gate.py, the only place coupled to the interpreter) implements T26's ReviewTaskPort.

  1. Open. The handler evaluates the policy over the extraction, validations and classification of the run. With a reason it marks the node waiting, calls ports.review.open(...) (which creates the task(s) idempotently, in the same transaction) and emits node.waiting.
  2. Pause. It parks on T14's ctx.wait_for(review_topic(node_id, scope_key), timeout): the run is waiting_review (event run.waiting_review, one per run), leaves its queue slot and holds no thread, connection or worker. It survives restarts and kill -9: the task and its due_at are rows, the wait is a durable DBOS parking.
  3. Signal. Completing the last task verifies the document with T32's ReviewService (exactly one review.completed) and, after the verdict is committed, sends the payload {"decision": "approve"|"reject", "corrections": {...}, "effective_data": {...}, "reviewer": "…"} to complete_review with the idempotency key sha256(run_id:node_id:review). A second delivery answers noop. Verifying the document directly (POST /v1/runs/{id}:approve) closes its pending tasks and wakes the node the same way. POST /v1/runs/{id}/nodes/{node_id}:complete remains the low-level door.
  4. Resume. The node leaves waiting, applies the reviewer's values to the extraction (review_apply) and continues; reject fails the node with REVIEW_REJECTED; the timeout fails it with REVIEW_TIMEOUT and expires its tasks. Parse and extract are not repeated: their node runs are succeeded with one attempt, and the replay of the interpreter reuses them.
  5. Cancel. Cancelling a waiting_review run finalizes it and expires the tasks it still had.

Until the run finishes, GET /v1/runs/{id}/result of a run waiting for its review answers a provisional result (the extraction as it stands), so the review screen can open it.

Tests: tests/review_tasks/ (policy, assignment, SLA, double review, guard, metrics, listing at scale) and tests/workflows/interpreter/test_review_tasks_run*.py (the real interpreter: pause, kill -9, resume, double review, cancellation, auto-approval, audit tasks); web/src/features/inbox/ and web/e2e/mocked/inbox.spec.ts for the screen.