Skip to content

ADR-012: the builder agent measures, but a person decides

Status: accepted (T43 introduced the agent; T44 adds the quality loop).

Context

The builder agent (T43) proposes ChangeSets that a person approves. T41 optimizes field descriptions on measured data and T38 evaluates versions on managed datasets. The agent should be able to use both โ€” launch evaluations, start optimizations, attach before/after evidence to its proposals โ€” without becoming a way to spend money, read the sealed test split or change production.

Decisions

  1. Approval by cost. launch_eval creates a RunRequest with an estimated cost. At or under 2.00 USD and 20 documents (per project: approval_threshold_usd, approval_max_docs) the evaluation starts (auto); above either, it is proposed and nothing runs until a person (or an API key with workflow:write) approves. An optimization always needs approval, and its max_cost_usd never exceeds the approved amount. The agent actor gets 403 on every decision.
  2. The agent never reads test. Its evaluations run with the agent access context; a scope with test is 403 TEST_SPLIT_SEALED. The agent has no promotion tool and POST /v1/optimizations/{id}:promote refuses its context (AGENT_CANNOT_PROMOTE); a finished optimization is reported with a promote_suggestion card that carries no operations. The test result it quotes is the one the API returns (the optimizer's single logged read).
  3. Evidence, not opinion. A ChangeSet may carry {before, after, compare, cost_delta}: two evaluations of the same snapshot on at most 20 validation documents, the comparison being T38's paired one, copied. The base evaluation is reused by (graph hash, snapshots, config hash, scope). The draft is evaluated as a candidate draft (invisible, never published). Without ground truth it is unmeasured. The UI shows n and the interval.
  4. Policy require_evidence_before_approval (project, default false): on, approving without measured evidence is 409 EVIDENCE_REQUIRED; off, it is allowed and recorded as approval.evidence = "none". Unmeasured evidence does not satisfy the policy.
  5. Code is never written by the model. Integration examples (Python, TypeScript, curl) come from deterministic templates and types generated from the output_contract; the model writes only the explanation. Keys are read from DX_API_KEY, never embedded.

Consequences

  • The agent's spend is bounded by what a person approves; its tools cannot reach the test split.
  • A small validation set gives wide intervals: the card says so instead of claiming a gain.
  • The ChangeSet contract (06 ยง14) widens evidence[] from run references to measures; readers that only knew run_sample references keep working (those entries are unchanged).