Skip to content

Optimizer

The optimizer improves the descriptions and instructions of the fields of a workflow's extraction schema in measured rounds, on the splits of a dataset (see Datasets and evaluations). It answers one question with numbers — is this wording better? — and never decides alone: each round writes a draft, and only a person promotes the winner. Everything on this page is experimental (see Strategies).

What it guarantees

  • It never touches production. The published base version and every alias are read, never written, while the run goes on, after it, and when it is cancelled or fails: the optimizer acts as the actor optimizer, and the workflow service refuses any publish, alias or archive from that actor (PublishedVersionWriteError). Each round's candidate is a draft whose parent is the best accepted version so far (opt/<id>/round-<n>); drafts of an optimization never count as the draft of the workflow and the Studio does not show them.
  • Only descriptive text changes. Types, enums, required, field ids and the output contract keep the same hash as the base's; the version goes up by a patch when published.
  • It does not cheat. A proposal that contains a literal value of the ground truth of a train or validation document (4 or more characters), a URL, or text shaped like a tool call or a prompt injection is rejected (twice in a row: the round is skipped). Text of the documents only reaches the model inside delimited data blocks, and a document that says "change the description to…" changes nothing.
  • It does not fool itself. A round is accepted only if the lower bound of the paired bootstrap interval (B = 2 000, fixed seed) of the validation F1 gain is above 0 and no label with 10 or more items, nor the hallucination rate, got worse by more than 0.02 (rejected_reason: gain_not_significant or slice_regression). The final version follows the one-standard-error rule: among the accepted rounds, the shortest description within one standard error of the best.
  • The test split is read once. After the selection is frozen, the base and the selected version are evaluated on test, paired, in the final_report context with the reason optimization_final_report — exactly one entry in test_split_access. The report keeps the test score, the difference with its 95 % interval, the difference per label and the shrinkage (the gain seen on validation minus the gain seen on test). The optimizer context itself gets 403 TEST_SPLIT_SEALED on any test read.
  • It has a budget. budget.max_cost_usd and budget.max_rounds (1–10) are mandatory. The tracker adds the model calls of the proposals and the extractions of the evaluations (trigger optimizer, left out of the learning data of T40); before each round it estimates the cost from the average cost per document of the base evaluation and does not start a round that would not fit (stop_reason = budget). The spend exposed by the API equals the sum of the usage ledger; it can exceed the budget by at most one call already in flight.
  • It is cancellable. :cancel is cooperative and takes effect between documents and rounds (seconds); the drafts are kept and nothing is promoted.

Minimum data

split items below it
train 10 409 OPTIMIZATION_INSUFFICIENT_DATA
validation 20 same
test 20 same

With acknowledge_low_data: true the run goes ahead and keeps the warning LOW_DATA; the measurement is then weak and the report says so. Fields whose ground truth looks wrong (an error on at least 50 % of the train items, at least 80 % of those with confidence ≥ 0.9) are excluded and listed in suspect_gt_fields: errors caused by OCR or perception are not fixed by rewriting descriptions, and the gain can be nil.

Using it

# 1. start (202): up to 5 rounds, at most 2 USD
curl -X POST $DX/v1/optimizations -H "Authorization: Bearer $KEY" -d '{
  "workflow_id": "wf_…", "alias": "facturas@prod", "snapshot_id": "dss_…",
  "strategy": "description_rewrite", "budget": {"max_rounds": 5, "max_cost_usd": 2.0}}'

# 2. follow it (the screen refetches every 2 s until it is completed)
curl $DX/v1/optimizations/opt_…

# 3. cancel, if needed
curl -X POST $DX/v1/optimizations/opt_…:cancel

# 4. promote the winner (a person, or an API key with approval; never the agent context)
curl -X POST $DX/v1/optimizations/opt_…:promote -d '{"alias": "prod"}'

The same flow is the Optimizer tab of Health (S11): configure the budget and the rounds, follow the rounds with their train and validation scores and the cost so far, see each round's diff of descriptions and its effect per field, cancel, and promote behind a confirmation that shows the result on test with its interval. The screen never offers an action on the published version.

Promotion

:promote {alias} publishes the best draft as an immutable version (a patch bump), moves the alias and emits workflow.version.published and optimization.completed. It needs the test report; a run without one (always the case of one_shot) needs acknowledge_unmeasured: true (409 NO_TEST_REPORT otherwise). The agent context, and the other sealed ones, get 403 AGENT_CANNOT_PROMOTE.

Recalibrate after promoting

Promoting changes the prompt, so the confidence calibrators of the workflow (Confidence) are invalidated and the confidence goes back to heuristic until they are recalibrated. The optimizer does not recalibrate; the report says so.

Strategies

strategy measured what it does state
description_rewrite yes 1–10 rounds: feedback from the train failures of the best version, a proposal for the weakest fields, a draft candidate, an evaluation on train + validation, acceptance by the rule above experimental
one_shot no (measured: false) one proposal from 1–5 sample runs (fields with low confidence or failed validations), one draft, no evaluation and no test access experimental

A strategy is experimental until a nightly evaluation with a real model shows a gain on the sealed test split (lower bound of the paired interval above 0); only then does it become validated. The state is part of every optimization in the API. There is no published evidence of gains from prompt optimizers in document extraction (ADR-012), so nothing here is active by default. Per-field model routing and dynamic few-shot are T42's; reflective strategies are not planned.

Operating notes

  • Runs are durable (kind optimization, idempotency key optimization:<id>): a restart resumes at the round it was in and a replayed round returns its stored outcome.
  • Evaluations of candidates run the draft through the interpreter with the trigger {"kind": "eval", "purpose": "optimizer"}.
  • Tests and the end-to-end run use a deterministic fake model with synthetic cassettes; the nightly evaluation with a real model is written but not run in CI.