Skip to content

Learning from evaluated data

T42 adds two mechanisms that learn from data the platform has already measured, behind the same acceptance rule (paired_gate, see Quality contracts):

  • Per-field model routing (OPT-04): the cheapest tier and model per group of fields that is not measurably less accurate. Exposed as a RoutingRun (rte_…).
  • Dynamic few-shot from corrections (CLS-05): reviewer corrections become examples that are retrieved per document and inserted as data in the prompts of classify and extract.

Neither mechanism ever publishes: what they learn lands as a draft version, and promotion is a person's decision. After promoting a routing or activating examples, recalibrate the confidence and re-certify the thresholds (Calibrated confidence, T39): a new model, prompt or set of examples invalidates the calibrators (needs_recalibration = true).

Per-field routing (/v1/routing-runs)

POST /v1/routing-runs
{"workflow_id": "wf_…", "base_version_id": "wfv_…", "snapshot_id": "dss_…",
 "min_accuracy_drop": 0.01, "max_cost_usd": 50}
  1. Groups. The base version is evaluated on the snapshot's validation split. Fields are grouped by type and difficulty bucket (error rate ≤ 5 % easy, ≤ 20 % medium, else hard); a group with fewer than 50 validation instances is merged into its nearest neighbour (same type first, then the nearest difficulty).
  2. Search. The route space is tier (fast, standard, agentic) × model of the capability registry (GET /v1/models). For every group, the routes cheaper than the base route are tried in increasing price; each probe is a T38 evaluation of validation with the sealed optimizer access context (a test scope is impossible by construction). The search stops at the first route whose Wilson lower bound of accuracy reaches base accuracy − min_accuracy_drop; if none does, the group keeps the base route with no_feasible_cheaper_route.
  3. Budget and cancellation. Every probe's spend is appended to calls tagged with the routing_run_id, the group and the route; spent_usd is their sum. The budget and the cancel flag (POST /v1/routing-runs/{id}:cancel) are checked before every probe: the run ends with budget_exhausted (spend ≤ budget + one probe) or cancelled before the next probe.
  4. Draft. When some group moved, a draft of the workflow is opened with field_routes on the extract node.
  5. Final verdict. One evaluation phase on test (final_report context, logged in test_split_access) compares base and routed. promotable = true only if the paired lower bound (hierarchical bootstrap, B = 2 000) of the accuracy difference is ≥ −1.0 point and the cost per 1 000 pages drops ≥ 10 % (non_inferiority_failed, insufficient_savings otherwise). The verdict also records the validation accuracy and the shrinkage (validation − test).
  6. Promotion. POST /v1/routing-runs/{id}:promote {"alias": "production"} publishes the draft and moves the alias. Only a person (web session) may promote; API keys get 403. The production alias is never touched before.

The field_routes contract (06 §13)

The extract node gains an additive, optional key, validated by the node model of T25 and published as schemas/learning/field-routes.schema.json:

{"type": "extract", "config": {"schema_id": "invoice", "tier": "standard",
  "field_routes": {
    "number_easy": {"tier": "fast", "model": "gemini-2.5-flash", "fields": ["total", "currency"]},
    "string_hard": {"tier": "agentic", "model": "gemini-2.5-pro", "fields": ["supplier_name"]}}}}

fields are field ids or normalised paths (items[*].amount). The extraction makes one pass per distinct route (fields in no group use the node's own tier/model_policy), merges the candidates under the same output contract and evidence, and reports usage.by_route (calls, tokens and cost per route; omitted when there are no routes, so older outputs are unchanged). Versions saved before T42 serialise and hash exactly as before.

Note. T41 exposes its optimizations as OptimizationRun. T42 duplicates a minimal budget / cancel / promote loop over paired_gate of T38 instead of depending on T41; the two resources should be unified once both exist (one run resource with a kind).

Few-shot from corrections (/v1/learning)

  1. Harvest (POST /v1/learning/examples:harvest {"workflow_id"}): only reviews whose review_document.state = verified (T32). extract: the corrected value with the field's evidence quotes; classify: the corrected label (decision path classification:<node_id>) with the first page, ≤ 800 tokens. At most 5 examples per document; two with cosine ≥ 0.95 count once. Examples start as candidate.
  2. Retrieval: TF-IDF over character 3–5-grams (scikit-learn, BSD-3), fitted per example set and cached by its hash, exact cosine. No model download, no network. The 3 most similar examples go in the prompt as a delimited data block: escaped (JSON strings, delimiter look-alikes broken), labelled as example data that grants no tools, at most 1 500 tokens (the least similar is dropped first).
  3. Gate (POST /v1/learning/gate {"workflow_id", "node_id", "snapshot_id"}): the snapshot's test split is evaluated twice in the learning context (logged in test_split_access): without and with examples. The examples become active only if the paired lower bound of the improvement is > 0, no field loses more than 1 point, hallucination grows ≤ 0.2 points and the cost stays ≤ 110 %. One attempt per set of examples (409 GATE_ALREADY_RUN). Fewer than 20 examples or 20 test items: 422 INSUFFICIENT_DATA, nothing is evaluated.
  4. Activation: a passing gate freezes the examples in an ExampleSnapshot (lsn_…, content_hash = sha256 of the RFC 8785 JSON of the ordered examples; immutable) and opens a draft whose node has learning_snapshot_id. Each execution records the snapshot and the example_ids it used, so the same version and document rebuild the same prompt_hash. :disable opens a draft without the snapshot; :activate re-enables a disabled one.

Data requirements

  • A dataset snapshot of the workflow with validation and test splits (T38), group_key set when the corpus has templates.
  • Routing: ≥ 50 validation instances per group (smaller groups are merged).
  • Few-shot: ≥ 20 learned examples and ≥ 20 test items; corrections from verified reviews.

Leakage and safety

  • The routing search never reads test; the only test read is the final verdict (once).
  • The gate never shows a document an example from its own packet, from any packet of the evaluated snapshot (including T38's copies with the same file content) or a near-duplicate (cosine ≥ 0.95).
  • Examples of another tenant are never retrieved (RLS on every table; tested by the leak battery).
  • An example that reads like an instruction ("ignore previous instructions") stays an inert quoted string inside the data block.
  • Deleting a packet deletes its examples (ON DELETE CASCADE) and marks every snapshot that contained one stale (it then contributes no examples at run time).
  • TF-IDF is lexical: weaker than dense embeddings for paraphrases across languages. It sits behind one retrieval port and can be replaced.

Errors

Code Status When
GATE_ALREADY_RUN 409 the gate already ran for this set of examples
ROUTING_RUN_STATE 409 cancel of a finished run, promotion of a run that is not promotable
SNAPSHOT_STATE 409 activating a stale snapshot, disabling a disabled one
INSUFFICIENT_DATA 422 fewer than 20 examples or test items
MODEL_NOT_AVAILABLE 422 the base model is not in the registry
permission-denied 403 promotion by an API key (a person is required)