Learning from evaluated data
T42 adds two mechanisms that learn from data the platform has already measured, behind the same
acceptance rule (paired_gate, see Quality contracts):
- Per-field model routing (OPT-04): the cheapest tier and model per group of fields that is not
measurably less accurate. Exposed as a
RoutingRun(rte_…). - Dynamic few-shot from corrections (CLS-05): reviewer corrections become examples that are
retrieved per document and inserted as data in the prompts of
classifyandextract.
Neither mechanism ever publishes: what they learn lands as a draft version, and promotion is a
person's decision. After promoting a routing or activating examples, recalibrate the confidence
and re-certify the thresholds (Calibrated confidence, T39): a new model, prompt or
set of examples invalidates the calibrators (needs_recalibration = true).
Per-field routing (/v1/routing-runs)
POST /v1/routing-runs
{"workflow_id": "wf_…", "base_version_id": "wfv_…", "snapshot_id": "dss_…",
"min_accuracy_drop": 0.01, "max_cost_usd": 50}
- Groups. The base version is evaluated on the snapshot's validation split. Fields are
grouped by type and difficulty bucket (error rate ≤ 5 %
easy, ≤ 20 %medium, elsehard); a group with fewer than 50 validation instances is merged into its nearest neighbour (same type first, then the nearest difficulty). - Search. The route space is
tier (fast, standard, agentic) × modelof the capability registry (GET /v1/models). For every group, the routes cheaper than the base route are tried in increasing price; each probe is a T38 evaluation of validation with the sealedoptimizeraccess context (atestscope is impossible by construction). The search stops at the first route whose Wilson lower bound of accuracy reachesbase accuracy − min_accuracy_drop; if none does, the group keeps the base route withno_feasible_cheaper_route. - Budget and cancellation. Every probe's spend is appended to
callstagged with therouting_run_id, the group and the route;spent_usdis their sum. The budget and the cancel flag (POST /v1/routing-runs/{id}:cancel) are checked before every probe: the run ends withbudget_exhausted(spend ≤ budget + one probe) orcancelledbefore the next probe. - Draft. When some group moved, a draft of the workflow is opened with
field_routeson the extract node. - Final verdict. One evaluation phase on test (
final_reportcontext, logged intest_split_access) compares base and routed.promotable = trueonly if the paired lower bound (hierarchical bootstrap, B = 2 000) of the accuracy difference is ≥ −1.0 point and the cost per 1 000 pages drops ≥ 10 % (non_inferiority_failed,insufficient_savingsotherwise). The verdict also records the validation accuracy and the shrinkage (validation − test). - Promotion.
POST /v1/routing-runs/{id}:promote {"alias": "production"}publishes the draft and moves the alias. Only a person (web session) may promote; API keys get 403. The production alias is never touched before.
The field_routes contract (06 §13)
The extract node gains an additive, optional key, validated by the node model of T25 and published
as schemas/learning/field-routes.schema.json:
{"type": "extract", "config": {"schema_id": "invoice", "tier": "standard",
"field_routes": {
"number_easy": {"tier": "fast", "model": "gemini-2.5-flash", "fields": ["total", "currency"]},
"string_hard": {"tier": "agentic", "model": "gemini-2.5-pro", "fields": ["supplier_name"]}}}}
fields are field ids or normalised paths (items[*].amount). The extraction makes one pass per
distinct route (fields in no group use the node's own tier/model_policy), merges the
candidates under the same output contract and evidence, and reports usage.by_route (calls,
tokens and cost per route; omitted when there are no routes, so older outputs are unchanged).
Versions saved before T42 serialise and hash exactly as before.
Note. T41 exposes its optimizations as
OptimizationRun. T42 duplicates a minimal budget / cancel / promote loop overpaired_gateof T38 instead of depending on T41; the two resources should be unified once both exist (one run resource with akind).
Few-shot from corrections (/v1/learning)
- Harvest (
POST /v1/learning/examples:harvest {"workflow_id"}): only reviews whosereview_document.state = verified(T32).extract: the corrected value with the field's evidence quotes;classify: the corrected label (decision pathclassification:<node_id>) with the first page, ≤ 800 tokens. At most 5 examples per document; two with cosine ≥ 0.95 count once. Examples start ascandidate. - Retrieval: TF-IDF over character 3–5-grams (
scikit-learn, BSD-3), fitted per example set and cached by its hash, exact cosine. No model download, no network. The 3 most similar examples go in the prompt as a delimited data block: escaped (JSON strings, delimiter look-alikes broken), labelled as example data that grants no tools, at most 1 500 tokens (the least similar is dropped first). - Gate (
POST /v1/learning/gate {"workflow_id", "node_id", "snapshot_id"}): the snapshot's test split is evaluated twice in thelearningcontext (logged intest_split_access): without and with examples. The examples becomeactiveonly if the paired lower bound of the improvement is > 0, no field loses more than 1 point, hallucination grows ≤ 0.2 points and the cost stays ≤ 110 %. One attempt per set of examples (409GATE_ALREADY_RUN). Fewer than 20 examples or 20 test items: 422INSUFFICIENT_DATA, nothing is evaluated. - Activation: a passing gate freezes the examples in an
ExampleSnapshot(lsn_…,content_hash= sha256 of the RFC 8785 JSON of the ordered examples; immutable) and opens a draft whose node haslearning_snapshot_id. Each execution records the snapshot and theexample_idsit used, so the same version and document rebuild the sameprompt_hash.:disableopens a draft without the snapshot;:activatere-enables a disabled one.
Data requirements
- A dataset snapshot of the workflow with
validationandtestsplits (T38),group_keyset when the corpus has templates. - Routing: ≥ 50 validation instances per group (smaller groups are merged).
- Few-shot: ≥ 20 learned examples and ≥ 20 test items; corrections from verified reviews.
Leakage and safety
- The routing search never reads
test; the only test read is the final verdict (once). - The gate never shows a document an example from its own packet, from any packet of the evaluated snapshot (including T38's copies with the same file content) or a near-duplicate (cosine ≥ 0.95).
- Examples of another tenant are never retrieved (RLS on every table; tested by the leak battery).
- An example that reads like an instruction ("ignore previous instructions") stays an inert quoted string inside the data block.
- Deleting a packet deletes its examples (
ON DELETE CASCADE) and marks every snapshot that contained onestale(it then contributes no examples at run time). - TF-IDF is lexical: weaker than dense embeddings for paraphrases across languages. It sits behind one retrieval port and can be replaced.
Errors
| Code | Status | When |
|---|---|---|
GATE_ALREADY_RUN |
409 | the gate already ran for this set of examples |
ROUTING_RUN_STATE |
409 | cancel of a finished run, promotion of a run that is not promotable |
SNAPSHOT_STATE |
409 | activating a stale snapshot, disabling a disabled one |
INSUFFICIENT_DATA |
422 | fewer than 20 examples or test items |
MODEL_NOT_AVAILABLE |
422 | the base model is not in the registry |
permission-denied |
403 | promotion by an API key (a person is required) |