Skip to content

Calibrated confidence, certified thresholds and the audit

T08 gives every field a heuristic confidence (kind: heuristic): a product of signals, useful to rank fields but not a probability. T39 (extractit.quality.calibration) turns it into a calibrated probability per segment, certifies auto-approval thresholds with a statistical guarantee and audits at random a share of what was auto-approved. The flow is:

  1. Fit calibrators from completed evaluations (T38).
  2. Activate the ones that passed the quality gate.
  3. Certify a threshold policy with the active calibrators.
  4. Decide each document (auto-approve or review) and audit a random share.

Recalibrate after changing the model or the prompt

A calibrator belongs to one model snapshot and one prompt hash. After a model upgrade or any change of the prompt (schema, instructions), the next run keeps the heuristic score with reason: calibrator_stale, and every threshold policy that used those calibrators becomes invalidated the first time it is resolved. Run a new evaluation, fit, activate and certify again.

1. Fit

POST /v1/calibrators:fit
{"workflow_id": "wf_…", "evaluation_ids": ["evr_…"],
 "model_snapshot": "gemini-2.5-flash-001", "prompt_hash": "…"}
  • Labels come from evaluation_field_results (T38), read as the calibration access context: the sealed test split is never read (403 TEST_SPLIT_SEALED). A field is right when its class is correct or format_only; a right null is right too; unlabelled fields and fields without a heuristic score are left out.
  • Rows of train fit; rows of validation (other documents) gate. If a document is in train of one evaluation and in validation of another (a resplit), it is dropped from validation (excluded_overlap).
  • One calibrator per segment: each field type with enough data, plus the global segment * of the snapshot (the fallback of a field type without its own calibrator).
training data of the segment method
fewer than 300 fields or 30 errors none: the score stays heuristic (skipped, insufficient_data)
300–2 999 fields, or fewer than 200 errors Platt (logistic regression on logit(score))
3 000 fields or more and 200 errors or more isotonic (out_of_bounds = clip)

Quality gate on validation (15 bins): the adaptive ECE (equal-mass bins) must be at most the noise floor + 0.01 and, with 1 000 fields or more, at most 0.03 with an AUROC of at least 0.85. A calibrator that fails is rejected; below 1 000 validation fields it is provisional.

Each calibrator (cal_…) stores its scope, method, n_samples, n_errors, ece, adaptive_ece, noise_floor, auroc, model snapshot, prompt hash, fitted_at and the sha256 of its artifact: plain JSON (Platt coefficients or isotonic thresholds) in the encrypted blob store. Nothing is pickled; the loader accepts only that JSON.

2. Activate

POST /v1/calibrators/{id}:activate makes a fitted calibrator the active one of its segment and archives the previous one (:archive retires one by hand). In code, the hook of T08's pipeline is:

from extractit.quality.calibration.registry import CalibratorRegistry
from extractit.extract.pipeline import PipelineOptions, extract

hook = CalibratorRegistry(session, tenant_id, project_id, artifacts=store).confidence_post(wf_id)
out = extract(pdf, schema, options=PipelineOptions(confidence_post=hook))

A field then reports kind: calibrated_probability with calibrator {id, version, scope {field_type, model_snapshot}, ece, n_samples, fitted_at} — only when the run's model snapshot and prompt hash are the calibrator's. The document confidence stays heuristic.

3. Certify

POST /v1/threshold-policies:certify
{"workflow_id": "wf_…", "evaluation_ids": ["evr_…"]}

For each field type, Learn-then-Test certifies the lowest threshold λ whose selective error (wrong fields among those with probability ≥ λ) is at most α with probability 1 − δ:

  • fixed grid of 52 thresholds: 0.50 to 0.99 in steps of 0.01, then 0.995 and 0.999; its sha256 is the policy's grid_hash (a different grid is refused with GRID_MISMATCH);
  • exact binomial p-value P(Bin(n, α) ≤ k) on the n accepted fields of validation with k errors (with α = 0.01 and no errors, 230 accepted fields certify, 229 do not);
  • fixed sequence: the start is the lowest λ whose risk on train is at most α / 2; the test walks to laxer thresholds and stops at the first p > δ (the last success is certified). method: bonferroni certifies every λ with p ≤ δ / 52 instead;
  • α = 0.01 for amounts (number, integer) and 0.03 for the rest (send alpha per type to tighten identifiers, which are strings), δ = 0.10, at least 1 000 certification fields per type;
  • final threshold = max(λ_cost, λ_LTT), with the cost threshold p* = 1 − C_R / ((1 − e_H) · C_E) (review cost 0.30, cost of an error 30, human error 0.05: 0.9895).
error when
422 NOT_CALIBRATED no active calibrator: the scores are heuristic; nothing is created
422 CERT_SET_OVERLAPS_CALIBRATOR the certification documents were used to fit a calibrator
422 INSUFFICIENT_DATA fewer than 1 000 fields in every field type (n_required)
422 NO_CERTIFIABLE_THRESHOLD no threshold of the grid passes for any field type
403 TEST_SPLIT_SEALED split: test

The policy (thp_…) is immutable (a database trigger): per field type lambda, alpha, delta, n_cert, n_accepted, n_errors, upper_bound (Clopper-Pearson), coverage, the calibrators and evaluations used and a reference_histogram (20 bins of the calibrated scores of the certification set, for drift monitoring in T40). :tighten moves every threshold one grid step up and sets needs_recertification; only a new certification relaxes it. :invalidate retires it; a change of model snapshot, prompt or calibrator invalidates it when it is resolved for a run.

4. Decide and audit

decision = ThresholdService(session, tenant_id, project_id, actor=..., artifacts=store).decide(
    result, validations, policy_id
)  # AutoApprovalDecision {auto_approved, reasons[], field_decisions[]}

A document is auto-approved when every required field has a calibrated probability at least the threshold of its type, no blocking rule failed and the policy is valid. Without required fields in the schema, every field with a value counts. The reasons say why not (required_below_threshold, required_not_calibrated, required_missing, field_type_not_certified, blocking_rule_failed, policy_invalid, no_fields).

The audit (AuditSampler.consider(run_id, decision), or POST /v1/audit-samples:consider) selects an auto-approved run when HMAC-SHA256(seed, run_id) / 2^256 < π (2 % by default, 0.5–10 %): the same seed gives the same selection, and without it the selection cannot be predicted. The seed of each workflow is 32 random bytes encrypted with the tenant's key (T11 envelope); it never appears in the API or in logs, and rotating it starts a new seed_version. Every considered run is stored with its stratum, pi and selected; a selected run gets an audit.sampled event in its outbox and is handed to the AuditTaskSink (the review inbox of T33).