Monitoring formulas (T40)
Every number of the Monitoring screen (S12) and of GET /v1/workflows/{id}/metrics,
/monitoring/quality and /monitoring/reliability is defined here once. The code is
src/extractit/quality/monitoring/{formulas,silent_error,drift}.py; an independent
implementation that uses only the Python standard library lives in
tests/reference/monitoring_reference.py and the tests compare both (counts and rates exactly,
floats within 1e-9).
The window
A run belongs to the window [from, to) when it is a workflow run of the workflow, it finished
in the window (finished_at, UTC), its purpose is production (input_ref.trigger.purpose;
test, eval, optimizer and learning runs are excluded) and its status is succeeded,
partially_succeeded or failed. Cancelled runs never count. A window is at most 90 days (400
validation-error otherwise); the default is the last 30 days. Buckets (hour, day) are UTC.
Rates
| Metric | Numerator | Denominator |
|---|---|---|
| STP | succeeded runs that never entered waiting_review and have no review decision |
finished runs |
| Review rate | runs that entered waiting_review (outbox event run.waiting_review) or have any decision |
finished runs |
| Error rate | failed runs |
finished runs |
| Cost per document | Σ runs.cost_usd (the ledger of T14) |
finished runs |
| Cost per page | Σ cost_usd of the runs with pages |
Σ pages |
partially_succeeded runs are finished (they count in every denominator) but are not
straight-through. When T33's review tasks land, one line (register_review_touch) adds them to the
"touched by a person" test.
Accuracy from corrections (with its bias)
accuracy = fields whose every decision is approve / fields with any field decision
(approve, correct, mark_absent, reject of review_decision, T32), with n and the Wilson
95 % interval. A field approved and later corrected is not accurate.
Selection bias. Only fields a person reviewed are measured, and people review the hard cases
(low confidence, failing rules). Accuracy from corrections therefore under-states production
accuracy and must not be compared across workflows with different review policies. The unbiased
measure is the silent error below. With no reviewed field the accuracy is null with
reason = no_reviewed_fields.
Latency
latency = finished_at - started_at in ms; p50 and p95 are nearest-rank percentiles
(percentile_disc, the smallest value with at least q of the values at or below).
Confidence histogram
10 equal intervals of [0, 1] (the last one closed) of the field scores of the runs, from
run_confidence (a 20-bin histogram stored when the run result is composed; two bins of 20 are one
of 10). kind is heuristic, calibrated, mixed or none; a heuristic score is never
presented as a probability.
Silent error (the random audit)
Unit: an audited auto-approved document i with inclusion probability π_i (T39's
audit_sample), fields_i fields and errors_i wrong fields once a person verified it
(AuditLabeler: a field is wrong when it has a decision other than an unchanged approve).
silent_error = Σ (1/π_i)·errors_i / Σ (1/π_i)·fields_i
accepted_document_error = Σ (1/π_i)·[errors_i > 0] / Σ (1/π_i)
With equal π this is the plain proportion k/n; the upper bound is the one-sided
Clopper-Pearson bound at 1 - δ (δ = 0.05) and the 95 % interval is the two-sided Clopper-Pearson
interval. With strata (different π), the bound is the 1 - δ percentile of a document-level
stratified bootstrap (B = 2 000, fixed seed). In 200 seeded simulations the 95 % interval covers
the true error in ≥ 90 % of the cases with uniform and stratified sampling
(tests/quality/monitoring/test_monitoring_eval.py).
When the upper bound exceeds α_ref — the threshold of the silent_error_upper_above rule when
set, else the largest α of the latest threshold policy — the policy is tightened once
(ThresholdService.tighten), threshold.breach is deposited once and the rule fires; nothing
repeats until the bound is back below α_ref by 2 %. Without a threshold policy (or with an
invalidated one) there is no α check.
Drift
- PSI of the calibrated scores of the last 7 days against the policy's reference histogram
(20 bins):
Σ (a - e)·ln(a/e)over bin shares floored at 1e-4. Drift when PSI > 0.2. - Rolling ECE: adaptive (15 equal-mass bins) ECE of the calibrated audited fields of the last 30 days against the noise floor of T10 (the ECE a perfectly calibrated score would show with the same scores). Drift when ECE > noise floor + 0.01.
calibration_drift is deposited once until both are back.
Reliability diagram (CNF-02)
15 equal-mass bins over the audited fields (in the canonical order score, outcome) with two-sided
95 % Clopper-Pearson intervals; ECE and noise floor as in T10. Calibrated fields are used when
there are any (the calibrator is named with id, version and n), heuristic ones otherwise, and
calibrator_id restricts it to one calibrator.
Reasons of a null value
reason |
Meaning |
|---|---|
no_reviewed_fields |
no field of the window has a review decision (accuracy) |
no_audit_samples |
no audited and labelled document in the window |
insufficient_audit_labels |
fewer than 300 audited fields (silent error, rolling ECE, reliability ECE) |
no_reference_histogram |
no threshold policy with a reference histogram (PSI) |
insufficient_scores |
fewer than 300 calibrated scores in the last 7 days (PSI) |
scores_not_calibrated |
audited fields exist but none was calibrated (rolling ECE) |
no_threshold_policy |
the workflow has no threshold policy (no α check) |
A metric of an empty window is null, never 0, and alerts never evaluate a null.