Skip to content

Evaluation: extractit eval

extractit eval measures the pipeline on a dataset with one reproducible command: the metrics of research/09 §5 with bootstrap intervals, a paired comparison with significance, the baseline "PDF straight to the LLM" and a gate for CI.

extractit eval --dataset bench/datasets/invoices-mini --model oracle            # no LLM, deterministic
extractit eval --dataset bench/datasets/invoices-mini --model codex --baseline  # system vs baseline
extractit eval --dataset bench/datasets/invoices-mini --model oracle --fail-under 0.97
extractit eval compare reports/eval-A.json reports/eval-B.json

It writes eval-<run_id>.json (contract schemas/eval-report-1.0.json) and eval-<run_id>.md into --out (default reports/) and prints a summary. Exit codes: 0 done, 1 the metric is below --fail-under, 2 incorrect usage or a dataset whose files do not match its manifest.

Dataset format

A dataset is a directory:

manifest.json    name, version, schema{file, sha256}, generator, items[...], dataset_sha256
schema.json      the extraction schema (with field ids)
docs/<id>.pdf    the documents
gt/<id>.json     the ground truth of each one

items[] has {id, doc, gt, sha256_doc, sha256_gt, tags, origin, pages}. dataset_sha256 covers the ids, the document and ground-truth hashes and the schema hash: change any byte and extractit eval refuses to run (exit 2, naming the file). The contract is schemas/eval-manifest-1.0.json.

The ground truth of a document is {data, gt_field_status, raw_text, boxes}:

  • data: the expected object (typed values, ISO dates), like the extraction output's data.
  • gt_field_status[path]: labelled, absent (the document does not contain it: any value is a hallucination) or unlabelled (nobody labelled it: it never counts).
  • raw_text[path]: the value as printed. boxes[path]: [{page, bbox}], normalized to the page (0 <= x0 < x1 <= 1, origin top-left), one box per line.

The synthetic generator

bench/generators/invoices.py writes invoices in Spanish, English, French and German from three Jinja2 templates, rendered to PDF with WeasyPrint. Every value sits in <span data-gt="path">; the box of the value is read from WeasyPrint's box tree (one fragment per line) and its vertical extent is snapped to the glyph extent of the PDF, which is the rectangle a parser reports. A quarter of the invoices omits the purchase order and one more field (ground truth absent). The output depends only on the seed.

uv run python bench/generators/invoices.py --n 20 --seed 7 --out bench/datasets/invoices-mini
uv run python bench/generators/invoices.py --n 20 --seed 7 --degrade scan,rotate,photo --out ...

--degrade writes degraded/ with image-only PDFs (scan, rotation, photo) and the ground-truth boxes transformed by the same projective map as the image. They carry the tag requires_ocr: until OCR (T05) is integrated extractit eval omits them with a warning. WeasyPrint needs Pango and Cairo (Homebrew: brew install pango cairo libffi; the generator sets DYLD_FALLBACK_LIBRARY_PATH on macOS). Generation is a development task: evaluating a dataset needs no WeasyPrint.

Models

--model What it is
oracle Answers from the ground truth of the document in turn: it finds the printed text in the anchored markdown of the request and returns value, anchors and quote (or not_found for absent fields). It measures parse + verification + confidence without an LLM. Deterministic.
fake The oracle with a seeded noise (7 % wrong values, 4 % omissions, 30 % hallucinated absent fields; 14 %, 6 % and 50 % for the baseline). The noise of a field depends only on seed, document and path. It is a simulation, not a measurement of any model.
fake:<dir> Replays cassettes (T03).
codex[:model], gemini[:alias] Real backends (extractit doctor).

--tier fast|standard|agentic is passed to the pipeline. --repeat N runs N times and the report adds the variance of the accuracy; the metrics are those of the first run. --limit N takes the first N documents, --jobs 2 evaluates two at a time, --seed fixes the bootstrap and fake.

Metrics (research/09 §5)

All intervals are 95 % percentile intervals of a bootstrap that resamples documents (10 000 replicas by default; AUROC 1 000). The metrics section depends only on the dataset, the answers and the seed: repeating a run gives the same bytes. Wall-clock numbers are in operation.

  • Fields (§5.1). After normalizing (dates to ISO, numbers within ±0.005 or 1e-6 relative, strings NFKC with collapsed spaces and lower case unless normalize.case = "none"), each labelled or absent field is TP, FP+FN (wrong value), FN (omission), FP (hallucination, absent field) or TN. Accuracy (TP+TN)/N, precision, recall, F1; hallucination rate (hallucinated / absent fields); correct abstention (TN / nulls); coverage (non-null / N); ANLS for strings. Table cells are measured as tables, not here.
  • Perfect document. Every field and every table row right. Reported by page range (1–5, 6–15, 16–50, 51+) with Wilson intervals.
  • Tables (§5.2). Hungarian matching of rows by the x-dx.table.row_key (or the whole row): row recall and precision, duplicates, Kendall τ of the order and cell accuracy over matched rows.
  • Citations (§5.4). Over the literal fields (labelled, with a ground-truth box): page accuracy, IoU between the union of the cited regions and the box, share with IoU ≥ 0.5 and mean IoU, word precision (cited words that lie in the box), verified rate and verified-and-correct rate. A missing prediction scores IoU 0, so abstaining cannot raise the numbers.
  • Calibration (§5.5), informative. AUROC of correctness, AURC, AUGRC, precision at 80/90/95 % coverage, ECE with 15 equal-width and 15 equal-mass bins, Brier and skill score, log-loss, calibration slope (IRLS), reliability tables, error by tail band and the noise floor (the ECE a perfectly calibrated evaluator would show with the same N). The confidence of the pipeline is a heuristic: this describes it and gates nothing (calibration proper is T39).
  • Operation (§5.6). Latency p50/p95/p99, tokens, cost per document and per 1 000 pages. Cost is n/a for subscriptions, agents and simulated models; it is never estimated from list prices.

A document that fails counts as an error with all its fields as omissions and is listed in the report. With the baseline the report has both columns; the baseline sends the page images with the same field guide but the output schema {value, status, page} (plus bbox with --baseline-bbox), no anchors, no verifier; its IoU is n/a without boxes.

Comparing two runs

extractit eval compare A.json B.json [--bootstrap 10000] [--alpha 0.05] pairs the documents and reports A − B for field accuracy, F1, hallucination rate, citations with IoU ≥ 0.5 and perfect documents, each with a paired bootstrap interval (documents, then fields inside each), and an exact McNemar test on perfect document and its three partial versions (all fields, all tables, all citations) with Holm's correction over that family. If the reports cover different documents it compares the intersection and says so. --system-a/--system-b baseline compares the system with the baseline of one report.

The gate in CI

The job eval-gate of ci.yml runs the oracle over invoices-mini and fails below 97 % field accuracy (--fail-under 0.97), 97 % verified literal fields (--metric verified_rate) or 90 % of citations with IoU ≥ 0.5 (--metric iou_ge_05 --fail-under 0.90). --metric also accepts f1. A regression of the verifier makes the second one exit 1 (see tests/eval/test_cli_eval.py).

Limits

Degraded variants: their ground-truth boxes are in the frame of the image, while the parser puts its boxes in the frame it corrected (rotation, deskew), so citation IoU on rotate and photo variants is not comparable (0 % in the first measurement, bench/reports/eval-ev_8029*.md); field values are. Mapping the boxes through rotation_applied and deskew_angle_deg is open work. The synthetic ground truth overestimates real quality: results are reported by origin and tag, and DX-Bench (T52) adds public documents. Scans need OCR (T05). The simulated models measure the harness, not a model. Managed datasets and sealed splits are T38.