Skip to content

Normalization and citation verification

extractit.ground turns what the extraction harness proposes for a field (a FieldCandidate: value, raw_text, cited anchors and quote) into a typed value and auditable evidence pointing at the exact words of the document. It is deterministic, runs on CPU only and never calls a model (ADR-006, verifiable provenance and citations).

from extractit.ground import DocumentTextIndex, normalize, verify

index = DocumentTextIndex(doc)  # once per ParsedDocument, shared by all fields
nv = normalize(candidate.raw_text, field, locale_hint="es-ES")
result = verify(candidate, nv, doc, field, index=index)

result.verification  # "verified" | "partial" | "block_only" | "unverified" | "contradicted"
result.evidence[0].regions  # one box per line, what the viewer highlights
result.citation_corrected  # True when the value was found somewhere else than cited
result.conflict  # True when the same label shows another value

Normalization

normalize(raw, field, *, locale_hint=None, doc_style=None) -> NormalizedValue dispatches on the field type and returns value, ok, reason, canonical_text, candidates and the untouched raw_text. An ambiguous input is never guessed: ok = False, value = None, and every admissible reading is listed in candidates.

Type Examples Notes
number, integer 1.234,56 € → 1234.56 · 1,234.56 → 1234.56 · (120,00) → -120 · 1'234.56 → 1234.56 · 15 % → 15 Currency, codes, units and spaces (incl. U+00A0, U+202F) are dropped. Negative: -, U+2212, parentheses, trailing -.
date 03/04/2026, 3 de marzo de 2026, March 3, 2026, 3. März 2026, 20260304 Month names in es/en/fr/de/it/pt from Babel. Two-digit years pivot at 70.
datetime 2026-03-04T10:30:00Z, 3 de marzo de 2026, 14:05 ISO 8601 output, with offset when given.
boolean Sí, No, oui, ja, ☑, ☒, ✓, [x], Sí ☒ No ☐ A form line takes the label next to the only checked box.
enum Pagado → paid rapidfuzz WRatio ≥ 85 against options and enum_descriptions (each synonym of a description is scored).

How a separator is read:

  • with both . and ,, the last one is the decimal separator;
  • one separator repeated (1.234.567) groups thousands;
  • a single separator followed by exactly three digits (1.234) is ambiguous, as is a lone decimal comma with no hint at all (1,5). They are resolved, in this order, by x-dx.normalize.decimal_separator, the locale hint (es, de, it, pt, fr, nl, pl, ru… use a decimal comma; en a point) or infer_number_style(doc), which counts the unambiguous numbers of the document (1.234,56, 12,50…).

Dates a/b/y with both parts ≤ 12 are ambiguous unless x-dx.normalize.date_order or the locale hint (Babel's short date pattern: es-ES → DMY, en-US → MDY) decides. Impossible dates (31/02/2026) fail with invalid_date.

Schema options read from x-dx.normalize: decimal_separator, date_order, percent (points, default, or fraction: 15 % → 0.15), enum_threshold (0–100, default 85), case, strip.

Failure reasons: ambiguous, invalid_date, not_numeric, unsupported_format, enum_no_match, enum_ambiguous (two options within 3 points).

The text index

DocumentTextIndex(doc) is built once per document (well under 20 ms for three pages) and reused by every field. Per line it keeps the text, a folded copy of the same length (lower case, no diacritics, unified spaces, dashes and quotes, so offsets in one are offsets in the other), a character → word map, and the typed tokens: every number with all its readings and every date with all its readings. Across the document it keeps inverted indexes of words, number values and dates, the anchor → lines map (a block is its lines; a table row is the lines of its cells) and the word → table cell map. It is designed to be shared with the pipeline (T08) and the long-document consolidation (T23).

Matching methods

match.method says how the value was found:

Method Meaning Example
exact raw_text appears literally, with word boundaries (a number never matches the head or tail of a longer one) 4.594,62 € in Total: 4.594,62 €
normalized the folded spelling or the canonical value appears; dates compare by parsed value ACME corporation ↔ Acme Corporation; 2026-03-03 ↔ 3 de marzo de 2026
numeric a number token has the same Decimal value at the precision shown 4594.62 ↔ 4.594,62; -120 ↔ (120,00)
fuzzy strings only: partial_ratio_alignment ≥ 85 (match.score in [0, 1]) and at least 80 % of the value's characters read as whole words in the span (a word up to 1 edit, 2 from 8 letters, or split in two) Acme Corporatlon ↔ Acme Corporation; not CALLED read_file ↔ Call read_file on …
semantic inferred value with no literal (block_only) is_signed = true on a signature block
none nothing matched (unverified, contradicted)

Numbers and dates are never compared fuzzily: 980,01 against 980,00 is a contradiction, not a near match.

Search order

  1. Resolve the cited anchors to lines (block → its lines; table row → the lines of its cells). A missing anchor is noted (missing_anchor:p9_l1).
  2. Look in the cited lines: exact, then normalized, then numeric/date tokens. Consecutive cited lines are searched as one text, so a value split over two lines is one match.
  3. Widen to the ±1 neighbouring lines on the same page (label above the value, split values).
  4. Strings only: fuzzy match in the cited and neighbouring lines → partial.
  5. Search the document through the inverted indexes.
  6. Nothing found: decide between contradicted, block_only and unverified (below).

When several occurrences compete, they are ordered by: inside the cited lines; inside the model's quote; similarity of the occurrence's label (text before it on the line, the line above, or the column header and row name in a table) to the field's name, title and description; on a cited page; distance in lines to the cited anchor; document order. A neighbour hit (step 3) competes with the document hits (step 5) under this order, so a repeated short value (1) next to the cited line does not beat the one next to the field's label.

The five states

State When Evidence
verified The value (raw, normalized or numeric) is in the cited region, or was found elsewhere and the citation corrected. Exact words, char_span, one region per line, table_cell + table_ref for a cell.
partial Only a fuzzy string match (≥ 0.85). Words of the aligned span, match.score.
block_only Boolean or enum without a literal, with an existing anchor (is_signed = true). An enum needs a phrase of the cited text that the normalizer reads as the option (abonado → paid); otherwise it is unverified. The quote's words if the quote is found, else the cited lines; match.method = semantic.
contradicted The cited region holds another value of the same type (another amount, another date, the opposite yes/no, another enum option). Points to the other value.
unverified Anything else, including non-existent anchors with no support in the document. Cited page and anchor if they exist, no words.

Example: the model cites p1_l6 for total = 999,00 €, and that line reads Importe total: 980,00 €. The value is nowhere in the document, so the result is contradicted and the evidence highlights 980,00.

Citation correction

If the value is not in the cited lines but is found in a neighbouring line or elsewhere in the document, the occurrence found becomes the primary evidence with citation_corrected = true (on the evidence and on the result); the other occurrences are added as supporting. This is the adjustment of rule 5 of research/06 §7.1 recorded in ADR-006 (v2, T07): the evidence must point to where the value really is.

cited p1_l1 "Factura F-1"            → value 812,40 not there
found p2_l2 "Base imponible: 812,40 €" → primary, verified, citation_corrected = true

Supporting and contradicting evidence

  • supporting: other occurrences of the same value (at most 5), e.g. a total repeated in the header and the footer.
  • contradicting: lines with the same label (similarity ≥ 0.9) and another value of the same type (at most 3), e.g. Total: 1.250,00 € in the header and Total: 1.320,00 € in the footer. Any contradicting evidence sets conflict = true. Labels are compared for numbers and dates.

Instructions addressed to the model

A line that is an instruction to the model (a prompt injection planted in the document) is never evidence: not as the primary match, not as supporting or contradicting evidence and not as the region of a block_only verdict (extractit.ground.instructions). An instruction that quotes the value it asks for would otherwise ground itself. The markers are categories, not known attacks: chat-template and role tokens (<|im_start|>, [INST], "role": "system"), a verb that discards instructions next to them (ignore all previous instructions, ignora las instrucciones anteriores), text that addresses the model (Dear model, Note to the AI reading this, you are now in … mode) and an imperative aimed at a schema key (set supplier_name to …). Markers are searched in the text of each layout block; the lines the marker covers and the lines that continue its sentence are excluded, never the lines before it. The result carries the note instruction_text_is_not_evidence; an honest value cited to such a line is still found where the document states it (citation corrected).

Determinism and performance

Same inputs, same evidence and same order: evidence ids are deterministic TypeIDs derived from the document, the field path, the role and the span. On the synthetic fixtures the index builds in < 1 ms and 40 verifications take ≈ 5 ms of CPU (bench/reports/grounding-synthetic.md; thresholds 20 ms and 50 ms).

Limits

  • The thresholds are measured on synthetic DocIR from DocIRBuilder. Real parsers may split words differently ($4,594.62 as one word, split digits); this is measured in T10.
  • Very short values (1, 0) with many occurrences rely on the cited anchor and the label; ambiguity remaining there is a confidence signal for T08.
  • Free text without a literal stays unverified (only booleans and enums can be block_only).
  • The instruction markers are a lexicon (English, Spanish, German, French, Italian, Portuguese): an instruction phrased without any of them is not detected, and then only the other rules (value not in the document, validation rules, confidence band) stop it. Hidden text (white on white, off page) is not told apart from visible text: DocIR does not carry visibility.