Normalization and citation verification
extractit.ground turns what the extraction harness proposes for a field (a
FieldCandidate: value, raw_text, cited anchors and quote) into a typed value and
auditable evidence pointing at the exact words of the document. It is deterministic, runs on
CPU only and never calls a model (ADR-006, verifiable provenance and citations).
from extractit.ground import DocumentTextIndex, normalize, verify
index = DocumentTextIndex(doc) # once per ParsedDocument, shared by all fields
nv = normalize(candidate.raw_text, field, locale_hint="es-ES")
result = verify(candidate, nv, doc, field, index=index)
result.verification # "verified" | "partial" | "block_only" | "unverified" | "contradicted"
result.evidence[0].regions # one box per line, what the viewer highlights
result.citation_corrected # True when the value was found somewhere else than cited
result.conflict # True when the same label shows another value
Normalization
normalize(raw, field, *, locale_hint=None, doc_style=None) -> NormalizedValue dispatches on the
field type and returns value, ok, reason, canonical_text, candidates and the untouched
raw_text. An ambiguous input is never guessed: ok = False, value = None, and every
admissible reading is listed in candidates.
| Type | Examples | Notes |
|---|---|---|
| number, integer | 1.234,56 € → 1234.56 · 1,234.56 → 1234.56 · (120,00) → -120 · 1'234.56 → 1234.56 · 15 % → 15 |
Currency, codes, units and spaces (incl. U+00A0, U+202F) are dropped. Negative: -, U+2212, parentheses, trailing -. |
| date | 03/04/2026, 3 de marzo de 2026, March 3, 2026, 3. März 2026, 20260304 |
Month names in es/en/fr/de/it/pt from Babel. Two-digit years pivot at 70. |
| datetime | 2026-03-04T10:30:00Z, 3 de marzo de 2026, 14:05 |
ISO 8601 output, with offset when given. |
| boolean | Sí, No, oui, ja, ☑, ☒, ✓, [x], Sí ☒ No ☐ |
A form line takes the label next to the only checked box. |
| enum | Pagado → paid |
rapidfuzz WRatio ≥ 85 against options and enum_descriptions (each synonym of a description is scored). |
How a separator is read:
- with both
.and,, the last one is the decimal separator; - one separator repeated (
1.234.567) groups thousands; - a single separator followed by exactly three digits (
1.234) is ambiguous, as is a lone decimal comma with no hint at all (1,5). They are resolved, in this order, byx-dx.normalize.decimal_separator, the locale hint (es, de, it, pt, fr, nl, pl, ru… use a decimal comma; en a point) orinfer_number_style(doc), which counts the unambiguous numbers of the document (1.234,56,12,50…).
Dates a/b/y with both parts ≤ 12 are ambiguous unless x-dx.normalize.date_order or the locale
hint (Babel's short date pattern: es-ES → DMY, en-US → MDY) decides. Impossible dates
(31/02/2026) fail with invalid_date.
Schema options read from x-dx.normalize: decimal_separator, date_order, percent
(points, default, or fraction: 15 % → 0.15), enum_threshold (0–100, default 85), case,
strip.
Failure reasons: ambiguous, invalid_date, not_numeric, unsupported_format,
enum_no_match, enum_ambiguous (two options within 3 points).
The text index
DocumentTextIndex(doc) is built once per document (well under 20 ms for three pages) and
reused by every field. Per line it keeps the text, a folded copy of the same length (lower
case, no diacritics, unified spaces, dashes and quotes, so offsets in one are offsets in the
other), a character → word map, and the typed tokens: every number with all its readings and
every date with all its readings. Across the document it keeps inverted indexes of words, number
values and dates, the anchor → lines map (a block is its lines; a table row is the lines of its
cells) and the word → table cell map. It is designed to be shared with the pipeline (T08) and the
long-document consolidation (T23).
Matching methods
match.method says how the value was found:
| Method | Meaning | Example |
|---|---|---|
exact |
raw_text appears literally, with word boundaries (a number never matches the head or tail of a longer one) |
4.594,62 € in Total: 4.594,62 € |
normalized |
the folded spelling or the canonical value appears; dates compare by parsed value | ACME corporation ↔ Acme Corporation; 2026-03-03 ↔ 3 de marzo de 2026 |
numeric |
a number token has the same Decimal value at the precision shown |
4594.62 ↔ 4.594,62; -120 ↔ (120,00) |
fuzzy |
strings only: partial_ratio_alignment ≥ 85 (match.score in [0, 1]) and at least 80 % of the value's characters read as whole words in the span (a word up to 1 edit, 2 from 8 letters, or split in two) |
Acme Corporatlon ↔ Acme Corporation; not CALLED read_file ↔ Call read_file on … |
semantic |
inferred value with no literal (block_only) |
is_signed = true on a signature block |
none |
nothing matched (unverified, contradicted) |
Numbers and dates are never compared fuzzily: 980,01 against 980,00 is a contradiction,
not a near match.
Search order
- Resolve the cited anchors to lines (block → its lines; table row → the lines of its cells).
A missing anchor is noted (
missing_anchor:p9_l1). - Look in the cited lines:
exact, thennormalized, then numeric/date tokens. Consecutive cited lines are searched as one text, so a value split over two lines is one match. - Widen to the ±1 neighbouring lines on the same page (label above the value, split values).
- Strings only: fuzzy match in the cited and neighbouring lines →
partial. - Search the document through the inverted indexes.
- Nothing found: decide between
contradicted,block_onlyandunverified(below).
When several occurrences compete, they are ordered by: inside the cited lines; inside the
model's quote; similarity of the occurrence's label (text before it on the line, the line above,
or the column header and row name in a table) to the field's name, title and description; on a
cited page; distance in lines to the cited anchor; document order. A neighbour hit (step 3)
competes with the document hits (step 5) under this order, so a repeated short value (1) next
to the cited line does not beat the one next to the field's label.
The five states
| State | When | Evidence |
|---|---|---|
verified |
The value (raw, normalized or numeric) is in the cited region, or was found elsewhere and the citation corrected. | Exact words, char_span, one region per line, table_cell + table_ref for a cell. |
partial |
Only a fuzzy string match (≥ 0.85). | Words of the aligned span, match.score. |
block_only |
Boolean or enum without a literal, with an existing anchor (is_signed = true). An enum needs a phrase of the cited text that the normalizer reads as the option (abonado → paid); otherwise it is unverified. |
The quote's words if the quote is found, else the cited lines; match.method = semantic. |
contradicted |
The cited region holds another value of the same type (another amount, another date, the opposite yes/no, another enum option). | Points to the other value. |
unverified |
Anything else, including non-existent anchors with no support in the document. | Cited page and anchor if they exist, no words. |
Example: the model cites p1_l6 for total = 999,00 €, and that line reads
Importe total: 980,00 €. The value is nowhere in the document, so the result is contradicted
and the evidence highlights 980,00.
Citation correction
If the value is not in the cited lines but is found in a neighbouring line or elsewhere in the
document, the occurrence found becomes the primary evidence with citation_corrected = true
(on the evidence and on the result); the other occurrences are added as supporting. This is
the adjustment of rule 5 of research/06 §7.1 recorded in ADR-006 (v2, T07): the evidence must
point to where the value really is.
cited p1_l1 "Factura F-1" → value 812,40 not there
found p2_l2 "Base imponible: 812,40 €" → primary, verified, citation_corrected = true
Supporting and contradicting evidence
supporting: other occurrences of the same value (at most 5), e.g. a total repeated in the header and the footer.contradicting: lines with the same label (similarity ≥ 0.9) and another value of the same type (at most 3), e.g.Total: 1.250,00 €in the header andTotal: 1.320,00 €in the footer. Any contradicting evidence setsconflict = true. Labels are compared for numbers and dates.
Instructions addressed to the model
A line that is an instruction to the model (a prompt injection planted in the document) is never
evidence: not as the primary match, not as supporting or contradicting evidence and not as the
region of a block_only verdict (extractit.ground.instructions). An instruction that quotes
the value it asks for would otherwise ground itself. The markers are categories, not known
attacks: chat-template and role tokens (<|im_start|>, [INST], "role": "system"), a verb that
discards instructions next to them (ignore all previous instructions, ignora las instrucciones
anteriores), text that addresses the model (Dear model, Note to the AI reading this,
you are now in … mode) and an imperative aimed at a schema key (set supplier_name to …).
Markers are searched in the text of each layout block; the lines the marker covers and the lines
that continue its sentence are excluded, never the lines before it. The result carries the note
instruction_text_is_not_evidence; an honest value cited to such a line is still found where the
document states it (citation corrected).
Determinism and performance
Same inputs, same evidence and same order: evidence ids are deterministic TypeIDs derived from the
document, the field path, the role and the span. On the synthetic fixtures the index builds in
< 1 ms and 40 verifications take ≈ 5 ms of CPU (bench/reports/grounding-synthetic.md; thresholds
20 ms and 50 ms).
Limits
- The thresholds are measured on synthetic DocIR from
DocIRBuilder. Real parsers may split words differently ($4,594.62as one word, split digits); this is measured in T10. - Very short values (
1,0) with many occurrences rely on the cited anchor and the label; ambiguity remaining there is a confidence signal for T08. - Free text without a literal stays
unverified(only booleans and enums can beblock_only). - The instruction markers are a lexicon (English, Spanish, German, French, Italian, Portuguese): an instruction phrased without any of them is not detected, and then only the other rules (value not in the document, validation rules, confidence band) stop it. Hidden text (white on white, off page) is not told apart from visible text: DocIR does not carry visibility.