Route C: VLM per page aligned to words (tier Standard)
extractit parse --tier standard adds a third way of reading a page. Route A reads the PDF text layer,
route B reads the page with OCR; route C sends the page image to a document VLM through the model
gateway and aligns its text to the words of route A or B. The VLM gives the best reading order,
tables and the text of hard scans; the words of A/B give the geometry. The VLM is never asked for
boxes: every line keeps the boxes of its anchor words, so citations stay word-level.
extractit parse invoice.pdf --tier standard # the complex pages (T21 router)
extractit parse scan.pdf --tier standard --vlm-model gemini-38-flash
extractit parse report.pdf --tier standard --vlm-pages 1,3-4 # only these pages call the VLM
extractit parse invoice.pdf --tier standard --prompt-hint "Factura española, importes con coma decimal"
Since T21 the tier standard goes through the page router: only pages with a hard
flag (photo, low DPI, script without OCR model, low OCR confidence) or complexity >= 0.40 call the
VLM; --vlm-pages forces route C on the listed pages and keeps the others on A/B.
Status lines go to stderr (vlm: page 1 ok (codex-gpt56-sol) precision 1.00 recall 1.00, or
vlm: page 2 failed: truncated). Pages outside --vlm-pages make no model call.
Engines (engines.toml)
The engine is configuration, never code: --vlm-model <id> or DX_VLM_MODEL=<id> picks an entry
of src/extractit/parse/vlm/engines.toml (bench/vlm/engines.toml is the same file).
DX_VLM_ENGINES points to another catalogue.
| Entry | Model | Needs | Notes |
|---|---|---|---|
codex-gpt56-sol (default) |
codex:gpt-5.6-sol |
codex login (ChatGPT plan) |
measured on align-v0 |
gemini-38-flash |
gemini:gemini-3.8-flash |
DX_GEMINI_API_KEY |
estimated 4.73 USD / 1,000 pages |
paddleocr-vl |
compat:PaddlePaddle/PaddleOCR-VL |
DX_COMPAT_BASE_URL (vLLM, GPU) |
fixed task prompt OCR:, plain text |
qwen3-vl-8b |
compat:Qwen/Qwen3-VL-8B-Instruct |
DX_COMPAT_BASE_URL (vLLM, GPU) |
JSON by guided decoding |
openai-gpt6-luna |
openai:gpt-6-luna |
DX_OPENAI_API_KEY |
estimated 0.63 USD / 1,000 pages |
anthropic-haiku55 |
anthropic:claude-haiku-5-5 |
DX_ANTHROPIC_API_KEY |
role crops (re-reads of T21/T56) |
Each entry declares model (a pinned provider:snapshot route, never latest), role (page
or crops), dpi and max_px of the page image, max_tokens per page, concurrency,
free_prompt, json_mode, air_gapped, regions (where the provider processes the page) and
cost_per_1k_pages_usd with its basis (estimated, measured, subscription,
self-hosted). EngineCapabilities of every engine says word_boxes=false (geometry comes
from the anchor words), whether it is air-gapped, takes a free prompt and what it costs.
PaddleOCR-VL or Qwen-VL on vLLM. Serve the model with vLLM (vllm serve PaddlePaddle/PaddleOCR-VL)
and set DX_COMPAT_BASE_URL=http://<host>:8000/v1 (and DX_COMPAT_API_KEY if the server wants
one). PaddleOCR-VL has no JSON mode and a fixed task prompt: the engine sends the image with
OCR: and wraps the plain-text answer into the page schema. GPU deployment and mirrored weights
are part of the air-gapped profile (T54).
Policy. The model policy of the gateway (DX_MODEL_POLICY: air_gapped, eu_only,
allowed_providers…) is applied before any call; an engine it rejects leaves the page with its
A/B result and the warning no_eligible_engine.
Concurrency and cost. Inside one document at most DX_VLM_CONCURRENCY pages (4, capped by
the entry) are read at once. A PageBudget (vlm_max_usd_per_page, 0.05 USD) refuses a call that
would exceed the cap (reread_budget_exhausted). Every call is recorded in the usage ledger
(tokens, cost, latency) and summed in provenance.cost; provenance.model_calls lists the calls
and provenance.engines the engine with its snapshot.
What the alignment produces
The VLM markdown is split into blocks (paragraphs, headings, list items, tables with their cells
flattened by rows, formulas), paired with the layout blocks of the page by a monotone dynamic
programme (cost 1 − Jaccard of word 4-grams plus a length prior, skip 0.8; pairs above 0.7 are
not trusted), then aligned word by word with rapidfuzz Levenshtein opcodes over normalized
tokens (NFKC, ligatures, quotes, dashes, case, end-of-line hyphens; digits compared as digits).
Tokens the monotone pass leaves unpaired are matched exactly afterwards, so a VLM that reads a
table row by row while the layout read it by columns still aligns.
Every aligned line keeps its anchor words (a word never moves to another line) and gets, in
Line.alignment (DocIR 1.2):
| Field | Meaning |
|---|---|
provenance |
how the text is grounded (ladder below) |
alignment_score |
share of the line's VLM characters aligned to an anchor word |
flags |
divergences (below) |
polygon |
envelope of the aligned words of the line |
spans |
text[start:end] → word (equal, replace, conflict, insert, kept) |
alternatives |
the VLM reading of a word whose anchor text was kept |
Lines and blocks are rewritten in the VLM reading order and their anchors renumbered
(p1_l1…, p1_b1…) deterministically. PageQuality gets alignment_precision,
alignment_recall and alignment_score.
Provenance ladder
exact_text_layer: a digital page; the text is the text layer (the VLM gives order and structure; a CER above 0.10 against the text layer addstext_layer_disagreement).ocr_aligned: VLM text aligned to OCR words.vlm_block_only: VLM text no word explains; it keeps only the box of its layout block.interpolated: VLM text with a box interpolated from its neighbours.none: no geometry at all.
Flags
| Flag | When |
|---|---|
interpolated |
the VLM wrote tokens no anchor word explains (table markup and formulas excepted) |
possible_hallucination |
8 or more interpolated tokens in a row |
omission |
3 or more anchor words (or 15 characters) in a row that the VLM left out |
numeric_conflict |
a substituted token whose digits differ: the anchor text is kept, the VLM reading goes to alternatives |
text_layer_disagreement |
digital page, VLM text vs text layer CER > 0.10 |
Citations
ParsedDocument.words_for_span(element_id, (start, end)) turns characters of a line (or block)
text into anchor words. The citation verifier (T07) uses the alignment spans of aligned lines, so
a value read from VLM text is cited on the words that hold it.
When a page fails
Guards check every answer before aligning: repetition_loop (zlib ratio < 0.2 over 400
characters, or a 4-gram repeated 8 times), truncated (max_tokens reached), refusal, empty,
length_ratio (VLM/anchor characters outside [0.5, 2.0]), plus no_eligible_engine (policy),
reread_budget_exhausted and engine_error. The page keeps its route A/B result, gets the
warning vlm_failed and is listed in quality.pages_flagged; retries and engine fallback are the
router's job (T21). A failed document is not cached, so the next run tries again.
Prompt hint
--prompt-hint (≤ 500 characters, else invalid-prompt-hint, HTTP 422) describes the document.
Control characters and < > & are escaped and the hint travels in a <hint>…</hint> block that
the versioned system prompt (PROMPT_VERSION) declares as data, never instructions; the page
image is declared data too. The hint is part of parse_cache_key; only its length and sha256
are logged or stored. Engines with a fixed task prompt (free_prompt = false) do not receive it
and the document gets the warning prompt_hint_ignored.
Crop primitives (T21, T56)
render_crop(page_image, bbox, scale=2.0, max_px=2048) cuts a box and scales it (400 DPI from a
200 DPI render), lowering the scale so no side exceeds max_px. CropReader(gateway).read(crop,
Schema, purpose) returns a validated Pydantic object (the gateway repairs an invalid answer at
most twice; purpose goes to the ledger). Both declare the image as data.
Measuring engines
uv run python bench/vlm/eval_align_v0.py --write-report # bench/reports/align-v0.md
uv run python bench/vlm/run_matrix.py --write-report # bench/reports/vlm-engines.md
DX_LIVE=1 uv run python bench/vlm/record_align_v0.py --engine <id> # record a new engine (real calls)
bench/datasets/align-v0 holds 40 one-page invoices from the generator of T10 (20 digital, 20
scanned at 200 DPI with light degradation) with ground-truth values and boxes, and the frozen
inputs of the aligner (frozen.json): the page image the VLM read and the DocIR of route A/B,
each with its sha256. PDFium renders and extracts text differently on macOS and Linux, so the
evaluation and the cassettes (tests/cassettes/vlm/) read these frozen inputs; the cassette key
is the hash of the request, which contains the frozen image and never the local render.
DX_VLM_IMAGES=<dir> makes route C read frozen page images (<dir>/<pdf sha256>/p001.jpg) in
any parse, which is how the CLI tests replay cassettes on every platform.