Parsing digital PDFs (tier Flash)
extractit.parse turns a digital PDF into a DocIR (words with exact boxes, lines, typed blocks in
reading order, tables with cells, stable anchors) and renders it as anchored markdown. No language model
is called: Heron (layout) and TableFormer (tables) run locally on CPU, words come from the PDF text layer.
from pathlib import Path
from extractit.parse import FileParseCache, ParseProfile, parse, to_anchored_markdown
doc = parse(Path("invoice.pdf"), ParseProfile(), cache=FileParseCache())
am = to_anchored_markdown(doc) # AnchoredMarkdown: text, anchors, page_map
print(am.text)
extractit parse invoice.pdf -o out/ # docir.json, document.md, anchored.json, pages/p001.png
extractit parse invoice.pdf --format blocks # only the blocks, as JSON, on stdout
extractit parse invoice.pdf --layout heuristic # no models: XY-cut order, every block a paragraph
extractit models pull parse # download the pinned weights (~0.5 GB, once)
Status lines (cache: hit (3 ms), page and word counts, warnings) go to stderr, so --format output on stdout
stays clean. Exit code 0 on success (also for a page without a text layer, which only warns needs_ocr),
2 for an unreadable PDF (corrupt, encrypted, empty, io: ParseError, no traceback, no document text).
Pipeline
render.page_geometries(pypdfium2): displayed size,/Rotateand CropBox of every page. The displayed page is the space of all normalized boxes: origin top-left,[0, 1].DoclingEngine.analyze(the only module importing Docling):DocumentConverterwith OCR off, TableFormer on,generate_parsed_pages. Words and lines are docling-parse's; regions are Heron's clusters; the reading order is the position of the Docling document item that best overlaps (IoU ≥ 0.5) each cluster; tables are TableFormer's cells with spans. Pages go through the models in batches (page_batch, default 10) to bound memory.build.build_document(pure Python overRawPage): each word goes to the region that holds its center (the highest score wins) and to its docling-parse line; lines are cut where they cross a region border; blocks follow the engine's order (XY-cut for regions it did not order; headers first, footers last); lines and blocks are numbered from 1 per page in reading order; headings getlevel,parent_idand the outline.markdown.to_anchored_markdown: the single renderer of anchored text.
Without weights, or with layout='heuristic', docling-parse runs alone (no PyTorch) and xycut.py orders the page:
every block is a paragraph, Block.confidences.layout is None and the document carries the warning
layout_model_unavailable. A page without text layer has route = none, no words, and the warning needs_ocr
(the OCR is T05).
Anchors and ids
| Anchor | Names | Numbering |
|---|---|---|
p{n}_l{m} |
a line | from 1, per page, in reading order |
p{n}_b{m} |
a block | from 1, per page, in reading order |
p{n}_t{k} |
a table | from 1, per page |
p{n}_t{k}_r{j} |
a table row (only in anchored text and evidence) | j is the 0-based row of TableCell.row |
Anchors and element ids (el_…, tbl_…) derive from the content: parsing the same file with the same profile
gives the same anchors and ids. ParsedDocument.id derives from parse_cache_key.
to_anchored_markdown (the only renderer)
to_anchored_markdown(doc, granularity='line'|'block'|'both', include_furniture=True) -> AnchoredMarkdown is the
only implementation of anchored text; the extraction harness (T06), the pipeline (T08) and agent mode (T10)
consume its result and never render text themselves. The default granularity is line until T08 applies ADR-019.
Page headers and footers are included by default because they may hold data of the issuer.
line (one anchor per line):
<!-- page 1 -->
## <a id="p1_l1"></a>Sales by region
<a id="p1_t1"></a>
<table>
<tr data-r="p1_t1_r0"><th colspan="2"><a id="p1_l2"></a>Quarter 1</th><th rowspan="2"><a id="p1_l3"></a>Total</th></tr>
<tr data-r="p1_t1_r1"><th><a id="p1_l4"></a>Units</th><th><a id="p1_l5"></a>Revenue</th></tr>
</table>
<a id="p1_l6"></a>The supplier delivers the goods listed below
<a id="p1_l7"></a>to the address shown in the header.
block (one anchor per block, no line anchors; the heading prefix stays on the same line):
## <a id="p1_b1"></a>Sales by region
<a id="p1_b3"></a>The supplier delivers the goods listed below
to the address shown in the header.
both (hierarchical: the block anchor, then the anchors of its lines):
<a id="p1_b3"></a><a id="p1_l6"></a>The supplier delivers the goods listed below
<a id="p1_l7"></a>to the address shown in the header.
Rules that hold for every granularity:
AnchorRef.char_start:char_endis the exact span an anchor names: the escaped text of a line, the text of a block from its first line, the<table>…</table>element of a table, the<tr>…</tr>element of a row.page_maptilestextwithout gaps or overlaps; each page starts with<!-- page N -->.- Escaping. Document text is escaped (
&→&,<→<,>→>), so a PDF that contains<a id="p1_l1"></a>produces<a id="p1_l1"></a>and cannot forge an anchor: the only<a id=…>tags and<tr data-r=…>rows in the output are the renderer's. Consumers decode entities (html.unescape) to compare withLine.text. - Headings:
#fortitle,##for a level-1section_header,###for level 2…; a list item gets-unless its text already starts with a bullet or a number. Hyphens at line end and ligatures are kept as they are. - A table cell holds the anchored lines of its words; every line has exactly one anchor, even if the engine put a line outside the cells (it is written after the table).
- Rendering is deterministic and has no dependence on the engine: it is a function of the
ParsedDocument.
Profile, cache and models
ParseProfile(layout, tableformer_mode, dpi, page_batch): everything that changes the result is hashed
(page_batch is not). parse_cache_key = sha256(file_sha256 ‖ profile_hash ‖ engine_versions_hash), where the
engine versions are those of docling, docling-parse, docling-ibm-models, pypdfium2, the pinned layout and TableFormer
revisions (parse/models.lock.toml), the layout mode, the TableFormer mode and DOCIR_VERSION. Computing the key
imports none of them, so a cache hit does not load Docling or PyTorch (provenance.cache_hit = true).
FileParseCache stores $DX_HOME/cache/parse/<key>.json.gz and, for the CLI, <key>/p001.png; a corrupt entry
behaves as a miss.
The CLI imports each subcommand lazily (cli/main.py), so a cached extractit parse loads only the parse path:
about 0.13 s of wall time for the whole process in a runtime install (uv sync --no-dev, Apple M5), of which the
cache hit itself is 2 to 5 ms. A development environment is slower (about 0.5 s) because the dev group pulls
rfc3987-syntax through cyclonedx-bom, and jsonschema then builds its grammar at import.
Weights: $DX_MODELS_DIR or ~/.extractit/models; extractit models pull parse downloads the revisions pinned in
models.lock.toml and records the sha256 of every file in parse-manifest.json (for offline mirrors);
extractit models status parse checks them without network. DX_PARSE_THREADS sets the model threads (default
min(4, cpus)).
Known behavior of the models
- Heron labels the headings of the synthetic fixtures
section_header(nottitle) and readsIssue date: …lines askey_value.buildderiveslevelfrom numbering or font size, so the largest heading is level 1. - TableFormer can put a row-header cell at the end of its row's token sequence (the grid says
col=3for a cell at the left);tables.normalized_cellsre-seats the single-column cells of a row in x order. TableCell.bboxis the box of the cell's text (cell matching), not the ruled grid cell.- T19 refines every table after the DocIR is built (words reassigned to cells, headers, confidence, borderless tables, logical tables across pages): see Tables and logical tables.
- Only
Block.confidences.layoutis filled here; per-page confidence and the parse confidence belong to T05.
Performance (hypothesis H2)
bench/parse_timing.py over 30 PDFs (73 pages) at 150 DPI and 4 threads, CPU only: median page 0.29 s (p95 0.55 s),
peak RSS 2.3 GB, 6 s to load the models once. Full report with hardware: bench/reports/parse-timing.md.
Attributions
Docling (MIT), docling-parse (MIT), docling-ibm-models (MIT) and the layout model Heron
(docling-project/docling-layout-heron, Apache-2.0) are IBM / Docling Project work. The TableFormer weights
(docling-project/docling-models) are licensed CDLA-Permissive-2.0 and need this attribution (also in NOTICE).
pypdfium2 (Apache-2.0 OR BSD-3-Clause) renders the pages. PyMuPDF and other copyleft PDF libraries are not used.