Field sources: lookup, derived, metadata and packets
Not every value is in the text of the document. x-dx.source declares where each field comes from
and one mechanism resolves them, after extraction and verification and before the rules engine
(dialect extraction-schema-meta-1.2, output output-1.2, DocIR 1.2).
{
"supplier_name": { "type": "string" }, // document (default)
"supplier_code": { "type": "string", "x-dx": { "source": { // lookup
"kind": "lookup", "table_id": "suppliers", "match_on": ["supplier_name"], "returns": "code" } } },
"total": { "type": "number", "x-dx": { "source": { // derived (CEL, no model)
"kind": "derived", "expression": "sum(data.line_items.map(r, r.amount))" } } },
"customer_number": { "type": "string", "x-dx": { "source": { // package metadata
"kind": "metadata", "path": "customer.number" } } }
}
The legacy strings (source: "lookup" | "lookup_if_missing" | "derived" | "metadata" with lookup,
expression or metadata next to them) are still accepted and normalize to the same object
(FieldDef.source_spec); old snapshots of the contracts stay in schemas/. A source kind of another
package (visual, T56) registers through register_source_kind and is ignored here.
Statuses and evidence
| Source | status |
Evidence kind |
Extra evidence fields |
|---|---|---|---|
| lookup | looked_up |
lookup_row |
lookup_ref{table_id, table_version, row_key, matched_on, match_score}, match.method exact/fuzzy |
| derived | derived |
derived |
derivation{expression, inputs} |
| metadata | from_metadata |
metadata |
metadata_path |
Each resolved value also carries the evidence of its inputs as supporting. Confidence is
heuristic: lookup = match_score × the weakest input, derived = the weakest input, metadata = the
verifier's. Verification: verified for an exact lookup row and for metadata, partial for a row
chosen by the model or by the deterministic rule, the worst of its inputs for a derived value.
Order
Sources resolve in dependency order: metadata, then lookups, then derived fields, each after what
it reads (a lookup depends on its match_on fields, a derived field on the fields its expression
mentions). A cycle is rejected with source-cycle and the path. x-dx.source is checked when a
schema is loaded: tables, columns, expressions (≤ 1 000 characters and ≤ 200 nodes, references that
exist) and cycles give 422 invalid-field-source with the JSON pointer of the offending key.
extractit.extract.sources.validate_field_sources(schema) returns every problem.
Lookup
{ "kind": "lookup", "table_id": "suppliers", "table_version": "latest",
"match_on": ["supplier_name", {"field": "supplier_city", "column": "city"}],
"returns": "code", "mode": "always", "k": 5, "threshold": 0.80,
"require_all": true, "select": "llm" }
match_onlists fields of the schema (byfield_id, path or name); the table column defaults to the field's name, or is given with{field, column}.mode: always– the field is not in the extraction prompt;if_missing– the model is asked first and the table is consulted only when the field did not end upextractedwith verificationpartialor better.- A table is a CSV (UTF-8,
,;or tab; BOM allowed) or Parquet. Versions are immutable and named by the sha256 of their normalized content: the same content is the same version,latestresolves to a number that stays in the evidence, and a run with a pinnedtable_versionreproduces after a newer one is uploaded. Limits: 500 000 rows and 100 MB (DX_LOOKUP_MAX_ROWS,DX_LOOKUP_MAX_BYTES). - Matching: names are normalized (no accents, lower case, punctuation and legal-form suffixes removed:
suffixes.toml,DX_LOOKUP_SUFFIXES), candidates come from rare shared tokens and are scored with0.5·JaroWinkler + 0.5·token_sort/100per column. Only a row where every queried column reachesthresholdis a candidate for acceptance: there is never a partial match. - Selection: one normalized-equal row is accepted without a model (
exact). Otherwise a low-tier model (purpose='lookup') answers with the key of one candidate ornull(the schema is an enum of the candidate keys); a key outside the enum is retried once and thennullwithlookup_invalid_choice.select: deterministic(and a model failure) accepts only a top candidate with score ≥ 0.95 and a margin ≥ 0.10 over the second. At most one model call per query, and an in-process cache makes 100 documents of the same supplier cost one. - Table cells and document values reach the model as data inside a delimiter derived from the content, never as instructions.
Where tables live: $DX_LOOKUP_DIR (manifest.json + v<n>.parquet) for the library and the CLI, or
the encrypted per-tenant blob store and RLS tables lookup_tables / lookup_table_versions on the
server.
extractit lookup add suppliers.csv --key code --name suppliers # version 1
extractit lookup add suppliers.csv --table suppliers # same content: unchanged
extractit lookup list && extractit lookup versions suppliers
extractit lookup match suppliers supplier_name="Acme Industrial Supplies SL" # scored candidates, no model
DX_LOOKUP_DIR=~/.extractit/lookup extractit note.pdf --schema note.schema.json
API: POST/GET /v1/lookup-tables, PATCH /v1/lookup-tables/{id} (If-Match),
POST/GET /v1/lookup-tables/{id}/versions, GET …/versions/{n}?preview=20 (latest allowed) and
POST …/versions/{n}:match (candidates with scores, no model). Errors: lookup-too-large 413,
lookup-duplicate-keys 422, invalid-lookup-config 422, lookup-version-not-found 404. A key with the
read scope can GET and :match but not POST.
Evaluation: bench/datasets/lookup-v0 and bench/reports/lookup-v0.md.
Derived fields
kind: derived evaluates a CEL expression over data with the evaluator of the rules engine (T09:
num, sum, abs). References are stored as field_id, so renaming a field does not break the
expression. No model is called. An input without a value (null, not_found, abstained) gives
null with status not_found and the warning derived_missing_input; a result that does not fit the
field type gives derived_type_error; a runtime error derived_expression_error. Sums are exact
(Decimal, 6 decimals: 0.1 added ten times is 1.0) and JSON integers of number fields read as
doubles, so data.total * 2.0 works. Derived fields inside arrays of objects are not supported.
Recomputing after a human correction in the viewer is T32.
Metadata, notes and packets
extractit a.pdf b.pdf c.pdf --metadata meta.json --notes notes.txt (and the packet endpoints of T17)
send several files and what travels with them.
- One document. The files' DocIR (parsed and cached one by one) become a single document with
global pages: anchors
p{packet_page}_l{n},Page.file_id,Page.file_page,Page.packet_pageand the same three fields in every evidence. One file is the individual parse; adding a file parses only that file. Any tier uses the packet view. - Metadata (
sanitize_metadata): JSON ≤ 64 KiB (metadata-too-large), keys^[A-Za-z0-9_.-]{1,64}$, depth ≤ 4, ≤ 100 leaves (metadata_entries_dropped), ≤ 500 characters per value in the prompt (metadata_value_truncated), 8 000 characters in total, control characters removed and<,>,&escaped so no value can close the block or forge an anchor. The prompt gets<metadata>with one line per leaf:<a id="metadata.customer.number"></a>customer.number: ACME-77; notes (≤ 8 KiB) becomenotes.l1…. Everything inside is data, never instructions. - Metadata that informs. A
documentfield whose citation is a metadata anchor getskind='metadata'evidence,verifiedif the value matches the leaf andcontradictedif it differs. - Metadata that sets.
kind: metadatatakes the full value from the JSON (not the prompt's truncation), coerces it with the normalizers of T07 and needs no model call. A missing path isnot_foundwithmetadata_path_missing. Withcross_check: truethe model also reads the field: different values givestatus='conflict'with the metadata value,contradictingevidence and the document's value inalternatives; equal values addsupportingevidence.