Skip to content

Field sources: lookup, derived, metadata and packets

Not every value is in the text of the document. x-dx.source declares where each field comes from and one mechanism resolves them, after extraction and verification and before the rules engine (dialect extraction-schema-meta-1.2, output output-1.2, DocIR 1.2).

{
  "supplier_name": { "type": "string" },                       // document (default)
  "supplier_code": { "type": "string", "x-dx": { "source": {   // lookup
      "kind": "lookup", "table_id": "suppliers", "match_on": ["supplier_name"], "returns": "code" } } },
  "total": { "type": "number", "x-dx": { "source": {           // derived (CEL, no model)
      "kind": "derived", "expression": "sum(data.line_items.map(r, r.amount))" } } },
  "customer_number": { "type": "string", "x-dx": { "source": { // package metadata
      "kind": "metadata", "path": "customer.number" } } }
}

The legacy strings (source: "lookup" | "lookup_if_missing" | "derived" | "metadata" with lookup, expression or metadata next to them) are still accepted and normalize to the same object (FieldDef.source_spec); old snapshots of the contracts stay in schemas/. A source kind of another package (visual, T56) registers through register_source_kind and is ignored here.

Statuses and evidence

Source status Evidence kind Extra evidence fields
lookup looked_up lookup_row lookup_ref{table_id, table_version, row_key, matched_on, match_score}, match.method exact/fuzzy
derived derived derived derivation{expression, inputs}
metadata from_metadata metadata metadata_path

Each resolved value also carries the evidence of its inputs as supporting. Confidence is heuristic: lookup = match_score × the weakest input, derived = the weakest input, metadata = the verifier's. Verification: verified for an exact lookup row and for metadata, partial for a row chosen by the model or by the deterministic rule, the worst of its inputs for a derived value.

Order

Sources resolve in dependency order: metadata, then lookups, then derived fields, each after what it reads (a lookup depends on its match_on fields, a derived field on the fields its expression mentions). A cycle is rejected with source-cycle and the path. x-dx.source is checked when a schema is loaded: tables, columns, expressions (≤ 1 000 characters and ≤ 200 nodes, references that exist) and cycles give 422 invalid-field-source with the JSON pointer of the offending key. extractit.extract.sources.validate_field_sources(schema) returns every problem.

Lookup

{ "kind": "lookup", "table_id": "suppliers", "table_version": "latest",
  "match_on": ["supplier_name", {"field": "supplier_city", "column": "city"}],
  "returns": "code", "mode": "always", "k": 5, "threshold": 0.80,
  "require_all": true, "select": "llm" }
  • match_on lists fields of the schema (by field_id, path or name); the table column defaults to the field's name, or is given with {field, column}.
  • mode: always – the field is not in the extraction prompt; if_missing – the model is asked first and the table is consulted only when the field did not end up extracted with verification partial or better.
  • A table is a CSV (UTF-8, , ; or tab; BOM allowed) or Parquet. Versions are immutable and named by the sha256 of their normalized content: the same content is the same version, latest resolves to a number that stays in the evidence, and a run with a pinned table_version reproduces after a newer one is uploaded. Limits: 500 000 rows and 100 MB (DX_LOOKUP_MAX_ROWS, DX_LOOKUP_MAX_BYTES).
  • Matching: names are normalized (no accents, lower case, punctuation and legal-form suffixes removed: suffixes.toml, DX_LOOKUP_SUFFIXES), candidates come from rare shared tokens and are scored with 0.5·JaroWinkler + 0.5·token_sort/100 per column. Only a row where every queried column reaches threshold is a candidate for acceptance: there is never a partial match.
  • Selection: one normalized-equal row is accepted without a model (exact). Otherwise a low-tier model (purpose='lookup') answers with the key of one candidate or null (the schema is an enum of the candidate keys); a key outside the enum is retried once and then null with lookup_invalid_choice. select: deterministic (and a model failure) accepts only a top candidate with score ≥ 0.95 and a margin ≥ 0.10 over the second. At most one model call per query, and an in-process cache makes 100 documents of the same supplier cost one.
  • Table cells and document values reach the model as data inside a delimiter derived from the content, never as instructions.

Where tables live: $DX_LOOKUP_DIR (manifest.json + v<n>.parquet) for the library and the CLI, or the encrypted per-tenant blob store and RLS tables lookup_tables / lookup_table_versions on the server.

extractit lookup add suppliers.csv --key code --name suppliers      # version 1
extractit lookup add suppliers.csv --table suppliers                # same content: unchanged
extractit lookup list && extractit lookup versions suppliers
extractit lookup match suppliers supplier_name="Acme Industrial Supplies SL"   # scored candidates, no model
DX_LOOKUP_DIR=~/.extractit/lookup extractit note.pdf --schema note.schema.json

API: POST/GET /v1/lookup-tables, PATCH /v1/lookup-tables/{id} (If-Match), POST/GET /v1/lookup-tables/{id}/versions, GET …/versions/{n}?preview=20 (latest allowed) and POST …/versions/{n}:match (candidates with scores, no model). Errors: lookup-too-large 413, lookup-duplicate-keys 422, invalid-lookup-config 422, lookup-version-not-found 404. A key with the read scope can GET and :match but not POST.

Evaluation: bench/datasets/lookup-v0 and bench/reports/lookup-v0.md.

Derived fields

kind: derived evaluates a CEL expression over data with the evaluator of the rules engine (T09: num, sum, abs). References are stored as field_id, so renaming a field does not break the expression. No model is called. An input without a value (null, not_found, abstained) gives null with status not_found and the warning derived_missing_input; a result that does not fit the field type gives derived_type_error; a runtime error derived_expression_error. Sums are exact (Decimal, 6 decimals: 0.1 added ten times is 1.0) and JSON integers of number fields read as doubles, so data.total * 2.0 works. Derived fields inside arrays of objects are not supported. Recomputing after a human correction in the viewer is T32.

Metadata, notes and packets

extractit a.pdf b.pdf c.pdf --metadata meta.json --notes notes.txt (and the packet endpoints of T17) send several files and what travels with them.

  • One document. The files' DocIR (parsed and cached one by one) become a single document with global pages: anchors p{packet_page}_l{n}, Page.file_id, Page.file_page, Page.packet_page and the same three fields in every evidence. One file is the individual parse; adding a file parses only that file. Any tier uses the packet view.
  • Metadata (sanitize_metadata): JSON ≤ 64 KiB (metadata-too-large), keys ^[A-Za-z0-9_.-]{1,64}$, depth ≤ 4, ≤ 100 leaves (metadata_entries_dropped), ≤ 500 characters per value in the prompt (metadata_value_truncated), 8 000 characters in total, control characters removed and <, >, & escaped so no value can close the block or forge an anchor. The prompt gets <metadata> with one line per leaf: <a id="metadata.customer.number"></a>customer.number: ACME-77; notes (≤ 8 KiB) become notes.l1…. Everything inside is data, never instructions.
  • Metadata that informs. A document field whose citation is a metadata anchor gets kind='metadata' evidence, verified if the value matches the leaf and contradicted if it differs.
  • Metadata that sets. kind: metadata takes the full value from the JSON (not the prompt's truncation), coerces it with the normalizers of T07 and needs no model call. A missing path is not_found with metadata_path_missing. With cross_check: true the model also reads the field: different values give status='conflict' with the metadata value, contradicting evidence and the document's value in alternatives; equal values add supporting evidence.