Skip to content

Model backends

ExtractIt talks to language models through one port, generate_structured: you give it a conversation (text and images) and a JSON Schema, and you get back a JSON document that validates against that schema, together with token usage, cost and latency. Which model answers is configuration, not code:

export DX_MODEL=gemini:standard   # Gemini API key
export DX_MODEL=openai:standard   # OpenAI API key
export DX_MODEL=anthropic:fast    # Anthropic API key
export DX_MODEL=compat:qwen2.5-vl # any OpenAI-compatible server (vLLM, llama.cpp...)
export DX_MODEL=codex             # your ChatGPT subscription, through the official Codex CLI
export DX_MODEL=fake              # offline, deterministic, for tests

DX_MODEL is gemini|openai|anthropic[:<alias|snapshot>], compat:<model>, codex[:<model>] or fake[:<cassette directory>]. If it is not set and a Gemini key is present, gemini:standard is used; otherwise the command fails with the list of options. Run extractit doctor to see what is usable right now.

What every backend guarantees

  • Valid output. The schema is first down-levelled to the subset the provider accepts (what was lost is reported in StructuredResponse.downleveled). The answer is then validated against your original schema (JSON Schema Draft 2020-12). If it does not validate, the gateway asks the model to fix it, quoting the JSON Pointer of the error, at most 2 times; after that you get a SchemaViolationError carrying the pointer.
  • Retries. Up to 3 retries on 429, 5xx and timeouts, with exponential backoff (base 1 s, factor 2), full jitter and respect for Retry-After.
  • Accounting. Every call produces a ModelCall (tokens, latency, snapshot, purpose, prompt hash, cost) in the call ledger, which also sums per purpose. The ledger stores hashes and counters, never document text. Logs never contain document text or keys.
  • Typed errors. RateLimitedError (with retry_after), SchemaViolationError, TruncatedOutputError, ModelRefusedError, ModelUnavailableError, BackendNotInstalledError, BackendAuthError, InvalidRequestError and CassetteMissingError, all subclasses of GatewayError with a coarse kind (input, model, rate_limit, auth, internal).
  • Images up to 20 MB each.

Gemini (API key)

export DX_GEMINI_API_KEY=...          # also read: GEMINI_API_KEY, GOOGLE_API_KEY
export DX_MODEL=gemini:fast           # fast | standard | strong | an exact snapshot

Built on the official google-genai SDK (response_mime_type="application/json" plus response_json_schema). Aliases map to snapshots in src/extractit/llm/models.toml, where every entry carries its price per million tokens, the date it was verified and the source URL. The snapshot the service actually used (model_version) is what you get in model_snapshot. Billing is per token, so cost_usd is computed from that file (cache-aware).

Privacy: free tier

On the free tier of the Gemini API, Google may use the content you send to improve its products. For sensitive documents use a paid key. The tier of a key cannot be detected, so this notice is shown the first time you use the backend, always.

OpenAI, Anthropic and OpenAI-compatible servers

Three more adapters sit behind the same port and pass the same contract suite. Their models, regions, retention and prices live in the model registry.

export DX_OPENAI_API_KEY=...        # only DX_* variables are read: a stray OPENAI_API_KEY is ignored
export DX_ANTHROPIC_API_KEY=...
export DX_COMPAT_BASE_URL=http://localhost:8000   # + DX_COMPAT_API_KEY if the server wants one
export DX_MODEL_POLICY=policy.toml  # optional: see "Model policies"
Backend API How the schema is enforced
openai Responses API text.format of type json_schema, strict: true, schema compiled with the openai_strict profile (closed objects, every property required, optional ones nullable, unsupported keywords dropped)
anthropic Messages API a forced tool emit_result whose input_schema is the compiled schema (anthropic_tool profile) for snapshots that accept it (Haiku 5.5), or output_config.format for those that reject a forced tool_choice (Sonnet, Opus, Fable): the registry says which
compat:<model> Chat Completions response_format with the schema if the probe sees it working, otherwise JSON by prompt (structured_output='emulated')

In every case the answer is validated against your original schema and repaired at most twice. PDFs (PdfPart) are sent natively where the registry says native_pdf; images where vision. Log-probs are requested only where the registry declares them and the endpoint returns them (StructuredResponse.token_logprob_mean is None otherwise). The SDKs' own retries are off: the gateway's retry policy (3 retries, backoff, Retry-After) owns them, so a fallback sees one clean failure. Documents are placed in user turns only, never in the system prompt, which also tells the model that document text is data and never instructions.

OpenAI-compatible servers. probe(base_url, model) asks the endpoint instead of assuming: GET /v1/models; a minimal request with a json_schema response_format (native only if the answer really is that JSON, else a plain request decides emulated or none); a 1x1-pixel image (vision); and a request with logprobs=true. The result is cached per (base_url, model), runs automatically on first use, and can be added to the registry as a local route (locality='local', retention_class='local', zdr=True) with ModelCapabilities.to_entry(). billing='self-hosted' (default) costs 0.0; give a hosted clone billing='api' and prices to bill it per token.

Model policies, fallback and the registry

from extractit.llm.registry import (
    ModelPolicy,
    Needs,
    merge_policies,
    resolve,
    load_registry,
    route_gateway,
)

policy = merge_policies([platform, organisation, project, node])  # can only tighten
route = resolve(policy, Needs(vision=True), load_registry())  # pure and deterministic
gateway = route_gateway(route)  # primary + compliant fallbacks
  • Policy (ModelPolicy): allowed_providers, eu_only, zdr_required, max_usd_per_1k, air_gapped, preference, fallbacks. Merging intersects lists, OR-s the switches and takes the lowest budget, so a node can never relax what the organisation set. The effective policy has a policy_hash (SHA-256 of its RFC 8785 JSON). DX_MODEL_POLICY names a TOML or JSON file.
  • Resolution filters the registry (provider_not_allowed, not_local, region_not_eu, zdr_unavailable, capability_missing, over_budget, in that order) and ranks the rest by (index in preference, input + output price, id): no randomness. If nothing complies it raises NoCompliantModelError carrying every rejected candidate with its reason: it fails closed.
  • Fallback (FallbackGateway): walks the route and its compliant fallbacks on rate_limited, provider_unavailable and timeout (fallback_on is configurable; schema_violation and content_filtered do not fall back by default). A per-provider circuit breaker opens after 5 consecutive failures in 60 s for 30 s. If every candidate fails the call raises ProviderUnavailableError: it never improvises a provider the policy did not return.
  • Ledger: each logical call writes one row with policy_hash, the route used, the rejected candidates and fallback_attempts (provider, model, outcome, latency). Metadata only: no keys, no content.

Codex CLI (ChatGPT subscription)

codex login                # once; ExtractIt never reads or stores your credentials
export DX_MODEL=codex      # or codex:<model> to name a model

The adapter runs codex exec as a child process with your own ChatGPT session:

codex exec --ephemeral --skip-git-repo-check --sandbox read-only \
  --output-schema <tmp>/schema.json -o <tmp>/out.json --json -
  • The prompt goes through stdin; images are written to the temporary directory and passed with --image.
  • The working directory is an empty temporary directory, deleted afterwards. The sandbox is read-only and the session is ephemeral (nothing is written to your Codex history).
  • The child gets a minimal environment (PATH, HOME, CODEX_HOME, LANG, TMPDIR). OPENAI_API_KEY and CODEX_API_KEY are removed so that usage is billed to your plan and not to an API key, unless you set DX_CODEX_ALLOW_API_KEY=1.
  • At most 2 codex processes run at the same time in a process; each call has a 300 s timeout and the whole process group is killed when it expires.
  • model_snapshot is the output of codex --version. There are no log-probs with this backend, and cost_usd is None (the plan is not billed per token).

Limits and failure modes:

  • Plan limits. ChatGPT plans have usage windows (about every 5 hours, plus a weekly one). When they run out you get RateLimitedError immediately (no retries: waiting seconds does not refill a window), with retry_after when Codex says it, and, only if a Gemini key is configured, a fallback_hint of DX_MODEL=gemini:<alias>.
  • Latency is higher than an API call; that is why concurrency is capped.
  • No session (codex login status != 0) gives BackendAuthError with the action to take; a missing executable gives BackendNotInstalledError.
  • Local, personal use only. Do not put this backend behind a hosted multi-user service.

Privacy: ChatGPT account settings

With a ChatGPT login, your documents follow your account's data settings. On personal plans, turn off Improve the model for everyone before processing sensitive documents. The notice is shown once.

Both notices go to stderr (even without a TTY), once per backend; the state lives in $DX_HOME/state.json (default ~/.extractit/). DX_PRIVACY_ACK=1 silences them.

FakeModel (tests)

DX_MODEL=fake[:<directory>] replays cassettes and needs no network:

  • Key: sha256 of the canonical JSON of (model, system, messages with each image replaced by its sha256, schema, temperature, purpose). File: <directory>/<purpose>/<first 16 hex>.json, holding a summary of the request (sizes and hashes, no text), the answer and a fixed usage.
  • Missing cassette: CassetteMissingError with the full hash and the summary.
  • FakeModel("record", inner=<real gateway>) writes cassettes and needs DX_LIVE=1. The smoke tool does it for you:
DX_LIVE=1 DX_MODEL=codex python -m extractit.llm.smoke \
  --schema tests/fixtures/llm/invoice.schema.json --text-file tests/fixtures/invoice.txt \
  --record tests/cassettes/llm
  • FakeModel.scripted([...]) returns programmed answers in order (JSON, dicts, exceptions or callables) and FakeModel.stochastic(truth, p=0.8, seed=0) is right with probability p and otherwise changes one value, deterministically for a given seed.

Record cassettes only with synthetic documents, and check them for secrets before committing: grep -rIl -E 'AIza|key=|x-goog-api-key|sk-' tests/cassettes must print nothing.

Checking a live backend

DX_LIVE=1 DX_MODEL=codex python -m extractit.llm.smoke \
  --schema tests/fixtures/llm/invoice.schema.json --text-file tests/fixtures/invoice.txt
DX_LIVE=1 pytest -m live        # live tests; excluded from CI, they spend quota or money

tests/llm/test_live_providers.py holds the OpenAI, Anthropic and OpenAI-compatible live checks (one call each, synthetic invoice): DX_LIVE=1 DX_OPENAI_API_KEY=... pytest tests/llm/test_live_providers.py -k openai, and likewise for anthropic and compat (which also needs DX_COMPAT_BASE_URL and DX_COMPAT_MODEL). They were written but not run in T15: the cassettes under tests/cassettes/{openai,anthropic,compat}/ were written by hand from each provider's public documentation and must be compared with a real response before the adapters are announced as validated.

Real backends refuse to run without DX_LIVE=1.