Skip to content

Air-gapped install (commercial edition)

The air-gapped profile installs ExtractIt on a cluster without any network path to the internet: local models (vLLM on GPU, or llama.cpp on CPU), telemetry off for good, every egress outside the cluster denied, and a signed bundle verified file by file before anything is loaded. It is part of the commercial edition (ee/, FSL-1.1-ALv2); the Apache-2.0 chart of the standard install is unchanged and installs on its own.

Piece Where
Bundle build, verify, install ee/deploy/airgap/ (build-bundle.sh, verify.sh, install.sh)
Chart and profiles ee/deploy/helm/extractit-ee/ (values-airgapped.yaml, values-airgapped-cpu.yaml)
Models ee/tools/airgap/models.yaml → models.lock.json (mirror_models.py)
Tools in the image python -m extractit_ee.airgap … (also extractit airgap …)
CI .github/workflows/airgap-install.yml

What the bundle contains

dx-airgap-<version>.tar.zst is a zstd tar of one directory:

File Content
haul.tar.zst hauler store save: every image of images.yaml (T53) and images-ee.yaml by digest, both charts, and the model weights as Hauler files (<model>__<path>)
models.lock.json per model: name, pinned revision, role (llm / vlm), served name, SPDX license, every file with sha256 and size
charts/ extractit-<v>.tgz (Apache-2.0) and extractit-ee-<v>.tgz
values/ values-airgapped.yaml, values-airgapped-cpu.yaml
sbom/, signatures/ T53's CycloneDX SBOMs and cosign signatures of the release
images.yaml, images-ee.yaml, images.txt, hauler-manifest.yaml what went into the haul
bundle.manifest.json + .sig sha256 and size of every file above, signed with cosign sign-blob (offline release key)

The release public key (cosign.pub) and the verify kit (verify.sh, dx_airgap_verify.py, install.sh) travel out of band next to the bundle: never trust the copies inside it.

1. Build the bundle (online, at release)

export COSIGN_KEY=release.key COSIGN_PASSWORD=…          # offline release key
export SBOM_DIR=release/sbom SIGNATURES_DIR=release/signatures
export EXTRA_IMAGES=operators.txt                        # CloudNativePG operator, NVIDIA device plugin…
ee/deploy/airgap/build-bundle.sh 1.4.0                   # -> dist/airgap/dx-airgap-1.4.0.tar.zst

Order of the build: models-verify --lock-only (licenses), mirror_models.py download (weights at the pinned revisions, sha256 checked in streaming), models-verify again on the files, both charts packaged, hauler store add image|chart|file, hauler store save, bundle.manifest.json, cosign sign-blob, tar --zstd. MODELS_PROFILE=cpu builds a small bundle with the CPU model only.

2. Verify (offline, before loading anything)

./verify.sh dx-airgap-1.4.0.tar.zst --key cosign.pub     # exit 0 verified, 1 failed, 2 usage

The archive is extracted into a private directory, the manifest signature is checked with the release key (no transparency log: offline), then every file's sha256 and size is recomputed in 1 MiB chunks (a 16 GB weight is never held in memory). A file the manifest does not list fails too. One altered byte in an image layer, a weight, a chart or the manifest returns 1 and nothing is loaded.

3. Install

./install.sh dx-airgap-1.4.0.tar.zst --key cosign.pub --profile gpu \
  --values site.yaml                                     # StorageClass, publicUrl, ingress…

install.sh runs verify.sh, then hauler store load, serves the store on this host (hauler store serve registry on :5000 and fileserver on :8080, or uses --registry / --weights-url), and installs the ee chart with global.imageRegistry pointing at the local registry, models.lockJson and --wait (15 min). Before any workload starts, two hook Jobs run:

  • guard (extractit airgap check-endpoints): fails the install with external_endpoint_forbidden if any endpoint of the configuration leaves the cluster. The same rule is applied at render time (helm template fails) and by the API's start-up hook when the ee image runs it.
  • preflight: exit codes are a contract.
Code Meaning
0 StorageClass, memory, GPU and local registry OK
2 configuration error (no cluster credentials)
10 the StorageClass is missing (or no default one)
11 no schedulable node has the minimum allocatable memory (preflight.minMemory)
12 gpu.enabled but no node has an allocatable nvidia.com/gpu
13 the local registry does not answer GET /v2/

Profiles

GPU (values-airgapped.yaml) CPU degraded (values-airgapped-cpu.yaml)
Extraction LLM vLLM, qwen3-8b (Apache-2.0) llama.cpp, qwen3-1.7b-q8 GGUF (Apache-2.0)
Document VLM (route C of T20) vLLM, qwen3-vl-8b (Apache-2.0) none: the router keeps to paths A/B
Scheduling nvidia.com/gpu: 1, toleration and nodeSelector per server no GPU (the chart refuses gpu.enabled here)

Both profiles share the bundle. The core reaches the models through one OpenAI-compatible endpoint, dx-models (DX_COMPAT_BASE_URL=http://dx-models:8000/v1), which routes by model name to dx-vllm-llm / dx-vllm-vlm / dx-llamacpp. Model servers run with HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, usage statistics off and xgrammar guided decoding; their init container (extractit airgap models-fetch) copies the weights from the in-cluster file server and recomputes every sha256 before the server starts. The platform model policy (/etc/extractit/airgap/model-policy.toml) sets air_gapped = true: T15's resolver accepts only local routes. In the CPU profile the only VLM engine of the catalogue is refused by that policy, so a workflow that asks for a VLM tier still runs on A/B (warning no_eligible_engine).

Models are chosen in ee/tools/airgap/models.yaml and measured with T10's harness; change them there (then mirror_models.py lock), never in code. Only Apache-2.0, MIT and BSD weights are mirrored: AGPL/GPL, unknown or any other license is refused by mirror_models.py, by models-verify and by the chart.

Confidence. Local models have no calibrator until T39 fits one on their snapshot, so scores stay confidence.kind = heuristic. A calibrator fitted on another model_snapshot is never loaded (calibrator_snapshot_mismatch).

Telemetry and support

values.schema.json of the ee chart rejects, with airgapped.enabled=true: telemetry.enabled=true, observability.mode other than none, a Gatus page, a Langfuse or OTLP endpoint, OTEL_SDK_DISABLED other than true, HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE other than 1, network policies off, and any egress CIDR that is not private. No Collector, no error sink.

A support archive exists only when an operator asks for it:

kubectl -n extractit logs deploy/dx-worker --since=24h > logs/worker.log
python -m extractit_ee.airgap support-bundle --logs logs/ -o support.tar.gz

Only structured log lines are kept, with an allow list of keys (event, level, ids, codes, timings); document text, prompts, model answers and secrets are dropped, remaining strings are scrubbed (tokens, keys, credentials in URLs, e-mails, IBANs) and free-text lines are omitted.

Upgrades: the N-2 rule

An air-gapped release upgrades in place from the two previous minor versions of the same major.

python -m extractit_ee.airgap upgrade-check --from 1.2.0 --to 1.4.0   # allowed: prints the steps
python -m extractit_ee.airgap upgrade-check --from 1.0.0 --to 1.4.0   # refused (exit 1): go to 1.2.x first

Steps: back up (CloudNativePG backup and the dx-home Secret) → verify.sh the new bundle → hauler store load and serve → preflight → helm upgrade with the profile values (the migrations Job expands the schema first; never a downgrade) → models-verify and the smoke extraction → keep the old bundle until the smoke test passes (rolling back means restoring the backup).

Quality against the cloud

On the GPU test machine, run T10's harness on its synthetic dataset (≥ 200 documents, fixed seed) with the local models and with the cloud reference, same documents and schemas, then:

python -m extractit_ee.airgap quality-gate --local local.json --cloud cloud.json -o parity.md

It pairs the two reports with T10's comparison (hierarchical paired bootstrap) and passes when |ΔF1| ≤ 1.0 point, at least 200 documents are paired and the run recorded no external connection (run the eval inside SocketGuard or in the air-gapped cluster). If the gap is larger, the cause is documented and the model decided before closing.

Hardware matrix

Profile Hardware Driver / runtime Memory Status
GPU 1× NVIDIA GPU with ≥ 24 GB per model server (L4, A10G, L40S, A100) NVIDIA driver ≥ 550 with the precompiled driver container and device plugin mirrored 64 GB node not yet measured: pending the GPU test machine
CPU degraded 8 vCPU — 16 GB node chart and fake-model path tested in CI; real llama.cpp run pending

GPU images and drivers must come from a precompiled driver container or your own mirror (add them to EXTRA_IMAGES). Certify a row only after the smoke test and the quality gate pass on it.

ADR-017 addendum: bundle format

  • Hauler (Apache-2.0) is the bundle tool: hauler store add image|chart|file, hauler store save and, offline, hauler store load plus hauler store serve registry|fileserver.
  • Verified on day one (Hauler 1.2.5): local charts are added by absolute path; files are stored flat (we name weights <model>__<path>); the served registry exposes repositories without the source registry host (so global.imageRegistry rewriting works). Finding: an image added with --platform from a multi-arch index is stored under the index digest but only one platform manifest, and serve registry then rejects it (DIGEST_INVALID); the build therefore keeps all platforms by default (PLATFORM= empty), since the charts pin index digests.
  • Not yet verified: files of tens of GB (the lock holds 16–18 GB per GPU model). If Hauler's file store does not cope on the build machine, the fallback is Zarf (Apache-2.0) for the weights, keeping the same signed manifest and verify.sh.
  • Two layers on purpose: Hauler's archive has no place for an external signed manifest, so the outer tar holds the haul plus bundle.manifest.json signed with cosign sign-blob.
  • No offline license file in this scope (and no Zarf package for defence/government).

CI: airgap-install

.github/workflows/airgap-install.yml (release tags and on demand) builds the bundle, creates an internal: true Docker network, a canary DNS (ee/deploy/airgap/ci/canary_dns.py, the node's only resolver: logs every query and answers NXDOMAIN), a kind cluster on that network with an iptables audit chain counting packets to public addresses, runs install.sh --fake-llm, the smoke extractit FILE of a sample invoice and fails on any external DNS query, any public packet, a pod not Ready or an install over 15 min. A host on an internal: true network has no default route, so a packet to a public address dies in the routing decision before any iptables chain could count it: the job therefore routes the node through the gateway (the audit chain counts and drops everything public, and the host drops it again) and, after measuring, makes one deliberate connection from the node that must move the counter; otherwise the 0 proves nothing and the job fails. ee/deploy/airgap/ci/network-rehearsal.sh exercises the same isolation with Docker only and the fake model (no kind, no release images): a structured extraction call through dx-models must leave 0 external DNS queries and 0 packets to public addresses; then a deliberate lookup of api.openai.com must show up in the canary log and a deliberate connection with a forced default route must be counted by the audit rule and never connect. Its last run is kept in ee/deploy/airgap/ci/evidence/.