Observability
OpenTelemetry is the only contract. API and workers export OTLP/HTTP to the Collector
(OTEL_EXPORTER_OTLP_ENDPOINT); without that variable they export nothing. Customer-facing data
(run timeline, costs) comes from ExtractIt's own tables, never from the tracing backend.
The Collector
deploy/observability/otelcol.yaml (Compose) and the chart's ConfigMap share the same processors
block (deploy/helm/extractit/files/otelcol-processors.yaml, kept identical by a test):
- Content never leaves the Collector.
attributes/drop-contentdeletesgen_ai.input.messages,gen_ai.output.messages,gen_ai.system_instructions,gen_ai.tool.definitions, the legacygen_ai.prompt*/gen_ai.completion*keys,db.query.parametersanddx.content*, on spans, logs and metric data points;transform/drop-contentdoes the same inside span events. - Zero data retention. For a tenant marked
dx.zdr=true(resource or span) only identifiers (tenant_id,project_id,run_id,node_id,dx.*ids), timing, status andgen_aiusage attributes survive; span-event attributes are dropped and log bodies replaced by[redacted: zdr]— even when content capture was turned on in the process. - Tail sampling keeps every error and every trace slower than 30 s, and
DX_OTEL_SAMPLE_PERCENT(default 10 %) of the rest. - Span metrics (
dx_calls_total,dx_duration_*by route and status) are computed from every API server span before sampling: the input of the availability SLO. - Queue metrics (
dx_queue_queued_runs,dx_queue_running_runs,dx_queue_oldest_age_seconds) come from theops.queue_backlogview (receiversqlquery, roledx_dbos).
If the Collector is down, API and runs continue: the SDK exporters retry briefly and drop.
Proof: tests/ops/test_ops_observability.py runs the pinned Collector image, sends traces, metrics
and logs carrying the canary CANARY-7f3a91 in content attributes and for a ZDR tenant, and asserts
0 occurrences in every exported file and in what reaches a stand-in Langfuse OTLP endpoint, while ids
and gen_ai usage attributes are kept.
Exporters
observability.mode |
Chart | Compose |
|---|---|---|
otlp |
Collector → observability.otlp.endpoint (any OTLP backend) and/or Langfuse |
— |
bundle |
Collector + Jaeger | compose.observability.yaml: Collector, Prometheus (SLO rules), Jaeger |
none |
no Collector; processes export nothing | default |
Langfuse (MIT) needs ClickHouse, Valkey and S3, so it is deployed separately with its own chart
or Compose; ExtractIt's Collector only needs its OTLP endpoint (langfuse.otlpEndpoint, e.g.
https://langfuse.example.com/api/public/otel) and a Secret with LANGFUSE_AUTH
(Basic base64(public_key:secret_key)). Compose: compose.langfuse.yaml.
Grafana, Loki and Tempo are AGPL-3.0: they are not bundled or modified. Use them as external
backends if you like: Grafana on Prometheus/Jaeger, or Tempo/Loki as the otlp endpoint.
SLOs and alerts
deploy/observability/rules/slo.yaml (Prometheus rules), tested with
promtool test rules deploy/observability/tests/slo_rules_test.yaml:
| Alert | Condition |
|---|---|
ApiAvailabilityBurnRateFast (page) |
5xx ratio > 14.4 × 0.1 % over 1 h and 5 min (SLO 99.9 % / 30 days) |
ApiAvailabilityBurnRateSlow (ticket) |
5xx ratio > 6 × 0.1 % over 6 h and 30 min |
QueueBacklog (page) |
p95 over 10 min of the oldest queued run's age > 300 s for 10 min |
The tests show that 2 % errors for an hour pages, 0.05 % does not, 1 % for 7 h opens a ticket and
a 400 s queue age fires QueueBacklog.
Status page
GET /v1/status (public, cached 5 s, numbers and fixed reason codes only):
{"status": "up", "checked_at": "…",
"components": [{"name": "database", "status": "up", "detail": {}},
{"name": "queue", "status": "up", "detail": {"queued": 3, "running": 2, "oldest_queued_age_seconds": 4.1}},
{"name": "storage", "status": "up", "detail": {}},
{"name": "workers", "status": "up", "detail": {"executors": 2, "last_activity_age_seconds": 3.0}}]}
down when PostgreSQL does not answer within 5 s, the object store fails, the oldest queued
run waits ≥ 30 min, or no worker touched a workflow for 10 min (the runtime reconciles every 30 s);
degraded from 5 min queue age or 2 min worker silence. Gatus (deploy/status/gatus.yaml, or the
chart's status.gatus.enabled) probes /healthz, /readyz and /v1/status; run one instance
outside the deployment it watches.