Skip to content

Unreleased

Changes merged and not yet in a release.

Added

  • Self-service sign-up (DX_SIGNUP=open, default closed): «Continue with Google», «Continue with Microsoft» and «Email me a sign-in link» at /login and /signup. The first verified sign-in of a new person creates their organization (Owner, default project, welcome credits through identity.owner_verified) unless an invitation to that email is pending, which wins. Google and Microsoft Entra are DX_OIDC_ISSUERS presets ("type": "google" | "microsoft"); multi-tenant Microsoft tokens are checked against their own tenant issuer (tid) and their email counts as verified only with xms_edov. Sign-in links: POST /auth/email/start (always 202), GET /auth/email/link (never spends it), POST /auth/email/confirm (once, 15 minutes, hashed token, per-email/address/global limits). One person, one account: identities with the same verified email join (user_identities, migration 0031_t66). GET/PATCH /v1/organization renames the organization. Terms acceptance (DX_TERMS_VERSION) and the audit events auth.signup, auth.email_link_sent, auth.identity_linked. See docs/ops/signup.md.
  • Welcome credits (commercial edition, DX_BILLING_MODE=saas): each new organization gets 1 000 promotional credits when its first Owner has a verified email (once per organization and per verified email, never to disposable domains). They only pay Flash-tier pages, are spent before purchased credits, expire after 30 days and are not refundable; a run on a higher tier with only welcome credits left is refused with 402 credits-tier-not-covered. GET /v1/billing/balance now returns paid and promo; settings (S15) and the Studio show them. Configured in plans.yaml (welcome_credits). New core hook point identity.owner_verified.

Changed

  • extractit FILE extracts directly (no extract verb); new short alias xt; the dx alias is removed.
  • Core relicensed from Apache-2.0 to FSL-1.1-ALv2 (source-available; each release converts to Apache-2.0 after two years). SDKs, schemas, examples and skills stay Apache-2.0.
  • Canonical URLs moved to extractit.dev (docs.extractit.dev, schemas.extractit.dev, extractit.dev/problems, errors.extractit.dev; SaaS host app.extractit.dev).
  • Renamed: DataExtractor → ExtractIt (package, import, CLI with dx alias, npm scope, images, charts). Tagline: «Extract your data. Every value cited.» DX_* variables, DX-Version, database roles, id prefixes, error-type URIs and JSON Schema $ids are unchanged.

Fixed

  • Fields of type array with simple elements (array<string>, array<number>, array<date>, nested rows[*].tags[*]) no longer break assemble (path indexes a non-array value) on the workflow and agent-mode paths: finalize no longer adds a not_found placeholder for the whole array next to its skills[0]… elements, and data_from_fields lets the elements win over such a placeholder. Elements of array<number|integer|date|datetime|boolean> are now normalized and verified by their element type (they were read as text, so data.scores held strings). An absent top-level array is [] in data on every path. CSV/XLSX exports (and the column list) treat an array of scalars as one list cell, not as a one-column table. Output schema and DocIR are unchanged.

Added

  • T41 measured optimizer: description_rewrite (1–10 rounds on the train/validation splits, candidates as drafts, acceptance by a paired bootstrap, one report on the sealed test split, USD budget, cancellation) and one_shot (unmeasured), /v1/optimizations, promotion by a person only, the Optimizer tab of Health (S11). Both strategies are experimental until a nightly evaluation shows a gain on test.

  • T24 field sources: x-dx.source as one object (document, lookup, derived, metadata; legacy strings normalize to it), reference tables with immutable versions (/v1/lookup-tables, dx lookup), CEL derived fields, citable package metadata, multi-file packets as one document with global pages, and external validators (VIES, GLEIF, bank registry) as rules with cache, rate limit and an SSRF-safe client. Contracts move to dialect 1.2, output 1.2 and DocIR 1.2 (additive; the 1.0 and 1.1 snapshots stay).

  • Monorepo skeleton: Python package dataextractor (uv, hatchling + hatch-vcs), dx --version, web workspace (Vite, React 19, TypeScript, Biome, Vitest, Playwright).

  • make check quality gate, identical locally and in CI.
  • Supply-chain controls: license allow lists, SPDX headers, pinned GitHub Actions, pip-audit, Grype, gitleaks, DCO check.
  • Release workflow (PyPI/TestPyPI trusted publishing, multi-arch GHCR image, CycloneDX SBOM, keyless cosign signature, provenance attestation).
  • Development Docker Compose stack (PostgreSQL 16, Gotenberg, OpenBao).
  • Governance files: LICENSE (Apache-2.0), NOTICE, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY.