Skip to content

Storage connectors

Connectors move files in and out of ExtractIt without a person uploading anything:

  • Ingestion (ING-07, ING-08): a connector watches a folder or mailbox and starts one run of a workflow (by alias) per new file — continuously, or once for a bulk import from Drive or OneDrive.
  • Export (WF-17): the export node sends the result of a run to S3, SFTP, Drive, OneDrive / SharePoint, a local folder or an ERP webhook.

Both use the same resource, the connection: the credentials (sealed per tenant) plus the configuration of one remote. A connector binds a connection and a path to a workflow reference name@alias. The page Settings → Connectors manages both.

Connections

Kind Reaches Direction Credentials
drive Google Drive (shared drives too) source and destination OAuth (PKCE) or a token pasted from rclone authorize
onedrive OneDrive and SharePoint libraries source and destination OAuth (PKCE) or a pasted token
sftp an SFTP server source and destination password or private key; the host key is mandatory
s3 Amazon S3, MinIO, Wasabi, Ceph… source and destination access key id + secret
local a folder on the worker source and destination none (inside DX_WATCH_ROOTS)
imap a mailbox over IMAP source password
graph_mail a Microsoft 365 mailbox source application credentials (Mail.Read)
webhook your ERP destination signing secret (whsec_…)

GET /v1/connection-kinds returns, per kind, the JSON Schema of its configuration, its secret fields and its OAuth provider; the settings page draws its forms from it.

Secrets are write-only. They are split off the configuration, sealed with the tenant data key (AES-256-GCM, AAD organization | project | connection) and never appear in a response, a log, a trace or a database dump. A response lists secret_fields_set and the page shows those fields as saved; a PATCH that omits a secret keeps it. Reading needs project:read; changing a connection, running a connector and authorizing need project:manage. allow_private_endpoint (SFTP, S3 endpoints, IMAP, webhooks that resolve to a private network) can only be set by an organization administrator.

access is read or read_write. Only read_write connections can be export destinations: a read-only one used as a destination fails with CONNECTION_READ_ONLY (422).

rclone

Storage remotes are reached through rclone (MIT), an external binary pinned in src/extractit/connectors/pins.toml (version and the sha256 of the official release) and installed by the rclone stage of the root Dockerfile. scripts/check_rclone_pins.py fails when the Dockerfile and pins.toml differ. Each worker process starts one rclone rcd:

  • it listens on 127.0.0.1 only, on a free port, with a random user and password per process (given through RCLONE_RC_USER/RCLONE_RC_PASS, never the command line); --rc-no-auth and --rc-serve are never used, and core/command and config/* are never called — the rc API is equivalent to a shell for the process user;
  • the configuration file is empty and ephemeral: the remotes travel inline as JSON objects in fs/srcFs/dstFs ({"type": "s3", …, "_root": "bucket/prefix"}), so no rclone.conf with tokens ever exists.

For Drive and OneDrive the application refreshes the OAuth token (under a row lock, persisting the rotated refresh token) and hands rclone only an access token with its expiry. A Google Doc, Sheet or Slide is listed with a .pdf name and exported to PDF (export_formats=pdf).

H6: inline remotes with a token

The design depends on operations/list (with recurse, showHash and _filter.MaxAge), operations/copyfile, operations/stat and core/obscure accepting inline remotes that carry a token. This is verified with the pinned version in the connectors integration job (pytest -m integration_connectors; it ran against rclone v1.76.0 with local, SFTP through rclone serve sftp and S3 through a moto server: list, copyfile, stat and core/obscure accept inline remotes. Drive and OneDrive remotes are only exercised against the fake rc server). Two findings from that run: the listing Path is relative to the remote's _root, so the connector path always goes in _root and remote stays empty; and rclone 1.76 cannot upload to a plain-HTTP S3 endpoint with a signed payload (request stream is not seekable), so fs_for sets use_unsigned_payload=true when the endpoint starts with http:// (TLS endpoints keep signed payloads). If a remote type refuses it, the plan B is a configuration file with mode 0600 generated in tmpfs per call. The validation against real Google Drive and OneDrive/SharePoint accounts (listing, download and upload) is manual and an external dependency.

The local kind does not need rclone: the worker lists and copies the folder natively. The path is resolved with realpath and must be inside DX_WATCH_ROOTS (a list of folders separated by :; empty means no local connection is allowed), symbolic links are skipped, and an upload is written to a temporary name and renamed so a watcher on the same folder never sees half a file.

OAuth applications

Each deployment (or each customer, when self-hosting) uses its own OAuth application. Put the client id (and secret) in the connection, or set DX_OAUTH_GOOGLE_CLIENT_ID/_CLIENT_SECRET and DX_OAUTH_MICROSOFT_CLIENT_ID/_CLIENT_SECRET for the instance, plus DX_PUBLIC_URL (the redirect URI is $DX_PUBLIC_URL/v1/connections/oauth/callback).

Google. In Google Cloud Console create an OAuth client of type Web application, add the redirect URI above, enable the Drive API and add the scope drive.readonly (ingestion) or drive (export). Restricted-scope verification of the application for a hosted SaaS is outside this ticket; a self-hosted customer uses its own application in Testing or Internal mode.

Microsoft. In Entra ID register an application (Web), add the redirect URI, create a client secret and grant the delegated permissions offline_access, Files.Read.All and Sites.Read.All (Files.ReadWrite.All and Sites.ReadWrite.All for export). Prefer Sites.Selected with an explicit grant on the one site when the tenant allows it. An administrator may need to consent. For SharePoint choose drive_type=documentLibrary and give the site_url and library: the drive_id is resolved with Microsoft Graph once the connection is authorized.

Minimum permissions for mailboxes: Microsoft Graph application permission Mail.Read, limited to the mailbox with an application access policy.

The authorization flow: POST /v1/connections/oauth/{google|microsoft}:start stores the state and a PKCE (S256) verifier (sealed, 10 minutes) and returns the authorization_url; the page opens it in a popup, the provider redirects to GET /v1/connections/oauth/callback, which validates the state (single use), exchanges the code and marks the connection active. The page polls GET /v1/connections/{id} until it is active or error; a blocked popup is reported, and pasting the token from rclone authorize is the alternative without a browser.

Connectors

POST /v1/connectors takes connection_id, path, workflow_ref (name@alias; 422 ALIAS_NOT_FOUND if it does not resolve to a published version), poll_interval_s (default 60, minimum DX_CONNECTOR_MIN_POLL_S, 10), duplicate_policy, settle_s, mode (continuous or once) and ingest_existing.

A continuous connector is a DBOS scheduled task connector:<id> (cron with seconds when the interval is below a minute, automatic_backfill=false). Pausing, resuming and deleting the connector pause, resume and delete the schedule. Each tick:

  1. lists what is new since the checkpoint and drops what was already persisted;
  2. keeps only stable files — the same size as at the previous poll and an age of at least settle_s (two polls at least);
  3. starts one child workflow per element with the deterministic id ing-<connector>-<hash> on a queue of concurrency 4, so a re-listing or a restart cannot duplicate;
  4. each child downloads to a 0700 spool (deleted in a finally), computes the sha256, applies the duplicate policy, creates the packet and the connector_items row in one transaction and starts the run with the idempotency key conn:<connector>:<hash of remote_id and etag>. A crash between the packet and the run leaves a row without a run: the retry submits only the run;
  5. a final step advances the checkpoint with the elements that were persisted only.

A connector runs one tick at a time: a tick takes a lease (connectors.tick_lease_until, DX_CONNECTOR_TICK_LEASE_S, default 120 s) that finish_tick releases, so a schedule that fires every second does not pile up ticks while a long one is still ingesting (the children of one tick are awaited together, not one polling interval each). If the worker dies the lease expires and the next tick takes over. An explicit :run ignores it.

Checkpoints by type: files use a modification-time watermark with a 10-minute margin (idempotency absorbs the overlap); IMAP uses {uidvalidity, last_uid} (a changed UIDVALIDITY restarts the cursor, and the Message-ID based remote_id keeps it from duplicating); Graph stores the deltaLink. The alias is resolved when each element is ingested: after moving prod, new elements use the new version.

Duplicate policy (same sha256 again, other name): reuse links the run that already processed it, reject creates nothing, reprocess runs it anew. An element over DX_MAX_FILE_BYTES is skipped_too_large, an unsupported type skipped_unsupported, and failed ones are retried on later ticks up to 5 times. A connector whose poll fails applies an exponential backoff from 1 s to 15 min, shows state=error with last_error (no secrets) and does not affect the others. Remote names are sanitized: no separators, .. or control characters.

mode=once is the one-off import from Drive or OneDrive: POST /v1/connectors/{id}:run ingests what exists (at most 200 elements per call; truncated says there are more) and later runs bring only the changes. :run also works on continuous connectors and answers with the summary (new, duplicates, failed, skipped).

Size

rclone lists the whole folder at each poll (O(n)); the listing is narrowed with a modification-age filter and the minimum interval is 10 s. Keep a connector under about 50 000 files.

Mailboxes

IMAP (imap-tools) polls one folder without marking messages as read unless mark_seen is on. Each message becomes one packet with the original .eml (the conversion of T16 turns it into body and attachments) and the metadata {subject, from, message_id, received_at}. With require_attachments (default) a message without a supported attachment is skipped_unsupported. Microsoft 365 uses the messages/delta query of Graph (the deltaLink is the checkpoint) and downloads the MIME with /messages/{id}/$value. Against GreenMail, which rejects UID SEARCH UID n:*, the source falls back to SEARCH ALL filtered by UID, and the message is fetched with UID FETCH … BODY.PEEK[] (no search, nothing marked as read).

The export node

export accepts Extraction, Verdicts and File, has no outputs, is class io and is not cached.

{
  "destination": "facturas-s3",
  "path": "{prefix}/{workflow_id}/{packet_id}/{run_id}.json",
  "format": "json",
  "mapping": {
    "factura": "${fields.invoice_number}",
    "importe": "${fields.total}",
    "lineas": "${fields.line_items[*].amount}"
  },
  "exclude": ["debug"],
  "on_failure": "fail_run"
}
  • destination is the name (or con_… id) of a connection. Placeholders are ${fields.path} (the extraction's data), ${fields.rows[*].col}, ${document.name|sha256|pages}, ${run.id} and ${segment.id}; there is no Jinja and no expression evaluation. A template that is exactly one placeholder keeps the type. include and exclude take dotted paths. format: "csv" with rows_from: "fields.line_items" writes a header and one row per element (${row.amount} reads the element); a File input is uploaded as it is, with its name and sha256.
  • path accepts {prefix}, {workflow_id}, {packet_id}, {run_id}, {node_id}, {scope_key}, {date}, {name} and {ext}, and every segment is sanitized.
  • Publishing validates the mapping against the schema of the upstream extract nodes, the path keys and the destination (it must exist and be read_write): every violation comes back at once as EXPORT_MAPPING_INVALID, CONNECTION_NOT_FOUND or CONNECTION_READ_ONLY. The destination check starts once the project has at least one connection.
  • Idempotency. idempotency_key = sha256(run_id, node_id, scope_key, connection_id, mapping_hash, path), stored in export_deliveries. An exported row answers already_exported without calling the destination; a started row (a crash in between) repeats the write on the same deterministic path.
  • Retries. Up to 6 attempts, exponential backoff from 2 s with jitter, Retry-After honoured (at most 120 s), no retry of a 4xx except 408 and 429. When they are exhausted the node fails with EXPORT_FAILED and retryable=true — POST /v1/runs/{id}:resume runs only this node — or, with on_failure: "continue_with_warning", succeeds with a warning.
  • Webhook. POST of the JSON (at most 5 MiB) with Idempotency-Key and the Standard Webhooks headers webhook-id, webhook-timestamp and webhook-signature (secret whsec_…), through the SSRF-safe route of the import service: the validated IP is pinned in the URL, there is no proxy and redirects are never followed; loopback, link-local and private addresses (and DNS rebinding to them) are refused unless allow_private_endpoint is set.

Operations

Variable Default Meaning
DX_RCLONE_BIN rclone on PATH the pinned binary
DX_WATCH_ROOTS empty folders a local connection may use
DX_CONNECTOR_SPOOL $TMPDIR/dx-spool spool root (0700)
DX_CONNECTOR_MIN_POLL_S 10 smallest accepted poll_interval_s
DX_CONNECTOR_TICK_LEASE_S 120 how long a running tick holds its connector
DX_MAX_FILE_BYTES plan limit (25 MB) bigger elements are skipped_too_large
DX_PUBLIC_URL http://127.0.0.1:8000 base of the OAuth redirect URI
DX_OAUTH_<GOOGLE\|MICROSOFT>_CLIENT_ID/_SECRET — the instance's OAuth application
DX_OAUTH_<…>_AUTH_URL/_TOKEN_URL the provider's point at a simulated IdP (tests)
DX_GRAPH_URL https://graph.microsoft.com/v1.0 Microsoft Graph base

The worker loads the connector workflows through the extractit.run_handlers entry points connectors and export_node. Tests of the integrations run with pytest -m integration_connectors (rclone, GreenMail and a moto server; the moto server needs moto[server], i.e. flask); everything else uses the fake rc server in tests/fixtures/connectors/fake_rc.py.