Skip to content

Upgrades without downtime

helm upgrade extractit oci://ghcr.io/<owner>/charts/extractit --version <new> \
  -n extractit --reuse-values

What happens, in order:

  1. Migrations Job (helm.sh/hook: pre-upgrade): python -m extractit.store.admin migrate as dx_migrator while the old pods keep serving. Migrations follow expand/contract: a release only adds (columns, tables, views); removing what the old code still uses ships one release later. A failed Job stops the upgrade and leaves the old release running. There is never an automatic downgrade.
  2. Rolling update of API, web and workers with maxUnavailable: 0: a new pod receives traffic only after /readyz answers; the old one gets 10 s (preStop) to leave the endpoints and 20 s to finish its requests.
  3. Runs in flight survive: workers stop on SIGTERM after terminationGracePeriodSeconds, and DBOS resumes an interrupted workflow from its last completed step on another worker.
  4. PodDisruptionBudgets (API, web, workers, and CloudNativePG's) keep at least one pod of each during node drains.

Start-up preflight

Every API and worker pod runs python -m extractit.ops.preflight --wait 600 as an init container. It reads alembic_version and never writes:

Exit Meaning
0 the database is at this code's head
2 no database URL, database unreachable or no privilege (retried while --wait lasts)
3 the database revision is unknown to this code: it was migrated by a newer release. The pod refuses to start and the database is left untouched. Roll forward.
4 the database is behind this code (the migrations Job has not run yet; retried while --wait lasts)

Rolling back

helm rollback restores the previous manifests but not the schema. The previous code runs on the expanded schema (expand/contract), so a rollback right after an upgrade is safe; an old pod whose code does not know the new revision exits 3 instead of corrupting data, which is why releases keep the contract step one release behind. Restoring the database itself is a point-in-time restore.

Measured

deploy/tests/upgrade_test.sh (kind): upgrade from the installed chart to the next version with a parse run in flight and a probe of 100 requests per minute through the Service: 0 responses 5xx, the run ends succeeded, the migrations hook completes, evicting every API pod at once is refused by the PodDisruptionBudget and the preflight exits 3 against a database at a future revision.