Upgrades without downtime
helm upgrade extractit oci://ghcr.io/<owner>/charts/extractit --version <new> \
-n extractit --reuse-values
What happens, in order:
- Migrations Job (
helm.sh/hook: pre-upgrade):python -m extractit.store.admin migrateasdx_migratorwhile the old pods keep serving. Migrations follow expand/contract: a release only adds (columns, tables, views); removing what the old code still uses ships one release later. A failed Job stops the upgrade and leaves the old release running. There is never an automatic downgrade. - Rolling update of API, web and workers with
maxUnavailable: 0: a new pod receives traffic only after/readyzanswers; the old one gets 10 s (preStop) to leave the endpoints and 20 s to finish its requests. - Runs in flight survive: workers stop on SIGTERM after
terminationGracePeriodSeconds, and DBOS resumes an interrupted workflow from its last completed step on another worker. - PodDisruptionBudgets (API, web, workers, and CloudNativePG's) keep at least one pod of each during node drains.
Start-up preflight
Every API and worker pod runs python -m extractit.ops.preflight --wait 600 as an init
container. It reads alembic_version and never writes:
| Exit | Meaning |
|---|---|
| 0 | the database is at this code's head |
| 2 | no database URL, database unreachable or no privilege (retried while --wait lasts) |
| 3 | the database revision is unknown to this code: it was migrated by a newer release. The pod refuses to start and the database is left untouched. Roll forward. |
| 4 | the database is behind this code (the migrations Job has not run yet; retried while --wait lasts) |
Rolling back
helm rollback restores the previous manifests but not the schema. The previous code runs on the
expanded schema (expand/contract), so a rollback right after an upgrade is safe; an old pod whose
code does not know the new revision exits 3 instead of corrupting data, which is why releases keep
the contract step one release behind. Restoring the database itself is a
point-in-time restore.
Measured
deploy/tests/upgrade_test.sh (kind): upgrade from the installed chart to the next version with a
parse run in flight and a probe of 100 requests per minute through the Service: 0 responses 5xx, the
run ends succeeded, the migrations hook completes, evicting every API pod at once is refused by the
PodDisruptionBudget and the preflight exits 3 against a database at a future revision.