Chartreuse: Automated Alembic SQL schema migrations within kubernetes
"How to automate management of Alembic database schema migration at scale using CI/CD and Kubernetes"
Chartreuse is a wrapper around Alembic to ease, detect and automate migrations on deployed Python applications.
Chartreuse leverages Helm Hooks, the Hooks are defined in Chartreuse Helm Chart.
Usage
Requirements
- Python >= 3.7
- Using Helm to deploy you application
- This Python package requires the
expecteddeploymentscales.wiremind.ioKubernetesCustom Resource Definitionfromwiremind-kubernetesrepository:
kubectl apply -f https://raw.githubusercontent.com/wiremind/wiremind-kubernetes/main/CustomResourceDefinition-expecteddeploymentscales.yaml
- Please make sure Chartreuse Python Package version and Chartreuse Helm Chart version, you use, share
major.minorotherwise Chartreuse won't start.
Configuration
Using Helm
Chartreuse comes with a Helm Chart ready to be used as a Helm Subchart in your own Helm Chart.
All you have to do is build your own container image containing:
- Chartreuse Python package
- Your Alembic migrations in an
alembicdirectory - All required dependencies to run your alembic migrations.
Usually, it will be the same container image for your project with your code as usual, with Chartreuse added as dependency in your setup.py.
and state in the Chartreuse Helm Chart values.yaml:
- the image repository and tag
- URL to connect to your PostgreSQL
During install and/or upgrade of your Helm Release, Chartreuse will run as Kubernetes Job and automatically migrate PostgreSQL shchema to HEAD if needed.
If required, it will also scale down Deployments that should NOT run during a Deployment using ExpectedDeploymentScale CRD.
Please refer to the example directory for example.
Diagram
The state diagram of your application while upgrading using Helm and using Chartreuse for your migrations is as follows:
Notes
- PG clusters managed by postgres-operator (Patroni PG):
- When Chartreuse starts running against a PG cluster managed by
postgres-operator(Patroni PG), it may run the migrations before that the cluster is configured, and by configured we mean:- the Roles, especially
wiremind_owner_userandwiremind_ownerused by Chartreuse and Alembic, are created. - The default privileges are set, so the other Roles, like
wiremind_writer_userused by the application, can interact with the created objects. To ensure that, Chartreuse will not start untilpostgres-operatorhas performed the above two actions. To make Chartreuse wait, the environment variableCHARTREUSE_PATRONI_POSTGRESQLshould be set:
# in the appropriate Helm values file chartreuse-for-a-patroni-pg: additionalEnvironmentVariables: CHARTREUSE_PATRONI_POSTGRESQL: "anything" CHARTREUSE_ALEMBIC_POSTGRES_WAIT_CONFIGURED_TIMEOUT: 100 # It's set to 60s by default
- the Roles, especially
- The default privileges above-mentioned are set for the NOLOGIN owner
wiremind_owner, e.g. tables should be created bywiremind_ownersowiremind_writer[_user]can insert to them. This is why we need toSET ROLE wiremind_ownerin the beginning of the transaction before running the migrations, Chartreuse does set-x patroni_postgresql=yestoalembic upgrade headwhen the environment variablePATRONI_POSTGRESQLis set, you can then retrieve the argument and set the role in yourenv.py:... patroni_postgresql: bool = "patroni_postgresql" in context.get_x_argument(as_dictionary=True) ... with connectable.connect() as connection: ... with context.begin_transaction(): if patroni_postgresql: context.execute("SET ROLE wiremind_owner") context.run_migrations() ...
- When Chartreuse starts running against a PG cluster managed by
- Chartreuse in pre-upgrade mode:
- When running Chartreuse in pre-upgrade mode (
upgradeBeforeDeployment: true), it will not start running (The Chartreuse Pod will hang inInitstate) until one PG Pod (and ES Pod if ES is used) is running, so make sure these Pods are available to Chartreuse. To fix that:- You will need to delete the Chartreuse Job so the upgrade can resume and fix you PG and ES pods (or create them if they don't exist), then you can redeploy so your migrations can run.
- You can also try the
upgradeBeforeDeployment: falsemode (maybe temporarily).
- When running Chartreuse in pre-upgrade mode (
Development
Test
There are three kind of tests:
- Unit tests
- Integration tests: allows to run in a real environment, but still control chartreuse from the inside
- blackbox test: deploy a real Helm Release and test if databases are migrated.
Documentation
- The diagram has been drawn using the free online software https://draw.io, the source code is located at
doc/chartreuse_sd.xml, feel free to correct it or make it more understandable.
10 Nov 14, 2022
2 Feb 22, 2022
648 Jan 02, 2023
70 Dec 31, 2022
13.3k Jan 01, 2023
29 Dec 14, 2022
1.3k Dec 30, 2022
3 May 01, 2022
94 Oct 10, 2022
51 Oct 16, 2022
6 Aug 28, 2022
152 Dec 25, 2022
226 Dec 29, 2022
1 Dec 20, 2021
19 Dec 17, 2022
2 Feb 11, 2022
3 Nov 04, 2021
8.7k Jan 08, 2023
149 Dec 22, 2022