kubernetesapache-airflow

Upgrading Notes

Improve this page by contributing to our documentation.

To 13.0.0

This major update the Redis™ subchart to its newest major, 17.0.0, which updates Redis™ from its version 6.2 to the latest 7.0.

To 12.0.0

This major release renames several values in this chart and adds missing features, in order to be inline with the rest of assets in the Bitnami charts repository. Additionally updates the PostgreSQL & Redis subcharts to their newest major 11.x.x and 16.x.x, respectively, which contain similar changes.

What changes were introduced in this major version?

  • auth.forcePassword parameter is deprecated. The new version uses Helm’s lookup functionalities and forcing passwords isn’t required anymore.
  • config and configurationConfigMap have been renamed to configuration and existingConfigmap, respectively.
  • dags.configMap and web.configMap have been renamed to dags.existingConfigmap and web.existingConfigmap, respectively.
  • web.containerPort and worker.port have been regrouped under the web.containerPorts and worker.containerPorts maps, respectively.
  • web.podDisruptionBudget, scheduler.podDisruptionBudget and worker.podDisruptionBudget maps have been renamed to web.pdb, scheduler.pdb and worker.pdb, respectively.
  • worker.autoscaling.replicas.min, worker.autoscaling.replicas.max, worker.autoscaling.targets.cpu and worker.autoscaling.targets.memory have been renamed to worker.autoscaling.minReplicas, worker.autoscaling.maxReplicas, worker.autoscaling.targetCPU and .Values.worker.autoscaling.targetMemory, respectively.
  • service.port and service.httpsPort have been regrouped under the service.ports map.
  • ingress map is completely redefined.
  • metrics.service.port has been regrouped under the metrics.service.ports map.
  • Support for Network Policies is dropped and it’ll be properly added in the future.
  • The secret keys airflow-fernetKey and airflow-secretKey were renamed to airflow-fernet-key and airflow-secret-key, respectively.

Upgrading Instructions

To upgrade to 12.0.0 from 11.x, it should be done reusing the PVC(s) used to hold the data on your previous release. To do so, follow the instructions below (the following example assumes that the release name is airflow and the release namespace default):

NOTE: Please, create a backup of your database before running any of those actions.

  1. Obtain the credentials and the names of the PVCs used to hold the data on your current release:

     export AIRFLOW_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
 export AIRFLOW_FERNET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
 export AIRFLOW_SECRET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-secretKey}" | base64 --decode)
 export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
 export REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
 export POSTGRESQL_PVC=$(kubectl get pvc -l app.kubernetes.io/instance=airflow,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")

  2. Delete the Airflow worker & PostgreSQL statefulset (notice the option –cascade=false) and secrets:

     kubectl delete statefulsets.apps --cascade=false airflow-postgresql
 kubectl delete statefulsets.apps --cascade=false airflow-worker
 kubectl delete secret postgresql --namespace default
 kubectl delete secret airflow --namespace default

  3. Upgrade your release using the same PostgreSQL version:

     CURRENT_PG_VERSION=$(kubectl exec airflow-postgresql-0 -- bash -c 'echo $BITNAMI_IMAGE_VERSION')
 helm upgrade airflow bitnami/airflow \
   --set loadExamples=true \
   --set web.baseUrl=http://127.0.0.1:8080 \
   --set auth.password=$AIRFLOW_PASSWORD \
   --set auth.fernetKey=$AIRFLOW_FERNET_KEY \
   --set auth.secretKey=$AIRFLOW_SECRET_KEY \
   --set postgresql.image.tag=$CURRENT_VERSION \
   --set postgresql.auth.password=$POSTGRESQL_PASSWORD \
   --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC \
   --set redis.password=$REDIS_PASSWORD \
   --set redis.cluster.enabled=true

  4. Delete the existing Airflow worker & PostgreSQL pods and the new statefulset will create a new one:

     kubectl delete pod airflow-postgresql-0
 kubectl delete pod airflow-worker-0

To 11.0.0

This major update the Redis™ subchart to its newest major, 15.0.0. Here you can find more info about the specific changes.

To 10.0.0

This major updates the Redis™ subchart to it newest major, 14.0.0, which contains breaking changes. For more information on this subchart’s major and the steps needed to migrate your data from your previous release, please refer to Redis™ upgrade notes..

To 7.0.0

On November 13, 2020, Helm v2 support was formally finished, this major version is the result of the required changes applied to the Helm Chart to be able to incorporate the different features added in Helm v3 and to be consistent with the Helm project itself regarding the Helm v2 EOL.

What changes were introduced in this major version?

  • Previous versions of this Helm Chart use apiVersion: v1 (installable by both Helm 2 and 3), this Helm Chart was updated to apiVersion: v2 (installable by Helm 3 only). Here you can find more information about the apiVersion field.
  • Move dependency information from the requirements.yaml to the Chart.yaml
  • After running helm dependency update, a Chart.lock file is generated containing the same structure used in the previous requirements.lock
  • The different fields present in the Chart.yaml file has been ordered alphabetically in a homogeneous way for all the Bitnami Helm Chart.
  • Several parameters were renamed or disappeared in favor of new ones on this major version:
    • The image objects have been moved to its corresponding component object, e.g: workerImage now is located at worker.image.
    • The prefix airflow has been removed. Therefore, parameters prefixed with airflow are now at root level, e.g. airflow.loadExamples now is loadExamples or airflow.worker.resources now is worker.resources.
    • Parameters related to the git features has completely been refactored:
      • They have been regrouped under the git map.
      • airflow.cloneDagsFromGit no longer exists, instead you must use git.dags and git.dags.repositories has been introduced that will add support for multiple repositories.
      • airflow.clonePluginsFromGit no longer exists, instead you must use git.plugins. airflow.clonePluginsFromGit.repository, airflow.clonePluginsFromGit.branch and airflow.clonePluginsFromGit.path have been removed in favour of git.dags.repositories.
    • Liveness and readiness probe have been separated by components airflow.livenessProbe.* and *airflow.readinessProbe* have been removed in favour of *web.livenessProbe*, *worker.livenessProbe*, *web.readinessProbe* and *worker.readinessProbe*.
    • airflow.baseUrl has been moved to web.baseUrl.
    • Security context has been migrated to the bitnami standard way so that securityContext has been divided into podSecurityContext that will define the fsGroup for all the containers in the pod and containerSecurityContext that will define the user id that will run the main containers.
    • ./files/dags/.py* will not be include in the deployment any more.
  • Additionally updates the PostgreSQL & Redis subcharts to their newest major 10.x.x and 11.x.x, respectively, which contain similar changes.

Considerations when upgrading to this version

  • If you want to upgrade to this version using Helm v2, this scenario is not supported as this version does not support Helm v2 anymore.
  • If you installed the previous version with Helm v2 and wants to upgrade to this version with Helm v3, please refer to the official Helm documentation about migrating from Helm v2 to v3.

Upgrading Instructions

To upgrade to 7.0.0 from 6.x, it should be done reusing the PVC(s) used to hold the data on your previous release. To do so, follow the instructions below (the following example assumes that the release name is airflow and the release namespace default):

NOTE: Please, create a backup of your database before running any of those actions.

  1. Obtain the credentials and the names of the PVCs used to hold the data on your current release:

     export AIRFLOW_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
 export AIRFLOW_FERNET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
 export AIRFLOW_SECRET_KEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-secretKey}" | base64 --decode)
 export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
 export REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
 export POSTGRESQL_PVC=$(kubectl get pvc -l app.kubernetes.io/instance=airflow,app.kubernetes.io/name=postgresql,role=primary -o jsonpath="{.items[0].metadata.name}")

  2. Delete the Airflow worker & PostgreSQL statefulset (notice the option –cascade=false):

     kubectl delete statefulsets.apps --cascade=false airflow-postgresql
 kubectl delete statefulsets.apps --cascade=false airflow-worker

  3. Upgrade your release:

NOTE: Please remember to migrate all the values to its new path following the above notes, e.g: airflow.loadExamples -> loadExamples or airflow.baseUrl=http://127.0.0.1:8080 -> web.baseUrl=http://127.0.0.1:8080.

    helm upgrade airflow bitnami/airflow \
      --set loadExamples=true \
      --set web.baseUrl=http://127.0.0.1:8080 \
      --set auth.password=$AIRFLOW_PASSWORD \
      --set auth.fernetKey=$AIRFLOW_FERNET_KEY \
      --set auth.secretKey=$AIRFLOW_SECRET_KEY \
      --set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
      --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC \
      --set redis.password=$REDIS_PASSWORD \
      --set redis.cluster.enabled=true

  1. Delete the existing Airflow worker & PostgreSQL pods and the new statefulset will create a new one:

     kubectl delete pod airflow-postgresql-0
 kubectl delete pod airflow-worker-0