Service: Airflow - EyevinnOSC/community GitHub Wiki

Airflow

Apache Airflow (serviceId apache-airflow, catalog page) is an open source workflow orchestration platform for authoring, scheduling and monitoring data pipelines as code. It is a long-running service instance with up to 10 GiB of storage per instance. License: Apache-2.0.

What it can do

  • Define pipelines as Python DAGs (directed acyclic graphs).
  • Schedule and monitor runs from the Airflow web UI.
  • Trigger runs programmatically through the Airflow REST API.
  • Store DAG metadata and task state in a PostgreSQL, MySQL or SQLite metadata database.

Creating an instance

Airflow needs a metadata database. Create a PostgreSQL instance first (serviceId birme-osc-postgresql, see PostgreSQL) and confirm it is running before creating Airflow.

  1. Create PostgreSQL with create-service-instance, serviceId: "birme-osc-postgresql", config { "name": "airflowdb", "PostgresPassword": "{{secrets.pgpwd}}" } (create the secret first with create-service-secret). Read the address and port with describe-service-instance.
  2. Store the connection string as a secret named dburl (create-service-secret, serviceId: "apache-airflow"). Format: postgres://postgres:<password>@<host>:<port>/postgres.
  3. Create Airflow with serviceId: "apache-airflow":
{
  "name": "myairflow",
  "AdminPassword": "{{secrets.airflowadmin}}",
  "DatabaseUrl": "{{secrets.dburl}}"
}
  1. Verify with wait-for-service-ready or get-instance-status; get the URL with describe-service-instance. Log in to the web UI as user admin with the password you set.

Options

Option Required Type Meaning
name Yes string Name of the instance. Alphanumeric, at most 20 characters.
AdminPassword No string, sensitive Password of the admin user for the web UI and administrative operations. Store as a secret and pass {{secrets.NAME}}.
DatabaseUrl No string, sensitive Connection string to the metadata database where Airflow stores DAG information and task state. Supports PostgreSQL, MySQL and SQLite. Contains the database password, so keep it in a secret.

{{secrets.NAME}} must be the entire field value. Overwriting a secret does not restart a running instance.

PostgreSQL options (birme-osc-postgresql)

Option Required Type Meaning
name Yes string Instance name.
PostgresPassword Yes string, sensitive Password of the PostgreSQL superuser.
PostgresUser No string Superuser name. Defaults to postgres.
PostgresDb No string Default database to create at start. Defaults to the same name as the user.
PostgresInitDbArgs No string Extra arguments passed to initdb.
PostgresInitDbSql No string SQL commands or script content run at initialization.

Using the instance

Change the admin password after first login under Security, Users. Trigger a DAG run through the REST API:

curl -X POST \
  -H "Content-Type: application/json" \
  -u admin:<password> \
  https://<instance>.apache-airflow.auto.prod-se.osaas.io/api/v1/dags/<dag_id>/dagRuns \
  -d '{"conf": {}}'

Example with the OSC CLI:

npx @osaas/cli create apache-airflow myairflow \
  -o AdminPassword="{{secrets.airflowadmin}}" \
  -o DatabaseUrl="{{secrets.dburl}}"

Limitations

  • Storage is limited to 10 GiB per instance.

Resources

⚠️ **GitHub.com Fallback** ⚠️