Service: Airflow - EyevinnOSC/community GitHub Wiki
Apache Airflow (serviceId apache-airflow, catalog page) is an open source workflow orchestration platform for authoring, scheduling and monitoring data pipelines as code. It is a long-running service instance with up to 10 GiB of storage per instance. License: Apache-2.0.
- Define pipelines as Python DAGs (directed acyclic graphs).
- Schedule and monitor runs from the Airflow web UI.
- Trigger runs programmatically through the Airflow REST API.
- Store DAG metadata and task state in a PostgreSQL, MySQL or SQLite metadata database.
Airflow needs a metadata database. Create a PostgreSQL instance first (serviceId birme-osc-postgresql, see PostgreSQL) and confirm it is running before creating Airflow.
- Create PostgreSQL with
create-service-instance,serviceId: "birme-osc-postgresql", config{ "name": "airflowdb", "PostgresPassword": "{{secrets.pgpwd}}" }(create the secret first withcreate-service-secret). Read the address and port withdescribe-service-instance. - Store the connection string as a secret named
dburl(create-service-secret,serviceId: "apache-airflow"). Format:postgres://postgres:<password>@<host>:<port>/postgres. - Create Airflow with
serviceId: "apache-airflow":
{
"name": "myairflow",
"AdminPassword": "{{secrets.airflowadmin}}",
"DatabaseUrl": "{{secrets.dburl}}"
}- Verify with
wait-for-service-readyorget-instance-status; get the URL withdescribe-service-instance. Log in to the web UI as useradminwith the password you set.
| Option | Required | Type | Meaning |
|---|---|---|---|
| name | Yes | string | Name of the instance. Alphanumeric, at most 20 characters. |
| AdminPassword | No | string, sensitive | Password of the admin user for the web UI and administrative operations. Store as a secret and pass {{secrets.NAME}}. |
| DatabaseUrl | No | string, sensitive | Connection string to the metadata database where Airflow stores DAG information and task state. Supports PostgreSQL, MySQL and SQLite. Contains the database password, so keep it in a secret. |
{{secrets.NAME}} must be the entire field value. Overwriting a secret does not restart a running instance.
| Option | Required | Type | Meaning |
|---|---|---|---|
| name | Yes | string | Instance name. |
| PostgresPassword | Yes | string, sensitive | Password of the PostgreSQL superuser. |
| PostgresUser | No | string | Superuser name. Defaults to postgres. |
| PostgresDb | No | string | Default database to create at start. Defaults to the same name as the user. |
| PostgresInitDbArgs | No | string | Extra arguments passed to initdb. |
| PostgresInitDbSql | No | string | SQL commands or script content run at initialization. |
Change the admin password after first login under Security, Users. Trigger a DAG run through the REST API:
curl -X POST \
-H "Content-Type: application/json" \
-u admin:<password> \
https://<instance>.apache-airflow.auto.prod-se.osaas.io/api/v1/dags/<dag_id>/dagRuns \
-d '{"conf": {}}'Example with the OSC CLI:
npx @osaas/cli create apache-airflow myairflow \
-o AdminPassword="{{secrets.airflowadmin}}" \
-o DatabaseUrl="{{secrets.dburl}}"- Storage is limited to 10 GiB per instance.