Airflow dev - zhongjiajie/zhongjiajie.github.com GitHub Wiki

Airflow-dev

You need to take care of when you read Apache Airflow code base.

environment

  • 安装Airflow的时候,需要安装apache-airflow[devl],如果zsh会报错,需要更正的命令是pip install 'apache-airflow[devel]'
  • 在本地使用airflow webserver -p 8080可能会报FileNotFoundError: [Errno 2] No such file or directory: 'gunicorn'这是因为airflow webserver启动时会调用subprocess.Popen创建子进程,webserver使用gunicorn,需要将gunicorn加入到$PATH中: export PATH=${PATH}:/home/admin/python3.6/bin或者建立软链接ln –fs /home/admin/python3.6/bin/gunicorn/bin/gunicorn /bin/gunicorn(如果使用virtualenv,需要找到virtualenv的路径)

code

  • 主要逻辑都集中在~~models.py~~airflow.models文件夹PR-5056重构了models的代码和jobs.py

breeze

  • [最新的Breeze文档]
  • 重新拉取最新的images: ./breeze --force-pull-images --force-build-images

常规操作流程

  • 进入breeze环境: ./breeze
  • 在breeze中启动webserver/scheduler: airflow webserver,此时的本机28080端口就能访问webserver, airflow scheduelr可以启动scheduler
  • 运行某些test: pytest tests/test_core.py
  • 退出breeze: ./breeze -k(不再breeze中运行)
⚠️ **GitHub.com Fallback** ⚠️