Motivation

• ML Model 실험의 체계적 파악의 필요성
  • 어떤 데이터를 이용해서 학습했는가?
  • 이 모델에서 사용된 Feature는 어떤 것이었는가?
  • 어떤 Parameter를 이용해서 학습했는가?
  • 실험 결과 관리, 모델 버전 관리, 모델 코드 관리
• 해결 방법
  • DB Architecture 밑바닥 부터 구성
  • mlflow와 같이 이미 고안된 솔루션 활용

Overview

• Tracking
  • Code version (from an mlflow project)
  • Start & End Time
  • Source (from an mlflow project)
  • Parameters (key-value)
  • Metrics
    • records and visualize metric's full history
  • Artifacts
    • output files any format.
      • image, model, data files
• Project
  • Name : human-readable
  • Entry Points
    • mlproject files
  • Enviroment
    • include all library dependencies by the project code
    • Conda, Virtualenv, Docker Containers
• Models
  • time_created : UTC ISO 8601 format
  • run_id
  • signature (JSON)
  • input_example
  • databricks_runtime
  • mlflow_version
• Model Registry
  • Model
    • created from an experiment or run that is logged
  • Registered Model
  • ModelVersion
    • Same model name increments the version number
  • ModelStage
    • Staging / Production / Archived
  • Annotations and Descriptions
    • annotate the top-level model , using Markdown (description, relevant information, algo description, dataset employed )

Ref

• Mlops
  • docs : https://github.com/jaeaehkim/awesome-mlops
  • pattern : https://mercari.github.io/ml-system-design-pattern/README_ko.html
  • https://www.youtube.com/watch?v=I-sa8bOcOUk
  • https://speakerdeck.com/mlopskr/mlops-cuncu-jeongug-sidae-jeongri-byeonseongyun?slide=58
  • uber : https://openproceedings.org/2020/conf/edbt/paper_217.pdf
  • google
    • https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning?hl=ko
    • https://research.google/pubs/pub43146/
    • https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
      • review : https://zzsza.github.io/data/2018/01/28/hidden-technical-debt-in-maching-learning-systems/
• Mlflow + ray :
  • https://medium.com/distributed-computing-with-ray/ray-mlflow-taking-distributed-machine-learning-applications-to-production-103f5505cb88
  • https://zzsza.github.io/mlops/2021/01/03/python-ray/
  • https://zzsza.github.io/mlops/2019/01/16/mlflow-basic/
  • https://www.youtube.com/watch?v=H-4ZIfOJDaw
  • https://towardsdatascience.com/feature-factories-pt-2-an-introduction-to-mlflow-873be3c66b66
  • https://towardsdatascience.com/tracking-ml-experiments-using-mlflow-7910197091bb
  • mlflow blog
    • Mlproject : https://dailyheumsi.tistory.com/263
    • Models : https://dailyheumsi.tistory.com/262?category=980484
    • Model registry : https://dailyheumsi.tistory.com/261?category=980484
    • Tracking Server : https://dailyheumsi.tistory.com/260?category=980484
• Async
  • https://www.youtube.com/watch?v=zAvWv_Wi0z0&list=TLPQMjgwNzIwMjI19aJ1ZCV9rw&index=1
  • https://wikidocs.net/115650
  • https://hwangheek.github.io/2019/asynchronous-python/
• python socket
  • https://chichi-story.tistory.com/19?category=884466
  • https://wikidocs.net/115648
  • https://itfordoit.tistory.com/82
• django
  • https://bingbingba.tistory.com/2
• test/log
  • https://hwangheek.github.io/2019/python-logging/
  • https://docs.pytest.org/en/7.1.x/contents.html