Azure Platform - BKJackson/BKJackson_Wiki GitHub Wiki

Azure for Data Science

Team Data Science Process Documentation - The front page for the whole section in the MS Azure Docs for Machine Team Data Science Process for Developer Operations - High-level table with lots of links to more references and Azure features/products
A Data Science Microsoft Project Template You Can Use in Your Solutions - Blog post by Buck Woody The Keys to an Effective Data Science Project – Part 1: The Team Data Science Process - Blog post by Buck Woody, starts out with the same Havard CS data science workflow chart in my pres.
Individual Contributor tasks for a data science team -

Building and Deploying your Code with Azure Pipelines

Enabling CI/CD for Machine Learning project with Azure Pipelines - by Azure DevOps Labs, May 15, 2019
How to bring your Data Science Project in production - Using Azure Databricks with Spark, Azure Machine Learning Service and Azure DevOps
Breaking the wall between data scientists and app developers with Azure DevOps
CI/CD with Azure Machine Learning service - Azure DevOps how-to-guide docs
Building and Deploying your Code with Azure Pipelines - Video tutorial, 9 minutes
Pushing docker images to Azure Container Registry
Using Docker Compose with Azure
Deploy to an Azure Web App for Containers
Running a self-hosted agent in Docker
Azure Pipelines agents - To build your code or deploy your software using Azure Pipelines you need at least one agent.

Azure VMs tips

Get list of vm sizes available in your region

az vm list-skus -o tsv -l westus

Containers

Using Docker with notebooks and AzureML

From NBSETUP.md in the Azure Machine Learning Notebooks module:

You need to have Docker engine installed locally and running. Open a command line window and type the following command.

Note: We use version 1.0.10 below as an exmaple, but you can replace that with any available version number you like.

# clone the sample repoistory
git clone https://github.com/Azure/MachineLearningNotebooks.git

# change current directory to the folder 
# where Dockerfile of the specific SDK version is located.
cd MachineLearningNotebooks/Dockerfiles/1.0.10

# build a Docker image with the a name (azuremlsdk for example) 
# and a version number tag (1.0.10 for example).
# this can take several minutes depending on your computer speed and network bandwidth.
docker build . -t azuremlsdk:1.0.10

# launch the built Docker container which also automatically starts
# a Jupyter server instance listening on port 8887 of the host machine
docker run -it -p 8887:8887 azuremlsdk:1.0.10

Now you can point your browser to http://localhost:8887. We recommend that you start from the configuration.ipynb notebook at the root directory.

If you need additional Azure ML SDK components, you can either modify the Docker files before you build the Docker images to add additional steps, or install them through command line in the live container after you build the Docker image. For example:

# install dataprep components
pip install azureml-dataprep

# install the core SDK and automated ml components
pip install azureml-sdk[automl]

# install the core SDK and model explainability component
pip install azureml-sdk[explain]

# install the core SDK and experimental components
pip install azureml-sdk[contrib]

Azure DevOps CI/CD Git Pipelines

Creating a CI/CD pipeline with Azure Pipelines and Google Kubernetes Engine - In this tutorial, you learn how to use Azure Pipelines (previously called Visual Studio Team Services), Google Kubernetes Engine (GKE), and Container Registry to create a continuous integration/continuous deployment (CI/CD) pipeline. The tutorial uses the ASP.NET MusicStore web application, which is based on ASP.NET Core.

Need two pipelines:

  1. Build pipeline - for branches
  2. Release pipeline - for dev/stage/production environments

Azure YAML Files

Used to customize the Azure build.