Environments - Azure/azureml-assets GitHub Wiki

Files

In addition to the files common to all assets, the following are used for environments.

spec.yaml

An Azure CLI v2 spec file. An environment example would look like this:

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json

description: >-
  A sample environment.

name: "{{asset.name}}"
version: "{{asset.version}}"

build:
  path: "{{asset.repo.url}}#{{asset.repo.commit_hash}}:{{asset.repo.build_context.path}}"
  dockerfile_path: "{{image.dockerfile.path}}"

os_type: linux

tags:
  PyTorch: "1.11"
  GPU: Cuda11
  OS: Ubuntu20.04
  Training: ""
  Preview: ""

Template tags

In addition to the template tags that are common to all assets, environment assets support the following additional ones:

Tag Resolves to
{{asset.repo.build_context.path}} Path to the environment's Docker build context in the release branch
{{image.name}} Name of the environment's Docker image
{{image.context.path}} Path to the Docker build context, relative to the spec file
{{image.dockerfile.path}} Path to the Dockerfile, relative to the Docker build context root
{{image.publish.hostname}} Hostname of the container registry to which the image will be published

environment.yaml

An environment config file provides configuration specific to environments.

image:                          # Image configuration
  name: azureml/curated/my_env  # Name of the environment's Docker image
  os: linux                     # OS type, either linux or windows
  context:                      # Docker build context information
    dir: context                # Directory containing the build context, relative to the environment config file
    dockerfile: Dockerfile      # Dockerfile location, relative to the build context directory. Defaults to Dockerfile if unspecified.
    template_files:             # Optional list of files that contain template tags that should be resolved before building the image
    - Dockerfile                # Template tags in this file will be replaced
  publish:                      # Publishing settings for the image
    location: mcr               # Location to which the image will be published. Must be set to mcr.
    visibility: public          # Visibility of published image. Options are public, internal, staging, or unlisted.

Note that the image name will need to be registered with MCR before the environment is added to azureml-assets. Otherwise, the environment will fail to create in any registry. The image name should largely match the environment name and be prefixed with azureml/curated/.

Template tags

The following template tags are supported:

Tag Normally used in Resolves to Example
{{latest-image-tag}} Dockerfile Either an image tag that with the same digest as latest, or the digest itself if a tag is not found FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:{{latest-image-tag}}
{{latest-image-tag:<some_regex>}} Dockerfile The most recent (descending lexicographical sort) tag that matches the provided regular expression FROM mcr.microsoft.com/azureml/aifx/stable-ubuntu2004-cu113-py38-torch1110:{{latest-image-tag:monthly\.\d{6}}}
{{latest-pypi-version}} requirements.txt or conda_dependencies.yaml The latest version of a package in PyPI azureml-core=={{latest-pypi-version}}
{{latest-pypi-version:pre}} requirements.txt or conda_dependencies.yaml The latest version of a package in PyPI, including prerelease ones azureml-core=={{latest-pypi-version:pre}}

Any of the above tags are resolved if they appear in any of the files listed in an environment config's template_files property.

Asset tags

The tags dictionary contains a list of asset tag names/values that are displayed in the UI. Empty values result in just the tag name being shown. Asset tags that apply to environments are:

Name Example value Description Required
Cuda 11.1.1 CUDA version
CuDnn 8.0.5.39 cuDNN version
DeepSpeed 0.7.3 DeepSpeed version
Inferencing <empty> Supports AzureML inferencing
Nccl 2.8.4 NCCL version
OnnxRuntime 1.12 ONNX Runtime version
OpenMpi 4.1.0 Open MPI version
OS Ubuntu20.04 Operating system name and version X
Preview <empty> Public preview
Python 3.9 Python version X
PyTorch 1.12 PyTorch version
ScikitLearn 1.1 Scikit-learn version
TensorFlow 2.8 TensorFlow version

Testing environment image builds

To test your environment image builds locally, you can do something like the following:

cd <build_context_dir>
docker build . --file <dockerfile>

This isn't going to work if you're using template tags (and you should). An approach that handles template tags is:

cd <azureml-assets_repo>
pip install -e scripts/azureml-assets
cp -r <build_context_dir> /tmp/context
scripts/azureml-assets/azureml/assets/environment/pin_versions.py -i /tmp/context/<dockerfile>
# Repeat the above step for other files you need to update
docker build /tmp/context --file <dockerfile>
⚠️ **GitHub.com Fallback** ⚠️