Daskhub images - gcp4hep/analysis-cluster GitHub Wiki
Client and worker environment
Client environment (e.g. the Jupyter notebook or your python client) and the Dask Worker environment need to match. If you are using any particular package, compatible versions of it need to be installed on both sides. In the case of a LOCAL cluster client and workers run locally on the same environment and you don't need to pay attention on matching environments.
When starting JupyterHub, you will be given the choice of the client image.
When starting the Dask Cluster, you will have to provide matching Worker image to your JupyterHub selection, based on the table below.
from dask_gateway import GatewayCluster
cluster = GatewayCluster(image="xxx/yyy:zzz"...)
...
Available images
We are building on top of the default images available for Daskhub. We are building a selection of images to kickstart the activity and we need to gain some experience with the dependencies and also image generation. Adding new dependencies (in particular with Conda) and uploading it to the image repository requires a lot of patience, so requests for new images need to be streamlined.
This table summarizes the images with matching environments.
| Jupyter image | Name | Description |
|---|---|---|
| eu.gcr.io/gke-dev-311213/jupyter-ml-gpu:20211209 | ML environment | Includes root, tensorflow, keras, flatbuffers, joblib, pillow, pytz, scikit-learn, scipy, energyflow, root-numpy, sklearn, awkward, uproot on top of the default pangeo image. |
| eu.gcr.io/gke-dev-311213/jupyter-physlite:prod | PHYSLITE environment | Includes PHYSLITE SW (numpy h5py numba uproot awkward pyarrow coffea aiohttp). |
Currently we are uploading images to the European GCP image repository (eu.gcr.io/gke-dev-311213/image:tag), so that the images are close to the compute cluster. The process to upload images to GCP image repository is the same as to any other repository, but you will have to install gcloud SDK and configure your docker configuration using https://cloud.google.com/sdk/gcloud/reference/auth/configure-docker.