Databricks CLI Demo - pathfinder-analytics-uk/dab_project GitHub Wiki

Links and Resources

• Databricks CLI Commands
• Databricks REST API Reference

---

Files

create-compute.json

This file contains the JSON configuration for an all-purpose compute cluster.

{
    "num_workers": 0,
    "cluster_name": "Databricks CLI Demo Cluster",
    "spark_version": "15.4.x-scala2.12",
    "spark_conf": {
        "spark.master": "local[*, 4]",
        "spark.databricks.cluster.profile": "singleNode"
    },
    "azure_attributes": {
        "first_on_demand": 1,
        "availability": "SPOT_WITH_FALLBACK_AZURE",
        "spot_bid_max_price": -1
    },
    "node_type_id": "Standard_DS3_v2",
    "driver_node_type_id": "Standard_DS3_v2",
    "ssh_public_keys": [],
    "custom_tags": {
        "ResourceClass": "SingleNode"
    },
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 20,
    "enable_elastic_disk": true,
    "init_scripts": [],
    "data_security_mode": "SINGLE_USER",
    "runtime_engine": "STANDARD"
}

Commands

Creating Clusters

The following command creates a compute cluster in the Databricks workspace configured in the DEFAULT profile.

It references the json configuration for the cluster in the cluster create command.

databricks clusters create --json @create-compute.json --profile DEFAULT

The following command creates a compute cluster in the Databricks workspace configured in the TEST profile.

databricks clusters create --json @create-compute.json --profile TEST

Listing Clusters

The following lists all clusters in the Databricks workspace configured in the DEFAULT profile.

databricks clusters list --profile DEFAULT

The following lists all clusters in the Databricks workspace configured in the TEST profile.

databricks clusters list --profile TEST

Terminating Clusters

To terminate a cluster you use the databricks clusters delete command and provide the CLUSTER_ID of the cluster your wish to terminate in addition to the profile flag of the target workspace.

Deleting Clusters

To permanently delete a cluster you use the databricks clusters permanent-delete command.