Creating a DatabricksSession to Run Scripts on the Terminal - pathfinder-analytics-uk/dab_project GitHub Wiki

Links and Resources

• Advanced Usage of Databricks Connect

---

Commands

This command runs the temp.py script

MacOS/Linux

python scripts/temp.py

Windows

python scripts/temp.py

Project Code

Adding the cluster_id key in the configuration file

.databrickscfg

[DEFAULT]
host = your-databricks-host
token = your-databricks-pat
cluster_id = your-cluster-id

Creating a Databricks Session

scripts/temp.py

The below code snippet create a Databricks Session.

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

We can provide a cluster id in the DatabricksSession

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.remote(cluster_id = "insert cluster_id here").getOrCreate()