Simple Workflow Job Demo and Deployment Modes - pathfinder-analytics-uk/dab_project GitHub Wiki

Links and Resources

• Databricks Asset Bundle Configuration
• Databricks REST API
• Deployment Modes

---

Project Code

notebooks/notebook_1.ipynb

We already created this notebook in a previous lecture. We will reference this in our demo job.

print("This is a notebook")

databricks.yml

You can paste the below configuration block into the databricks.yml file.

resources:
  jobs:
    demo_job:
      name: demo_job
      tasks:
        - task_key: notebook_task
          notebook_task:
            notebook_path: notebooks/notebook_1.ipynb
            source: WORKSPACE
          job_cluster_key: job_cluster
      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 15.4.x-scala2.12
            spark_conf:
              spark.master: local[*, 4]
              spark.databricks.cluster.profile: singleNode
            azure_attributes:
              first_on_demand: 1
              availability: SPOT_WITH_FALLBACK_AZURE
              spot_bid_max_price: -1
            node_type_id: Standard_DS3_v2
            driver_node_type_id: Standard_DS3_v2
            custom_tags:
              ResourceClass: SingleNode
            spark_env_vars:
              PYSPARK_PYTHON: /databricks/python3/bin/python3
            enable_elastic_disk: true
            data_security_mode: SINGLE_USER
            runtime_engine: STANDARD
            num_workers: 0
      queue:
        enabled: true

Adding a name preset to the TEST target

  test:
    mode: production
    presets:
      name_prefix: '[testing] '
    workspace:
      host: your-workspace-url # replace with your Workspace URL
      # We explicitly specify /Workspace/Users/[email protected] to make sure we only have a single copy.
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    permissions:
      - user_name: your-user-name # update with your user name
        level: CAN_MANAGE
    run_as:
      user_name: your-user-name # update with your user name

Commands

databricks bundle deploy

databricks bundle deploy -t TEST