Home - pathfinder-analytics-uk/dab_project GitHub Wiki

Welcome to the CI/CD with Databricks (Udemy Course) wiki!

Each page contains the code snippets used in the project files in the dab_project.

---

Databricks CLI

• Databricks CLI Workspace Configuration
• Databricks CLI Demo

Getting Started with Databricks Asset Bundles

• Initialising our Bundle Project
• Validating and Deploying the Bundle
• Deploying your Bundle to Different Targets
• Simple Workflow Job Demo and Deployment Modes
• Modularising your Configuration Files
• Destroying a Bundle

Set Up for Local Development

• Installing Databricks Connect
• Creating a DatabricksSession to Run Scripts on the Terminal

Notebook Tasks and Variables with DAB

• Bronze Notebook Walkthrough
• Silver Notebook Walkthrough
• Gold Notebook Walkthrough
• Local Module Imports in Databricks
• Configuring, Deploying and Running the ETL Job
• Configuring Base Parameters for our Notebook Tasks
• Regular Variables with Manual Override
• Regular Variables with Target Override
• Complex Variables

Python Script Tasks with DAB

• Walkthrough of Refactored Code

Distributing your Reusable Code as a Wheel

• Creating a Python Wheel (.whl)
• Installing Wheel Packages in Your Databricks Jobs
• Python Wheel Task

Delta Live Table Pipelines with DAB

• DLT Pipeline Source Notebooks
• Configuring the DLT Pipeline

Local SparkSession Development and Unit Testing

• Local SparkSession
• Introduction to Unit Testing with Pytest
• Adding Unit Tests to our Project
• Running Unit Tests on Databricks Connect
• Centralising Pytest Fixtures with conftest.py
• Coverage Reports

End to End CI/CD Workflows with DAB and GitHub

• Adding our Environment Dependencies in Requirements Text Files
• Introduction to GitHub Actions
• Implementing the CI Workflow (Part 1)
• Service Principal Set Up for Test Deployment
• Service Principal Set Up for Prod Deployment
• Implementing the CD Workflow