Home - pathfinder-analytics-uk/dab_project GitHub Wiki
Welcome to the CI/CD with Databricks (Udemy Course) wiki!
Each page contains the code snippets used in the project files in the dab_project
.
Databricks CLI
Getting Started with Databricks Asset Bundles
- Initialising our Bundle Project
- Validating and Deploying the Bundle
- Deploying your Bundle to Different Targets
- Simple Workflow Job Demo and Deployment Modes
- Modularising your Configuration Files
- Destroying a Bundle
Set Up for Local Development
Notebook Tasks and Variables with DAB
- Bronze Notebook Walkthrough
- Silver Notebook Walkthrough
- Gold Notebook Walkthrough
- Local Module Imports in Databricks
- Configuring, Deploying and Running the ETL Job
- Configuring Base Parameters for our Notebook Tasks
- Regular Variables with Manual Override
- Regular Variables with Target Override
- Complex Variables
Python Script Tasks with DAB
Distributing your Reusable Code as a Wheel
Delta Live Table Pipelines with DAB
Local SparkSession Development and Unit Testing
- Local SparkSession
- Introduction to Unit Testing with Pytest
- Adding Unit Tests to our Project
- Running Unit Tests on Databricks Connect
- Centralising Pytest Fixtures with conftest.py
- Coverage Reports