Data Engineer's Documentation (Wiki Home)

Welcome to our Data Engineer's Wiki! This space centralizes all documentation related to our data engineering workflows, best practices, and project conventions. Use the sections below to navigate through the main topics. The plan is to convert this into a GitHub pages in the near future.

---

Table of Contents

1. Getting Started
   • What’s inside? Essential info for new contributors, covering environment setup, required tools, and an overview of how we use GitHub Wiki or Pages.
2. Source Control
   • What’s inside? Guidelines for managing branches (naming, lifecycle) and commit messages (conventional commits, referencing tasks/issues).
3. Code Practices
   • What’s inside? Standards for notebook layouts, script organization, code documentation (docstrings), and overall coding guidelines.
4. Data Practices
   • What’s inside? Approaches to defining database schemas (DDLs), maintaining a data catalog, and establishing a consistent SQL style and formatting.
5. Project Structure
   • What’s inside? Instructions for setting up README files, organizing folders, and ensuring production-ready scripts (logging, error handling, style conventions).
6. Common General Best Practices
   • What’s inside? High-level guidelines, best coding practices, architecture considerations, and any other cross-cutting principles.
7. Team Reading Catalog
   • What’s inside? A curated list of recommended books, articles, courses, and additional learning resources for continuous improvement.

---