Modular Design for Scalable Data Architectures: A Python‐Based Use Case - ivinnyaraujo/dataengineer-datascience-python GitHub Wiki

Modular design is a foundational principle in software development. It promotes maintainability, reusability, and clarity by organising code into logical, self-contained components. As systems grow in complexity, whether in backend services, data engineering pipelines, or machine learning workflow, modularity becomes essential for scalability across teams, environments, and use cases.

Identifying common computational steps and encapsulating them in helper functions is essential for designing scalable and maintainable enterprise data architectures. This approach not only improves readability but also supports testing, debugging, and future extension.

When designing data applications and analytical pipelines, these principles become especially important. Modular design supports clean separation of logic, making it easier to test, refactor, and reuse components across ingestion, transformation, and modeling stages. In medallion architectures, modular functions can be reused across bronze, silver, and gold layers. In machine learning workflows, modularity supports feature engineering, model training, and evaluation pipelines. In notebooks, modular design helps isolate logic for reuse and clarity.

While Python and a .py file is used here for illustration, the concepts apply broadly across languages and platforms.

Example: Walking Speed Calculator

This example uses two files:

a_calculations.py contains reusable functions for unit conversion and speed computation.

a_main.py handles user input and calls the functions to compute and display walking speed.

a_calculations.py

"""Calculation module for walking speed.

This module contains reusable functions for unit conversion and speed computation.
It is designed to be imported by other scripts (e.g., `a_main.py`) and does not
execute any code on its own.

Note:
Both `a_main.py` and `a_calculations.py` must reside in the same project folder
or be accessible via the same local path to ensure successful imports.

Functions:
`feet_to_meters(feet)`: Converts feet to meters
`speed_mps(distance_meters, hours)`: Calculates speed in meters per second
`speed_kph(mps)`: Converts speed from m/s to km/h

Author: Ivy Araujo  
Date: 28/10/2025
"""

def feet_to_meters(feet):
    return feet * 0.3048

def speed_mps(distance_m, hours):
    return distance_m / (hours * 3600)

def speed_kph(mps):
    return mps * 3.6

a_main.py

"""Main module for walking speed calculator.

This script handles user interaction and output formatting. It prompts the user
to enter a distance in feet and a time in hours, then calls functions from the
`a_calculations` module to compute walking speed in meters per second (m/s)
and kilometers per hour (km/h). Results are printed with aligned formatting.

Note:
Both `a_main.py` and `a_calculations.py` must reside in the same project folder
or be accessible via the same local path to ensure successful imports.

Author: Ivy Araujo  
Date: 28/10/2025
"""

from a_calculations import feet_to_meters, speed_mps, speed_kph

def main():
    """Handles user input and displays calculated speeds."""
    distance_feet = float(input("Enter the distance traveled in feet: "))
    duration_hours = float(input("Enter the time taken in hours: "))

    distance_meters = feet_to_meters(distance_feet)
    mps = speed_mps(distance_meters, duration_hours)
    kph = speed_kph(mps)

    label = "Your average speed is: "
    print(f"{label}{mps:.4f} m/s")
    print(f"{'Which is: {kph:.4f} km/h")

if __name__ == "__main__":
    main()

How a_main.py Imports Functions

The line below tells Python to load specific functions from the a_calculations.py file. This works as long as both files are in the same folder or accessible via your Python path.

from a_calculations import feet_to_meters, speed_mps, speed_kph

Applying Modular Design in Notebooks and Cloud Data Workflows

The modular design principles shown in this example are not limited to .py scripts. They apply equally well in notebook-based environments such as Microsoft Fabric, Databricks, and so on.

For instance, in Microsoft Fabric, this modular design across multiple notebooks can be applied by using the %run command to call helper notebooks. For example, reusable functions can be stored in a notebook called utils.ipynb and imported into the main workflow notebook like this:

Conclusion

Modular design is more than a coding style. It provides a foundation for building scalable, testable, and maintainable data solutions. Whether you are creating a command line interface tool, developing a machine learning workflow, or designing a data integration pipeline, separating logic into reusable components brings clarity, supports flexible iteration, and contributes to long term success. This approach encourages code refactoring, fosters collaborative development, and helps ensure consistency across data engineering and analytics teams.