data mesh - ghdrako/doc_snipets GitHub Wiki
Data mesh is an approach, to organizing data architecture (and data teams) that treats data as a product, emphasizes domain ownership of data, provides a self-service data infrastructure, and implements federated governance across the organization. In other words, it’s not just a technical blueprint but also an organizational one.
In a data mesh, responsibility for data is decentralized: each domain or business unit (for example, Sales, Marketing, Finance) owns its data pipelines and datasets as products, delivering them to others in the company who need them. Data as a product means those teams treat their datasets with the same care and quality as a software team treats a customer-facing product, with clear documentation, quality assurances (tests, contracts), and an interface for users, most commonly through
SQL queries but also via APIs or data shares when needed. Consumers of a data product are like customers: they should have a good experience using the data, and their needs drive the product features.
To enable this, data mesh relies on a self-service data platform where a central infrastructure team provides tools and platforms (for ingestion, transformation, storage, and so forth) that domain teams can use easily, without needing deep expertise in the underlying tech. This lowers the barrier for each domain to create and manage data products. For instance, a platform team might provide templated pipelines, standardized CI/CD tooling for data (so that each team can deploy tests and contracts easily), a data catalog for discovery, and so forth.
The fourth principle, federated computational governance, ensures that while domains operate independently, they adhere to some global standards and policies. Instead of a central team manually enforcing rules, governance is embedded into the platform and automated. For example, common security controls, data cataloging requirements, and data contract enforcement can be built into the tooling. A federation of stakeholders (from each domain and central data governance) collaborate to define these standards, balancing autonomy with compliance. For SQL practitioners, this often means standardized testing frameworks, reusable dbt macros, or automated SQL validations that enforce governance rules consistently across domains.
In practice, implementing a data mesh means that SQL-driven solutions might be owned by different teams but made interoperable. One domain’s SQL pipelines produce a data product (say, a cleaned Customer table), which is made discoverable by and shareable with others, possibly via a data sharing mechanism or an API. Another team can use it as a building block in their own SQL models, with a data contract ensuring the schema and quality. The organization ends up with a distributed, but coordinated, data architecture.
Data mesh provides a vision of a large-scale SQL (and general data) architecture where data is treated as a first-class product, owned by domains, supported by common platforms, and governed in a federated way. It connects to modular pipeline design (each product pipeline is independent), emphasizes data contracts (clear interfaces), requires robust automation and CI/CD, and relies on layered architecture and anti-pattern avoidance to scale safely.