Data Governance - sgml/signature GitHub Wiki
Documentation
DataRails
- https://support.datarails.com/hc/en-us/articles/14445024642204-Creating-a-new-Data-Mapper
- https://support.datarails.com/hc/en-us/articles/14504869410716-Data-Mapper-Field-Types
- https://support.datarails.com/hc/en-us/articles/14521339725084-Data-Mapper-System-Fields
- https://support.datarails.com/hc/en-us/articles/14442192827164-Data-Mapper-Overview
DATABASE DOCUMENTATION TOOLKIT (1996 EDITION)
This guide outlines how to consolidate modular database documentation into a physical, reproducible format using tools available in 1996. It supports traceability, institutional memory, and operational hygiene.
A. ARTIFACT TYPES
A1. UML DIAGRAMS
- Multiple diagrams showing how different parts of the database relate
- Printed on letter-size paper, organized by domain or subsystem
A2. SPREADSHEETS
- Tables listing schema details: table names, data types, constraints, relationships
- Printed in landscape format for readability
- Include version and author in footer
A3. PDF SNAPSHOTS
- Printed versions of database structure at key points in time
- Used for archival reference and compliance
A4. REFERENCE MATERIALS
- Dictionary: defines technical terms
- Glossary: explains domain-specific language
- Thesaurus: maps synonyms and related concepts
A5. CHANGE LOG
- Printed log of updates: who made them, when, and why
- Includes timestamps, initials, and reason for change
B. BINDER STRUCTURE
B1. TAB DIVIDERS
- Use labeled tabs for each artifact type:
- "Diagrams"
- "Schema Tables"
- "Snapshots"
- "Glossary"
- "Change Log"
B2. COVER SHEETS
- Each section begins with a versioned cover sheet:
- Title
- Version number
- Date
- Responsible author
B3. PAGE FOOTERS
- Every page includes:
- Document ID
- Revision number
- Section code (e.g., A2 for Spreadsheets)
C. CROSS-REFERENCING
C1. MASTER INDEX
- Printed index at the front of the binder
- Lists all sections and page numbers
C2. REFERENCE CODES
- Use internal references like:
- "See Table 3.2 in Schema Tables"
- "Refer to Diagram D4 in Section A1"
D. DISTRIBUTION & REDUNDANCY
D1. PHYSICAL COPIES
- Maintain two binders:
- One active copy for daily use
- One archival copy in secure storage
D2. TEAM ACCESS
- Photocopy key sections for individual binders
- Use interoffice mail or courier for remote teams
D3. UPDATE MEMOS
- Include printed memos with update instructions
- Note affected sections and version changes
E. UPDATE PROTOCOL
E1. DOCUMENTATION STEWARD
- Assign one person to manage updates and binder integrity
E2. REQUEST FORMS
- Use printed forms to submit documentation changes
- Include reason, affected section, and proposed revision
E3. MONTHLY REVIEWS
- Schedule monthly binder audits
- Reprint and replace outdated sections
Roles
+-----------------------------+
| CEO |
| - Sets governance vision |
| - Prioritizes compliance |
| - Favors Top-Down models |
+-------------+--------------+
|
v
+-----------------------------+
| Chief Data Officer |
| - Defines governance rules |
| - Balances control vs agility |
| - Coordinates Federated models |
+-------------+--------------+
|
v
+-----------------------------+
| Data Architects |
| - Lineage-aware modeling |
| - Top-Down or Bottom-Up |
| - Document schema & flows |
+-------------+--------------+
|
v
+-----------------------------+
| Data Engineers |
| - Build ingestion pipelines|
| - Often favor Bottom-Up |
| - Post-hoc documentation |
+-------------+--------------+
|
v
+-----------------------------+
| ML/Analytics Teams |
| - Use Adaptive governance |
| - Prioritize experiments |
| - Document features ad hoc |
+-------------+--------------+
|
v
+-----------------------------+
| Business Analysts |
| - Interface with dashboards|
| - Need readable lineage |
| - Rely on Federated clarity|
+-------------+--------------+
|
v
+-----------------------------+
| IT Support / Ops |
| - Maintain infrastructure |
| - Enforce access controls |
| - Reference compliance docs|
+-------------+--------------+
|
v
+-----------------------------+
| Janitor |
| - Main arbitor of security |
| - Cleans up data mess |
| - Ultimate visibility |
+-----------------------------+
Examples
National Student Clearinghouse:
Description: The National Student Clearinghouse offers the NextGen API, allowing institutions to automate transcript ordering and delivery. This API provides a secure, real-time, automated approach to transcript ordering and electronic delivery between the Clearinghouse and your institution's student information system (SIS).
API Documentation: https://help.studentclearinghouse.org/pdp/knowledge-base/submitting-data-files-through-api/
GitHub Repository: https://github.com/NationalStudentClearinghouse
Bureau of Labor Statistics:
Description: The Bureau of Labor Statistics (BLS) provides a Public Data API that allows developers to retrieve published historical time series data in JSON format or as an Excel spreadsheet. The API supports both GET and POST requests and is available in two versions: Version 2.0 (requires registration) and Version 1.0 (open for public use).
API Documentation: https://www.bls.gov/developers/home.htm
GitHub Repository: https://github.com/dsagher/Bureau-of-Labor-Statistics-API-Project
Tools
mit_license_tools:
- name: "OpenRefine"
description: "A powerful tool for working with messy data and improving its quality. It allows users to clean, transform, and enrich data through a user-friendly interface."
url: "http://openrefine.org/"
- name: "Data Quality Tool Kit (DQTK)"
description: "A suite of tools for assessing and improving data quality, including data profiling, data cleansing, and data validation."
url: "https://github.com/open-dq/data-quality-toolkit"
- name: "Apache Griffin"
description: "An open-source Data Quality framework that provides a comprehensive set of tools for data quality management, including data lineage, data quality measurement, and data quality monitoring."
url: "https://griffin.apache.org/"
Videos
- Apache Atlas Introduction: Need for Governance and Metadata Management
- Installation & Configuration of Apache ATLAS Part 2
- Installation & Configuration of Apache ATLAS Part 1
- Data Governance using Apache ATLAS
- Apache Atlas: A Hands-on Course
- Apache Atlas Wiki
Prerequisites
data_governance_limits:
data_quality: "Automation relies on high-quality data. Inaccurate or incomplete data can lead to errors and poor decision-making."
complexity: "Data governance involves complex processes and policies, making automation difficult with diverse data sources and systems."
human_oversight: "Human oversight is necessary for complex decision-making, exception handling, and ensuring compliance with regulations."
integration: "Integrating automated tools with existing systems and processes can be challenging, especially with legacy systems."
scalability: "Maintaining the scalability of automated governance tools as data volumes grow can be difficult."
security: "Ensuring automated processes are secure and comply with data protection regulations is crucial."
Apache Atlas
Comparison Chart
| Tool | License Type | Key Features |
|---|---|---|
| Apache Atlas | Apache License 2.0 | Metadata management, data lineage tracking, data cataloging |
| Amundsen | Apache License 2.0 | Data discovery, metadata management, collaboration tools |
| DataHub | Apache License 2.0 | Data cataloging, metadata management, data lineage tracking |
| Magda | Apache License 2.0 | Data cataloging, metadata management, data lineage tracking |
| Open Metadata | Apache License 2.0 | Metadata management, data cataloging, data lineage tracking |
| Egeria | Apache License 2.0 | Metadata management, data lineage tracking, data cataloging |
| Truedat | Apache License 2.0 | Data cataloging, metadata management, data lineage tracking |
ENV
environment_variables:
- METADATA_CLIENT_HEAP: "1024m"
- JAVA_HOME: "/path/to/your/java"
- LOG_DIR: "/path/to/your/logs"
- METADATA_COLLECTOR_ENABLED: true
- KNOX_ENABLED: true
- LDAP_ENABLED: true
- TLS_ENABLED: true
- KERBEROS_ENABLED: true
- METADATA_OPTS: "-Xmx1024m"