Temp - dennisholee/notes GitHub Wiki

Key Decision Document (KDD): Integration Hub Deployment Architecture

Document ID	KDD-INT-HUB-001
Status	Draft
Author	Enterprise Architecture
Date	2026-07-15
Version	1.0

1. Problem

The Integration Hub must be deployed across multiple geographic and regulatory jurisdictions (initially UK and HK) while ingesting change streams from multiple MongoDB deployments that are logically separated by domain (Party, Relationship, Contact Points, Product Holdings). These independent change streams must converge into a unified event topology (Market → Line of Business → Domain) for global event visibility and downstream consumption. A deployment architecture must be selected that balances data sovereignty, event aggregation fidelity, operational cost, and future expansion capacity.

2. Background

The Integration Hub uses a Control Plane / Worker Plane architecture:

Control Plane: Centralised management service exposing REST APIs. Persists flow configurations in MongoDB. Uses MongoDB Change Streams to broadcast lifecycle commands to worker nodes in real-time. Also handles leader election, node heartbeat monitoring, and cluster coordination — all backed by MongoDB (TTL indexes, atomic lookups).
Worker Nodes: Java 21 (virtual-thread) engines that subscribe to the MongoDB Change Stream, react to configuration changes, and execute Source–Pipeline–Sink flows. Each node polls or listens on assigned sources and routes data through transformers.
Coordination Layer: MongoDB replaces Hazelcast/Redis for distributed locking, leader election, health monitoring, and heartbeats.

Currently, the codebase assumes a single MongoDB URI (application.yml default). Expanding to a federated, multi-region model requires fundamental architectural decisions:

Data Residency: Core domains (Party, Relationship, Contact Points) must remain in the UK. Product Holdings and other relationship-based domains are federated across UK and HK.
Multiple Change Streams: Each MongoDB deployment independently emits change stream events. These must be ingested and converged into a consistent event topology: Market → Line of Business → Domain.
Event Aggregation: Downstream consumers (analytics, data warehouses, audit, real-time dashboards) require a unified, ordered, and enriched view of events regardless of origin region.
Cost & Growth: The solution must be cost-efficient at launch yet support expansion into additional regions (e.g., APAC, EMEA, AMER) without architectural overhaul.

3. Constraints

#	Constraint	Description
C1	MongoDB 6.0+ Dependency	Change Streams (resume tokens, DDL support) and TTL-based heartbeats require MongoDB 6.0+ replica sets.
C2	Data Sovereignty	UK PII / core domain data must not leave the UK jurisdiction. HK Product Holdings data must remain in HK unless legally exempted.
C3	Control Plane Single-Writer Semantics	Leader election and cluster coordination rely on MongoDB atomicity. Only one control-plane leader should actively mutate flows at any time.
C4	Network Latency & Reliability	Cross-region latency (UK↔HK ~200-300ms) impacts synchronous operations. RPO/RTO targets must be achievable within these bounds.
C5	Event Ordering	Events within a single domain (e.g., Party changes in UK) must remain ordered. Cross-domain event ordering is best-effort; a causal consistency model is acceptable.
C6	Change Stream Idempotency	Each change stream cursor is unique to a MongoDB cluster. Aggregation logic must handle duplicate or out-of-order events across clusters.
C7	Cost of Ownership	Infrastructure (compute, network egress, storage), operational complexity (monitoring, deployment, DR), and licensing must fit within projected budget for ≥3 years.
C8	Expansion Capacity	The architecture must support adding new regions (e.g., Singapore, Dubai, EU) with minimal reconfiguration.

4. Assumptions

#	Assumption
A1	MongoDB Atlas (or equivalent managed service) will be used for production to minimise operational overhead. Self-managed replica sets are acceptable for non-prod.
A2	Cross-region network links are private (AWS Direct Connect / Azure ExpressRoute / GCP Interconnect) and meet 99.9% availability SLA.
A3	Event topology consumers tolerate up to 5-second end-to-end latency for cross-region events. Same-region events are sub-second.
A4	The current Integration Hub worker plane can be extended to connect to multiple MongoDB clusters (one per region) without core changes to pipeline execution.
A5	Kubernetes (K8s) will be the deployment substrate in all regions. Helm charts and K8s Operators will be used for lifecycle management.
A6	Authentication & authorisation across regions will use a centralised OIDC provider (e.g., Keycloak / Azure AD) rather than per-region identity stores.

5. Options

5.1 Option Comparison Table

Criteria	Option A: Single Global Deployment	Option B: Regional Deployment (Federated Control Planes)	Option C: Hybrid – Central Control with Regional Workers + Stream Aggregation Gateway
Summary	One centralised Integration Hub stack (single control plane, single MongoDB replica set) deployed in the UK. All event ingestion, transformation, and delivery occurs against a single global MongoDB cluster. Workers may be co-located in UK data centre; remote source connections use VPN/Direct Connect.	Fully independent Integration Hub stacks per region. Each region has its own control plane, worker pool, and MongoDB deployment. Cross-region event sharing is achieved via a separate event bridge (e.g., Kafka MirrorMaker 2 or custom bridge) that asynchronously replicates selected events.	A single central control plane resides in the UK. Each region hosts its own MongoDB cluster (UK core domains + HK/HK federated domains) with local worker nodes. A Stream Aggregation Gateway (SAG) — a lightweight, stateless service — subscribes to all regional MongoDB change streams, normalises events into the Market → LoB → Domain topology, and publishes them to a global event mesh (e.g., Kafka multi-region cluster or Pulsar). Workers and control plane consume from this global mesh.
Design (Deployment Topology)
	Topology Description:	Topology Description:	Topology Description:
	• UK Data Centre (primary region)	• UK Region: Full IH stack (control plane + workers + MongoDB for core domains).	• UK Region: Central Control Plane + UK MongoDB (Party, Relationship, Contact Points) + UK Worker Pool.
	– Single Control Plane (1 pod, K8s)	• HK Region: Full IH stack (control plane + workers + MongoDB for Product Holdings).	• HK Region: HK MongoDB (Product Holdings) + HK Worker Pool. Workers connect to central control plane REST API via private link.
	– Single MongoDB Replica Set (3-node) hosting ALL domains (Party, Relationship, Contact Points, Product Holdings)	• Event Bridge (Kafka MirrorMaker 2 / custom bridge) asynchronously replicates events between UK and Kafka clusters in each region.	• Stream Aggregation Gateway (SAG): Deployed as a sidecar or standalone K8s deployment per region. Each SAG subscribes to its local MongoDB Change Stream, applies topology enrichment (tagging events with Market → LoB → Domain), and publishes to the Global Event Mesh.
	– Worker pool co-located (or remote with cross-region latency to MongoDB)	• Each control plane manages flows only within its region. Global flows require manual coordination or a separate orchestration layer.	• Global Event Mesh: Multi-region Kafka (Confluent / MSK / Strimzi) or Apache Pulsar. Topics are partitioned by topology (Market.LoB.Domain).
	– All change streams originate from one MongoDB cluster		• Workers consume from the global event mesh for cross-region flows; local change streams are consumed directly by SAG for low-latency regional processing.
Pros	• Simplest Architecture: Single control plane, single MongoDB cluster, single set of change streams — minimal operational complexity.	• Full Data Sovereignty: Each region owns and controls its data and control plane — no cross-border data movement without explicit replication.	• Best Balance of Control & Scale: Single logically centralised control plane eliminates cross-region consistency issues. Regional workers + SAG preserve low-latency regional processing.
	• Strong Consistency: Since all domains exist in one MongoDB, cross-domain events (e.g., Party → Product Holding) are trivially consistent and ordered.	• Isolated Failure Domain: A control plane outage in one region does not affect other regions.	• Native Multi-Change-Stream Ingestion: SAG is explicitly designed for this. Each regional change stream is independently consumed and converged into the global event mesh.
	• No Integration Fabric: No need for cross-region event bridges, message replication, or aggregation gateways.	• Compliance-First: Meets the strictest data residency interpretation (no data leaves the region at the storage layer).	• Event Topology Enforcement: SAG normalises events into Market → LoB → Domain at ingress, so downstream consumers always see unified topology.
	• Lowest Initial Cost: Single region infrastructure, no cross-region networking.	• Regional Autonomy: Each region can independently upgrade, restart, or reconfigure its control plane without coordination.	• Expansion-Friendly: Adding a new region = deploy SAG + worker pool + MongoDB. No new control plane. SAG publishes to the global event mesh automatically.
	• Simpler Disaster Recovery: One replica set; failover to a second AZ/region using MongoDB Atlas global clusters.	• No Cross-Region Latency for Local Events: All intra-region processing is sub-second.	• Cost-Effective: Single control plane reduces infrastructure and operational overhead vs. Option B. No need for full stack per region.
Cons	• Data Sovereignty Violation Risk: Storing HK Product Holdings data in a UK MongoDB cluster may breach HK data residency regulations.	• Operational Complexity: N control planes to deploy, monitor, upgrade, and patch.	• Control Plane Single Point of Failure (SPOF): A central control plane outage impacts all regions. Mitigation via active-passive DR.
	• Single Region SPOF: A UK data centre outage takes down the entire Integration Hub globally. Mitigation requires multi-region MongoDB (Atlas Global Clusters, incurring egress costs).	• Event Consistency Challenges: Two independent change streams producing events that must be correlated across regions requires a complex event bridge with conflict resolution, deduplication, and ordering semantics.	• Global Event Mesh Dependency: Requires Kafka/Pulsar multi-region deployment, which has its own operational complexity and cross-region replication latency.
	• Cross-Region Latency: Workers in HK connecting to UK MongoDB suffer ~200-300ms additional latency per operation, impacting real-time ingestion throughput.	• Higher Infrastructure Cost: Duplicate control-plane services, load balancers, ingress controllers per region.	• SAG as New Component: The SAG must be built, tested, and maintained. It introduces a new failure mode (if SAG dies, change stream resumption must be handled).
	• Single Change Stream Bottleneck: All domain events flow through one MongoDB cluster's oplog. High event volume (especially from HK) may cause oplog window pressure and change stream lag for UK consumers.	• Global Flow Orchestration Gap: A flow that spans UK and HK (e.g., enrich Party data from UK with Product Holdings from HK) requires cross-control-plane coordination, which the current codebase does not support. Custom orchestration needed.	• Event Ordering Semantics: Events from different regions must be sequenced by the SAG / event mesh. Causal consistency across regions is non-trivial (e.g., Lamport clocks or Kafka partition key by entity ID).
	• Expansion Cost Grows: Adding a new region still requires all workers to connect to the UK MongoDB cluster — increased network egress costs and latency.	• Costly Expansion: Each new region requires a full Integration Hub stack + new event bridge connections.	• Requires Change Stream Resume Token Management Across Restarts: SAG must persist resume tokens for each regional MongoDB cluster.
Notes	• Atlas Global Clusters with sharding by region could mitigate some data sovereignty concerns but adds cost and complexity.	• The event bridge introduces at-least-once delivery semantics, which may cause duplicate events in the target region. Downstream deduplication is required.	• The SAG can be implemented as a lightweight Spring Boot service using MongoDB's `ChangeStreamPublisher` — minimal code changes to existing worker logic.
	• Workers in HK would use local read preference if a secondary is available, but all writes still go to UK primary.	• This option closely mirrors a "sovereign cloud" pattern used by financial institutions in highly regulated markets.	• The global event mesh topics can be sharded by `market.lob.domain` partition key for ordered event delivery within a topology branch.
	• Not recommended for any deployment that must comply with HK data residency.		• This option is the recommended approach for a scalable, compliant, and cost-effective deployment.
Score	2 / 5	3 / 5	4.5 / 5
Remarks	Feasible only if all data can legally reside in the UK. Violates constraints C2 (HK data residency). Strongly disfavoured for any multi-jurisdiction rollout.	Technically viable but operationally heavy. The event bridge adds latency, complexity, and potential inconsistency. Suitable only for extremely high regulatory separation requirements where even a shared control plane is prohibited.	Strongly recommended. Satisfies all constraints. Single control plane avoids consistency problems. SAG cleanly solves the multi-change-stream convergence requirement. Expansion cost is linear (add region = add SAG + worker + mesh topic). Four remediation actions should be prioritised: (1) resilience for central control plane (active-passive DR), (2) SAG resume token persistence, (3) global event mesh deployment standards, (4) causal consistency design for cross-region events.

5.2 Topology Diagrams

Option A: Single Global Deployment

┌─────────────────────────────────────────────────────┐
│                    UK Data Centre                      │
│  ┌─────────────┐  ┌──────────────────────────────┐   │
│  │ Control      │  │  MongoDB Global Replica Set    │   │
│  │ Plane (x1)   │──│  ┌──────────────────────┐    │   │
│  │ REST API     │  │  │ Party                │    │   │
│  └──────┬───────┘  │  │ Relationship         │    │   │
│         │          │  │ Contact Points       │    │   │
│         │ Change   │  │ Product Holdings     │    │   │
│         │ Stream   │  └──────────────────────┘    │   │
│         ▼          └──────────────┬───────────────┘   │
│  ┌─────────────┐                  │                    │
│  │ Worker Pods │◄─────────────────┘ Change Stream      │
│  │ (UK + HK)   │                                       │
│  └─────────────┘                                       │
└─────────────────────────────────────────────────────┘

Option B: Regional Deployment (Federated Control Planes)

┌──────────────────────────────────┐  ┌──────────────────────────────────┐
│          UK Region                │  │          HK Region                │
│  ┌─────────────┐                  │  │  ┌─────────────┐                 │
│  │ Control     │  ┌────────────┐ │  │  │ Control     │ ┌────────────┐ │
│  │ Plane UK    │──│ MongoDB UK │ │  │  │ Plane HK    │─│ MongoDB HK │ │
│  │ REST API    │  │ Party      │ │  │  │ REST API    │ │ Product    │ │
│  └──────┬──────┘  │ Relationship│ │  │  └──────┬──────┘ │ Holdings   │ │
│         │         │ Contact Pts│ │  │         │         └────────────┘ │
│         ▼         └────────────┘ │  │         ▼                         │
│  ┌─────────────┐                 │  │  ┌─────────────┐                  │
│  │ Worker      │  ┌───────────┐  │  │  │ Worker      │  ┌──────────┐  │
│  │ Pool UK     │  │ Event     │──│──│──│ Pool HK     │  │ Event    │  │
│  └─────────────┘  │ Bridge    │  │  │  └─────────────┘  │ Bridge   │  │
│                   │ (Kafka    │  │  │                    │ (Kafka   │  │
│                   │  MM2)     │──│──│────────────────────│  MM2)    │  │
│                   └───────────┘  │  │                    └──────────┘  │
└──────────────────────────────────┘  └──────────────────────────────────┘

Option C: Hybrid – Central Control + Regional Workers + SAG (Recommended)

                                    ┌─────────────────────────────┐
                                    │     Global Event Mesh        │
                                    │  (Multi-region Kafka/Pulsar) │
                                    │  Topics: Market.LoB.Domain   │
                                    └──────┬──────────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              │                            │                            │
              ▼                            ▼                            ▼
┌─────────────────────────┐  ┌─────────────────────────┐  ┌─────────────────────────┐
│       UK Region          │  │       HK Region          │  │    Future Region (SG)   │
│                          │  │                          │  │                          │
│  ┌─────────────┐        │  │  ┌─────────────┐        │  │  ┌─────────────┐        │
│  │ Control     │        │  │  │ SAG HK      │        │  │  │ SAG SG      │        │
│  │ Plane (x1)  │        │  │  │ (Stream     │        │  │  │ (Stream     │        │
│  │ REST API    │──┐     │  │  │  Aggregator)│        │  │  │  Aggregator)│        │
│  └─────────────┘  │     │  │  │             │        │  │  │             │        │
│                   │     │  │  └──────┬──────┘        │  │  └──────┬──────┘        │
│                   │     │  │         │ Change Stream  │  │         │               │
│                   ▼     │  │         ▼               │  │         ▼               │
│  ┌────────────────────┐ │  │  ┌────────────────────┐ │  │  ┌────────────────────┐│
│  │ MongoDB UK         │ │  │  │ MongoDB HK          │ │  │  │ MongoDB SG          ││
│  │ Party              │ │  │  │ Product Holdings    │ │  │  │ Loans               ││
│  │ Relationship       │ │  │  └────────────────────┘ │  │  │ Investments         ││
│  │ Contact Points     │ │  │                          │  │  └────────────────────┘│
│  └────────────────────┘ │  │                          │  │                          │
│                          │  │                          │  │                          │
│  ┌─────────────┐        │  │  ┌─────────────┐        │  │  ┌─────────────┐        │
│  │ Worker Pool │        │  │  │ Worker Pool │        │  │  │ Worker Pool │        │
│  │ (Local)     │        │  │  │ (Local)     │        │  │  │ (Local)     │        │
│  └─────────────┘        │  │  └─────────────┘        │  │  └─────────────┘        │
│                          │  │                          │  │                          │
│  Workers can consume     │  │  Workers can consume     │  │  Workers can consume     │
│  local Change Stream OR  │  │  local Change Stream OR  │  │  local Change Stream OR  │
│  Global Mesh             │  │  Global Mesh             │  │  Global Mesh             │
└─────────────────────────┘  └─────────────────────────┘  └─────────────────────────┘

5.3 Change Stream & Event Topology Flow (Option C)

┌────────────┐    ┌──────────────┐    ┌───────────────────┐    ┌───────────────────┐
│ MongoDB UK │───▶│ SAG UK       │───▶│ Global Event Mesh │───▶│ Downstream        │
│ Change     │    │ Normalises   │    │ Topic:            │    │ Consumers:        │
│ Stream     │    │ Market.LoB.  │    │ APAC.UK.WEALTH    │    │ • Data Warehouse  │
│ - Party    │    │ Domain       │    │ APAC.UK.INSURANCE │    │ • Real-time Dash  │
│ - Contact  │    │ Tags event   │    │ APAC.HK.WEALTH    │    │ • Audit Log       │
└────────────┘    └──────────────┘    └───────────────────┘    └───────────────────┘

┌────────────┐    ┌──────────────┐
│ MongoDB HK │───▶│ SAG HK       │───▶
│ Change     │    │ Normalises   │
│ Stream     │    │ Market.LoB.  │
│ - Product  │    │ Domain       │
│   Holdings │    │ Tags event   │
└────────────┘    └──────────────┘

Event Envelope (after SAG normalisation):
{
  "market": "APAC",
  "lineOfBusiness": "WEALTH",
  "domain": "PRODUCT_HOLDING",
  "originRegion": "HK",
  "eventType": "INSERT",
  "entityId": "ph-001",
  "payload": { ... },
  "timestamp": "2026-07-15T09:00:00Z",
  "traceId": "abc-123"
}

6. Recommendations

Option C (Hybrid – Central Control with Regional Workers + Stream Aggregation Gateway) is the recommended deployment architecture for the Integration Hub. It satisfies all constraints — data sovereignty (C2), multi-change-stream ingestion (implied by federated MongoDB), event topology convergence, and expansion capacity (C8) — while maintaining a single control plane for operational simplicity and strong event consistency.

6.1 Implementation Roadmap

Phase	Activity	Deliverable	Timeline
1	SAG Design & Prototype	Spring Boot service using MongoDB Change Streams publisher. Resume token persistence in a local MongoDB/Redis. Event topology normalisation logic.	4–6 weeks
2	Global Event Mesh Provisioning	Deploy multi-region Kafka cluster (Confluent or Strimzi Operator on K8s). Define topic naming convention (`{market}.{lob}.{domain}`). Partition key: `{entityId}` for ordering.	4–6 weeks
3	UK Pilot Deployment	Deploy SAG-UK + central control plane + UK MongoDB (Party, Relationship, Contact Points). Validate change stream ingestion → event mesh pipeline.	4 weeks
4	HK Region Onboarding	Deploy SAG-HK + HK MongoDB (Product Holdings). Configure private link to UK control plane. Validate cross-region event convergence.	4 weeks
5	Observability & DR	Prometheus/Grafana dashboards for SAG lag, mesh throughput, control plane health. Active-passive DR for control plane (standby in HK region).	4 weeks
6	Expansion Playbook	Document "Region Addition Runbook" for self-service onboarding of new regions (SG, EU, etc.).	2 weeks

6.2 Risk & Mitigation

Risk	Likelihood	Impact	Mitigation
Central control plane outage	Medium	High	Active-passive DR with MongoDB Atlas Global Cluster for control plane state. DNS failover.
SAG change stream resumption failure	Low	High	Persist resume tokens in a separate MongoDB collection with checkpointing every 100 events. SAG health check endpoint.
Global event mesh cross-region latency	Medium	Medium	Tune Kafka/Pulsar replication factor and min.insync.replicas for latency vs. durability trade-off. Use Pulsar for lower latency if needed.
Cost of egress between regions	Medium	Medium	Use private interconnects (Direct Connect / ExpressRoute). Negotiate egress-free data transfer with cloud provider intra-region.
Event ordering across regions	Medium	Medium	Use composite partition keys that include entity ID. Implement Lamport clocks or KSUID-based ordering for cross-region causal events.

7. Scoring Methodology

Each option was scored on a 1–5 scale across weighted criteria:

Criterion	Weight	Rationale
Data Sovereignty Compliance	25%	Non-negotiable regulatory requirement.
Multi-Change-Stream Ingestion & Convergence	20%	Core functional requirement for federated MongoDB.
Operational Simplicity	15%	Total cost of ownership includes operational burden.
Expansion / Growth Capacity	15%	Must support adding 5+ regions over 3 years.
Event Aggregation & Topology Alignment	15%	Ability to produce a unified Market → LoB → Domain event stream.
Cost Efficiency	10%	Infrastructure + operational cost over 3-year horizon.

Option	Score	Weighted Score
A. Single Global Deployment	2 / 5	2.0
B. Regional Federated Control Planes	3 / 5	2.7
C. Hybrid – Central Control + Regional SAG	4.5 / 5	4.3

Recommendation: Option C.

8. Appendices

8.1 Event Topology Taxonomy (Market → Line of Business → Domain)

Market
├── APAC
│   ├── UK
│   │   ├── Wealth
│   │   │   ├── Party
│   │   │   ├── Relationship
│   │   │   ├── ContactPoint
│   │   │   └── ProductHolding
│   │   ├── Insurance
│   │   │   ├── Party
│   │   │   ├── Relationship
│   │   │   ├── ContactPoint
│   │   │   └── ProductHolding
│   │   └── Banking
│   │       ├── Party
│   │       ├── Relationship
│   │       ├── ContactPoint
│   │       └── ProductHolding
│   └── HK
│       ├── Wealth
│       │   └── ProductHolding
│       └── Insurance
│           └── ProductHolding
└── EMEA (future)
    └── ...

8.2 Glossary

Term	Definition
Control Plane	Centralised management service exposing REST APIs for flow configuration, cluster monitoring, and lifecycle management.
Worker Plane	Distributed Java 21 engine pool that executes Source–Pipeline–Sink flows.
Change Stream	MongoDB feature that provides real-time, ordered stream of data changes on collections.
Stream Aggregation Gateway (SAG)	Lightweight service that subscribes to regional MongoDB Change Streams, normalises events into the event topology, and publishes to the global event mesh.
Global Event Mesh	Multi-region Kafka or Pulsar cluster that acts as the unified event bus across all regions.
Event Topology	Hierarchical classification: Market → Line of Business → Domain. Used to partition events for ordered, route-able delivery.
Resume Token	MongoDB Change Stream cursor token that allows a consumer to resume from the last processed event after a restart.
Event Bridge	Cross-region message replication layer (e.g., Kafka MirrorMaker 2) used in Option B to synchronise events between independent regional stacks.