VUS Design - illyfrancis/scribble GitHub Wiki

Components /or Streams

With no specific order or dependencies, the overall VUS development effort can be broadly categorized into multiple components/streams.

1. VUS Core

Determine VUS Core process logic.

Approach

Further breakdown by messages

  • VUS 1, 2 or .. 7

Or breakdown by type

  • Trade transactions
  • MF trade transactions
  • Corporate action transactions

Capture design approach for each breakdown and success criteria?

Core features

  • Core logic design from previous plus...
  • VUS Ref Id generation
    • ensuring uniqueness, durable with re-start
  • Persistence and logs

Concerns regarding NFR including:

  • reliability
    • redundancy
    • durable message
    • system recovery
  • capacity
    • scalability - both horizontal and vertical
    • on which component? core only?
  • performance
    • what, where, how and when to measure?
    • establish benchmark?

2. Event Source Adapters

Need to filter and/or transform each input data and supply that to the Core system. Preliminary level of validation on the received data/message may also be needed. And the rule to handle those validation failures. A general approach would be to direct the message to a separate queue for further assessment and analysis.

The source system involved are:

  • RTCR
  • FX Away
  • MFPS
  • InfoM(?) - depends on VUS 5 handling
  • SEB - for VUS 5 & 5 Prov handling

For each system identified, (may) need an adapter (or wraper?) with filter/transformer. The actual integration with respective systems are the concern of the integration stream.

3. Router and Event Sink (???)

There is one event sink which is to direct the output message to the client. When there's only one client the need for having the sink concept may be an overkill but when there're multiple clients the logic of determining who to send the message/data to (i.e. routing) becomes non-trivial.

Is this really needed? To have define a distinct sink vs. having it defined within the core system?

One of the additional responsibility here is the message transformation to the desired format according to the client's specification. The idea is the same message can be transformed and represented as an XML document or JSON etc.

In addition to the message format, the protocol and the transport with which the message is delivered may differ depending on the recipient. The specifics on these concerns however is managed by the integration stream.

Q: Two options, 1) Transform then Route or 2) Route then Transform. Not sure which makes more sense atm.

3.5 Reference data access ???

  • implementation of web service client (REST) or jdbc client (ETL) and provide implementation as a service. (e.g. DualAccountService, ReferenceDataService)

4. Integration

Define the communication channel, queues, (set up of MQ?), and message types between systems. Concerned with message consumption and dispatch on the end-points. Am I Referring to the Camel's message endpoint definition? What about the source then?

  1. Set up of MQ should be an external task and it shouldn't be part of VUS. ???

Integration with reference data. (expand this?)

Client (i.e. SEB)

  • Message transformation (is this really the concern of the integration layer? or should it be moved out/up to adapter?)
  • Defining the connectivity between VUS and the client
  • Design of Reply/Response (or Ack/Nack) paradigm
  • Consideration for durable messages and its implementation detail
  • Handling dead-letter queue
  • Concerned with system re-start

Reference data

  • UAF
  • Static database

For either of these, there are a few integration options:

  1. via RESTful API (Preferred)
  • via message (slow and overkill?)
  • replicate (ETL) database for VUS use only, only the data necessary
  • direct access to underlying data source (NOT preferred)
  • RPC (how?)
  • JCA (not relevant, we're not using JEE container)

Source systems / Upstream systems

  • RTCR
  • FX Away
  • MFPS
  • InfoM(?)
  • SEB

Destination systems

  • InfoM(?)
  • SEB

5. Operational Concerns

General question on this stream is shouldn't this concern be handled consistently and possibly with one solution/system for all P6 projects? If every sub-projects implement its own UI and monitoring solution in a disparate manner, there will be no consistency in its functionality and operations, resulting in mixture of procedures for the operator.

Instead, if the operational concerns are addressed as a separate sub-project whereby it defines a standard approach on how the errors are reported and alerts are raised etc (along with audit and logs), the other sub-projects (like VUS) can conform to it but it need not implement its own UI for it. (and managing users etc)

Monitoring / Alerts

  • Need more detailed requirements
  • Defining alertable events
    • Need to determine the criteria for raising alerts
  • Related to UI

Reporting (only in relation to VUS messages)

  • Identify the requirement and source system(s)

UI

  • What are the requirements?
  • ACL requirements
    • Role based?

Where to put these?

  • Handling message loss?
  • Exceptions?
  • What about tools selection decisions?
    • Build tools, Maven or Gradle, Jenkins
    • Testing framework
    • GraphDB, MongoDB for reporting, Akka for managing state?
    • Camel for EIP?

Things to think about and resolve

  • Come up with use cases (user stories) for finding/querying things, where things might refer to the domain objects such as transactions, vus message, accounts etc
  • From that, think about state transition and correlate to how it might be described in the context of:
    1. actor (akka)
    2. graph (neo)
    3. rel db
    4. or POJO (without actor)
  • Think about how to connect the dots. i.e. there's integration piece but how to funnel that to the start of core process in efficient and non-blocking way?
    • What's the right tool for the job? Akka and Camel for these? Or do we even need to think about at this point? Is this just the concern of message endpoints?