Enterprise Data Transfers & Governance

Large enterprises have thousands of interfaces, exchanging hundreds of thousands of types of messages. Enterprises will be unable to deliver Artificial Intelligence(AI) without Information Architecture(IA).

There's no AI without IA says IBM - Think 2019.

The question for many enterprises is, how do they extract the meta data about the hundreds of thousands of data transfers (transactions & file transfers) without interrupting the interfaces and business as usual? Many large corporations have the traditions that have been established upheld by personnel for more than a couple of decades, and data revolution & data governance ship has not arrived. Let's take for example one of the shops where, hundreds of applications utilize IBM's message oriented middleware(MoM) called, IBM MQ, exchanging hundreds of thousands of messages. Here's an example of a solution (architecture) that can help in a non-intrusive fashion and extract the meta-data & iteratively deliver the complete information necessary to deliver information architecture.

Keep in mind that this technique can be extended to all types of data integration middleware from managed file transfer to Web services message oriented middleware. There are also a number of limitations where one do not have the luxury to modify third party data producers to attach rich meta data, and the data integration middleware may be a third party service or hosted on a tamper proof appliance that allows access, only through APIs.

In this particular example, the meta-data is extracted by periodically connecting to the message transport & browsing or sniffing the messages to extract the metadata attached to the payload. (analogies from rocket science). The shop is open-source savvy and utilizea Kafka to store & stream data & ELK stack to run stream analytics. Once the meta data is identified including the structure of the data, it's then sent to Information Governance Catalog where the data governance committee can have a visibility into data architecture & data security. I'll be uploading the data model (AVRO schema for JSON) that can be utilized to store & transport the meta-data.