External risk event source integration - IBM/intelligence-suite-supply-chain-solutions GitHub Wiki

Weather Alerts Headlines - The Weather Alert Headlines API provides weather watches, warnings, statements and advisories issued by the NWS (National Weather Service), Environment Canada and MeteoAlarm. These weather alerts can provide crucial life-saving information. Weather alerts can be complicated and do not always follow consistent standards, format and rules. The Weather Company (TWC) strives to ensure that the information is consistent from all of the different sources but the content is subject to change whenever there is an update from the authoritative source. The Weather Alert Headline API returns active weather alert headlines related to Severe Thunderstorms, Tornadoes, Earthquakes, Floods, etc . This API also returns non-weather alerts such as Child Abduction Emergency and Law Enforcement Warnings. The Alert Headlines API also provides a key value found in the attribute to access the alerts detail in the Alerts Detail API.


Example

As an example of how to ingest external risk data, we've provide a small python script to use as reference. (Link to the script files)

This script will act as a simple adapter between IBM TWC API and Infohub's COS buckets. Any solution you make for your own purposes can use any external API that you choose.

The main point is that this script's purpose is to take the output of one API, and map the result to infohub-friendly json files that can be uploaded and processed.

Prerequisites

This script, or any that follows the same purpose requires a few things:

  1. A key to access the external API. Provided by you.

  2. A path to the COS bucket you wish to upload to: https://s3.us.cloud-object-storage.appdomain.cloud/.... To look at our official COS API and documentation on how to set one up, visit https://cloud.ibm.com/apidocs/cos/cos-compatibility?code=python. A recommended solution is to manage the cos bucket and authorization through this API. For demo purposes, we're making cos bucket calls explicitly.

  3. An API key to authorize the upload of data to the COS bucket.

  4. (optional) In this example, we are uploading to a tenant directly - so we also require the tenantId.

General outline of an adapter script for COS bucket upload:

  1. Grab data from any external API.
  2. Process data from the external API, storing relevant information.
  3. Iterate through the filtered data, and map each field to a corresponding infohub External Event model field.
  4. Format a file to be filled with your transformed JSON objects, where each line is a complete External Event model entry from step 3.
  5. Upload file to an existing COS bucket.

In our example, the 5 steps is divided into three functions:

  • getData covers steps 1 and 2
  • transformWeatherAtTime covers steps 3 and 4
  • uploadData runs the previous two functions, and covers step 5.

For more details, you can take a look at the examples for guidance.

Making your own adapter.

If you were to use this code as a base in order to set up your own External Event adapter, the process would be pretty simple:

Getting data: TwcGetData.py

  • Make changes to the GET request to accommodate for your chosen API's authorization, url and parameters.
  • Make your own criteria for what External Event data would be worth showing. Change code accordingly.

Transforming data: TwcTransformData.py

  • For each event you wish to transform into External Event data, create a new object and populate each field with whatever you want. Please refer to the implementation for details, and documentation for the External Events model. (link to External Events) NOTE: Every transformed event file consists of flattened JSON objects, where each row is an entire External Event model entry.

Uploading data: weather_poc_scheduler.py

  • Replace the sample bucket url with your own, which you can use to upload data to.
  • Add in your own authorization key in order to upload to the required COS bucket. Change parameters if needed to suit your own purposes, such as tenantId.