API for pipelines - rodekruis/IBF-system GitHub Wiki

This documentation is meant to make clear to pipeline owners how to exactly call IBF-system's API correctly.

• This is relevant both for internal (510) pipeline owners and for external pipeline owners.
• This goes further than how to accomplish 1 successful API-call, but aims to make clear which complete "set" of API-calls is needed for a correct working IBF-portal. This differs per disaster-type.
• It also aims to make clear which different scenarios there are. This also differs per disaster type. There is always at least 'trigger' vs. 'no trigger'. But for example disaster type 'typhoon' contains 'event below trigger', 'event without landfall', etc.

1. Pipeline Overview
   1. Flood
   2. Typhoon
   3. Flash flood
   4. Drought
   5. Malaria
2. Timing

The overview flowchart describes the structure of a pipeline run. The implementation details vary per disaster type and are detailed in the text below.

flowchart
    classDef wrap text-wrap: wrap;
    A(start pipeline)
    B[detect events from data sources like GloFAS stations, ECWMF, and river gauges]:::wrap
    C{how many events?}
    D[upload indicators for no events<br/><br/>/admin-area-dynamic-data/exposure]:::wrap
    E[upload indicators per admin level, and lead time<br/><br/>/admin-area-dynamic-data/exposure]:::wrap
    F[upload disaster type indicators like glofas_stations, and typhoon track<br/><br/>/event/alerts-per-leadtime<br/>/typhoon-track]:::wrap
    G[upload non-event data like river gauges, and raster files<br/><br/>/point-data/dynamic<br/>/admin-area-dynamic-data/raster<br/>/lines-data/exposure-status]:::wrap
    H[inform the portal that the upload is complete<br/><br/>/event/process]:::wrap
    I(stop pipeline)
    A --> B
    B --> C
    C -->|zero| D
    C -->|non-zero| E
    subgraph one["<div style="width:20em; height:10em; display:flex; justify-content: flex-start; align-items:flex-start;">for each event</div>"]
    E --> F
    end
    D --> G
    F --> G
    G --> H
    H --> I

• Start a loop per Glofas station, which is agreed to be the definition of an event for floods. It is theoretically possible to change event-definition (to e.g. river basin), but this should be agreed with IBF dev team beforehand.
• Per station/event
  • execute the following logic to identify the leadTime (between 0-day and 7-day) to use in API-calls for this event
    • determine the 1st leadTime above trigger threshold
      • if found: also determine if a warning threshold is already exceeded earlier, and remember this for the POST /event/alerts-per-leadtime call (see 2. below)
    • if nowhere above trigger threshold, determine 1st leadTime above any threshold (so only applicable if multiple thresholds defined for country)
    • if nowhere above any threshold, continue with next station/event (there are some API-calls to do after event-loop, see below)
  • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Upload alert data per leadTime for this event via POST /event/alerts-per-leadtime (see 2. below)
  • Upload Glofas station data for this event via POST /point-data/dynamic (see 3. below)
• After the event loop:
  • [Only if no stations above any threshold] Upload no alert exposure data via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Upload Glofas station data for all other stations via POST /point-data-dynamic (see 2. below)
  • Upload flood extent raster per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 4. below)
  • Inform API that upload is complete and it can process events and send notifications, via POST /events/process (see 5. below)

1. No event
2. Trigger event
3. Medium-warning event
4. Low-warning event
5. Warning-to-trigger event, which exceeds a warning threshold at an earlier lead time than it exceeds the trigger threshold
6. Ongoing event. Any of above events, but with lead time of 0 days.
7. Multi-event. Any combination of above events.

1. POST /admin-area-dynamic-data/exposure > example

   • This is the main API-call which uploads dynamic data per admin-area.
   • This is called once for every eventName, which is agreed to be defined per Glofas station.
   • Within the event-loop, start a loop over every adminLevel defined in IBF-system settings
   • Within this loop, make an API-call per dynamicIndicator defined in IBF-system settings
   • E.g. with 2 events, 3 adminLevels and 5 dynamicIndicators this means 2 * 3 * 5 = 30 API-calls.
   • Always include as dynamic indicator forecast_severity.
     • This has value 0 for no alert.
     • This has value 0.3 for low warning, 0.7 for medium warning and 1 for trigger. These relate to predefined threshold set in IBF-system settings.
     • This should be the case for exactly those admin-areas mapped to Glofas stations with eapAlertClass = min/med (see info for point-data/dynamic API-call below).
   • Also include as dynamic indicator forecast_trigger.
     • This indicates a trigger (value=1) or not (value=0). Other values are not allowed.
     • Currently forecast_trigger=1 should exactly align with forecast_severity=1
     • Theoretically this can be disconnected: forecast_severity=1 with forecast_trigger=0 will then indicate a 'High warning'.
     • The API-call with this indicator should always be done last.
     • If this layer is missing, then false is assumed.
   • Also always include the main exposure indicator (called mainExposureIndicator in disasters.json), e.g. population_affected.
   • Optionally include any other dynamic indicators, e.g. windspeed or population_affected_u5.
   • No alert scenario: this endpoint is also used to indicate a no alert scenario. Call this endpoint after the event-loop:
     • With eventName=null and leadTime=1-day and for each adminLevel
     • Upload forecast_severity layer with value=0 for all admin-areas
     • Upload forecast_trigger layer with value=0 for all admin-areas
     • Upload population_affected layer with value=0 for all admin-area

2. POST /event/alerts-per-leadtime > example

   • This API-call is used to indicate per event for which leadTimes warnings and/or triggers are determined (between 0-day and 7-day)
   • This functionality is only used for Floods at the moment.
   • The forecastAlert property per leadTime is used to signify no alert vs any alert (trigger or warning).
   • The forecastTrigger per leadTime is used to distinguish trigger alerts from warning alerts. This means it should be false for a warning, and true for a trigger.
   • Use these properties to indicate a warning-to-trigger scenario where an observed trigger already exceeds a warning threshold earlier. Do so by setting forecastAlert=true and forecastTrigger=false for the leadTimes where a warning is exceeded and the trigger is not. For the leadTimes where the trigger is exceeded, both are true.

3. POST /point-data/dynamic > example

   • Used for uploading Glofas station dynamic data
   • For Glofas data, a separate API-call must be made for each key: forecastLevel,forecastReturnPeriod,triggerLevel,eapAlertClass
   • eapAlertClass must be one of the values no,min,med or max, where the first means no alert, the last means trigger alert, and the middle categories represent lower threshold (only use this if multiple thresholds are defined for the country). These values should align with forecast_severity=0/0.3/0.7/1 in the exposure endpoint.
   • Note that the pipeline can decide per country on what basis/logic to put stations in a certain eapAlertClass, e.g. based on Glofas probability (which is the case for Zambia currently) or on Return period/water discharge level (which is the case for all other countries).
   • This API-call is made per event/station within the event-loop.
   • Additionally, 1 more API-call with all remaining stations is made after the event-loop. In case of no alerts, this means all stations are in this call.

4. POST /admin-area-dynamic-data/raster/{disasterType} > example

   • Used for uploading the flood extent rasters.
   • The file name should be flood_extent_<leadTime>_<countryCodeISO3>.tif
   • Note that multiple events with the same leadTime should be aggregated into one .tif first. This implies the aggregation and API-calls should be done after the loop over events.
   • Flood extents are only calculated for triggers, not for warnings.
   • The pipeline should also upload an empty raster for all no alert and all warning leadTimes, as otherwise Geoserver will keep showing the last available data from previous pipeline runs for those leadTimes. This means that in case of no alerts an empty raster is uploaded for every leadTime.

5. POST /events/process > example

   • This call starts event processing for given country and disasterType.
   • This involves updating pre-existing events, creating new events, closing old events and - if applicable - sending notifications.
   • This call must always be made wheter there are currently alerts or not.

• The typhoon pipeline also runs per event, but these are defined from the source and not by the pipeline itself.
  • The pipeline can contain logic to skip certain events, because not meeting certain thresholds, and continue with the next.
  • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Upload typhoon track data via POST /typhoon-track (see 2. below)
  • Note that there are multiple relevant types of events. In mock these are defined as:
    • eventTrigger (trigger)
    • eventNoTrigger (warning)
    • eventNoLandfall (track will not make landfall, can be either trigger or warning)
    • eventNoLandfallYet (track is too far away to predict to make landfall or not, can be either trigger or warning)
    • eventAfterLandfall (ongoing event, can be either trigger or warning)
• After event-loop
  • [Only if no events] Upload no events exposure data via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Inform API that upload is complete and it can process events and send notifications, via POST /events/process (see 3. below)

1. POST /admin-area-dynamic-data/exposure > example

   • See floods entry of this endpoint above
   • mainExposureIndicator = houses_affected
   • Trigger events are uploaded via forecast_severity=1 and forecast_trigger=1.
   • Warning events are uploaded via forecast_severity=1 and forecast_trigger=0.
   • Indicate no events scenario
     • with eventName=null, leadTime=72-hour, upload forecast_severity=0, forecast_trigger=0 and houses_affected with any value for all admin-areas.

2. POST /typhoon-track > example

3. POST /events/process > example

   • This call starts event processing for given country and disasterType.
   • This involves updating pre-existing events, creating new events, closing old events and - if applicable - sending notifications.
   • This call must always be made wheter there are currently alerts or not.

1. No events
2. Trigger event
3. Warning event
4. Event without landfall
5. Event where landfall cannot be determined yet
6. Event after landfall (ongoing event)
7. Multi-event

• Pipeline defines events by parent admin-area (so e.g. district/admin-level 2 in Malawi), with eventName=<parentAdminAreaName>
• Per potential event / parent admin-area:
  • Establish the worst leadTime in the configured leadTime window (currently 0-hour to 48-hour for all configured countries)
  • Establish whether the forecast exceeds the (currently single) severity threshold at this leadTime. If not, skip out of this event-loop.
  • Establish whether it is a trigger event or warning event. It can in principal be configured per country how to decide this.
    • E.g. in Malawi it is based on leadTime, if <= 12-hour it is a trigger, otherwise a warning.
    • E.g. in Ethiopia it is configured to always be a warning.
    • Instead of on leadTime it could also be based on severity or accuracy thresholds.
  • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
  • If configured, upload exposure data per point asset (schools, etc.) for this event via POST /point-data/dynamic (see 2. below)
  • If configured, upload exposure data for roads (lines) and buildings (polygons) for this event via POST /lines-data/dynamic (see 3. below)
• After event-loop
  • [Only if no events] Upload no alert exposure data via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Upload relevant rasters (flood depth and/or rainfall forecast) per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 4. below)
  • If configured, upload river gauge data, as this is not event-related, via POST /point-data/dynamic (see 2. below)
  • Inform API that upload is complete and it can process events and send notifications, via POST /events/process (see 5. below)

1. POST /admin-area-dynamic-data/exposure > example

   • See floods entry of this endpoint above
   • Trigger events are uploaded via forecast_severity=1 and forecast_trigger=1.
   • Warning events are uploaded via forecast_severity=1 and forecast_trigger=0.
   • Indicate no alerts with with eventName=null, leadTime=1-hour, upload forecast_severity=0, forecast_trigger=0 and population_affected with any value for all admin-areas.

2. POST /point-data/dynamic > example

   • Call this to upload dynamic attributes of point assets, per pointDataCategory and optionally per leadTime
   • The fids has to match with the fids of the initial (static) set of point assets data.
   • Used for uploading exposure of schools/health sites/waterpoints
     • with key=exposure and pointDataCategory=schools/health sites/waterpoints
     • Only exposed fids are uploaded, not the non-exposed ones.
   • Also used for uploading dynamic attributes of gauges
     • use pointDataCategory=gauges
     • and upload 3 values per gauge with key=waterLevel/waterLevelPrevious/waterLevelReference

3. POST /lines-data/exposure-status > example

   • Call this to upload a set of exposed line assets (which means roads and buildings in practice), per leadTime.
   • The set of exposedFids has to match with the fids of the initial (static) set of point assets data.
   • Only exposed fids are uploaded, not the non-exposed ones.

4. POST /admin-area-dynamic-data/raster/{disasterType} > example

   • The filename for 'flood depth' is defined as flood_extent_<leadTime>_<countryCodeISO3>.tif.
   • The filename for 'rainfall forecast' is defined as rainfall_forecast_<leadTime>_<countryCodeISO3>.tif
   • This means that if there are 2 events with the same leadTime, they have to be combined in the same tif-file first. And therefore this call is made outside of the event-loop.
   • The rasters should always be uploaded with the same bounding box (typically nation-wide), even if the event area is smaller.

5. POST /events/process > example

   • This call starts event processing for given country and disasterType.
   • This involves updating pre-existing events, creating new events, closing old events and - if applicable - sending notifications.
   • This call must always be made wheter there are currently alerts or not.

1. No event
2. Warning event
3. Trigger event
4. Ongoing event
5. Multi-event

• After an event has started, the pipeline will keep uploading it with leadTime = 0-hour for 5 days, without changing the forecast values any more. This means the event will stay visible in the portal long enough to still act upon.
  • If during this period of 5 days, a new or stronger forecast is identified in the same event area with leadTime > 0-hour, this trumps the ongoing event.
  • The pipeline owner could deviate from the number 5, but shouldn't without agreeing first.
• NOTE: that it is agreed that warnings are only uploaded for leadTime of more than 12 hours. The pipeline owner could deviate from this, but shouldn't without agreeing first.
• Only leadTimes 1-12/15/18/21/24/48 hours are defined. If the pipeline calculates un unavailable leadTime, this is rounded down.
• Also non-exposed Traditional Authorities within an exposed district/event are uploaded with forecast_severity=0 and population_affected present.

• Pipeline defines events by region and by season, as configured by droughtSeasonRegions in countries.json.
  • Each country is divided in 1 or more regions, which has 1 or 2 rainy seasons.
  • If it does not rain (enough) in that rainy season, that implies a drought event.
  • eventName should be in the format <season_name>_<region_name>. Example: MAM_National
• Per region and season:
  • Only start forecasting at most 3 months in advance. If start of season further away, no event, continue to next.
  • The leadTime is set by the number of months until the start of the season. If the season starts in March and it is now January, leadTime=2-month.
  • Establish if a warning or trigger is predicted for the first month of the season. If so, set forecast_severity=1 for the appropriate areas. If not, set forecast_severity=0 for all areas.
  • Only include the admin-areas that are part of this region, as documented by droughtRegions in countries.json
  • Upload exposure data per admin-area for this event via POST /admin-area-dynamic-data/exposure (see 1. below)
• After event-loop
  • [Only if no events] Upload no alert exposure data via POST /admin-area-dynamic-data/exposure (see 1. below)
  • Upload rainfall forecast raster per leadTime via POST /admin-area-dynamic-data/raster/{disasterType} (see 2. below)
  • Inform API that upload is complete and it can process events and send notifications, via POST /events/process (see 3. below)

1. POST /admin-area-dynamic-data/exposure > example

   • See floods entry of this endpoint above
   • Trigger events are uploaded via forecast_severity=1 and forecast_trigger=1. It should be configurable per country whether the pipeline uploads triggers or warnings.
   • Warning events are uploaded via forecast_severity=1 and forecast_trigger=0.
   • Indicate no alerts with with eventName=null, leadTime same as calculated in the event scenario, upload forecast_severity=0, forecast_trigger=0 and population_affected with any value for all admin-areas

2. /admin-area-dynamic-data/raster/{disasterType} > example

   • The filename is defined as rain_rp_<leadTime>_<countryCodeISO3>.tif.
   • This means that if there are 2 events with the same leadTime, they have to be combined in the same flood extent tif-file. And therefore this call is made outside of the event-loop.

3. POST /events/process > example

   • This call starts event processing for given country and disasterType.
   • This involves updating pre-existing events, creating new events, closing old events and - if applicable - sending notifications.
   • This call must always be made wheter there are currently alerts or not.

1. No events
2. Trigger events - Raise a trigger event for every season possible, based on current (or specified) calendar month.
3. Warning events - Raise a warning event for every season possible, based on current (or specified) calendar month.
4. Ongoing event

TO COMPLETE

• Events are (hackily) defined by leadTime: 0-month, 1-month and 2-month
• So use leadTime also as eventName, such that the API can interpret it as different events.
• Warning events are not allowed. So use forecast_trigger equal to forecast_severity (either both 0 or both 1).

Frequency of forecasts depends on the type of disaster,