Getting Internal Unpublished Plan Data - UN-OCHA/hpc-api GitHub Wiki

Getting Internal Un-published Plan Data (planning and monitoring Caseloads and Indicators)

This guide is for retrieving information relating to plans that may or may not yet be published. This will only work for users that have access permissions that allow them to view this data (such as Operation or Cluster leads for a particular plan).

If you are interested in retrieving published caseload and indicator data, please see the guide Getting Caseload & Indicator Data.

Caveats

The API endpoints presented here are currently the v2 endpoints, which are optimized for data-entry applications like RPM, rather than for consumption by external API users. As a result, they do not currently provide this data in an efficient manner.

This is something that we're in the process of addressing in the design and development of the next version of the HPC API, which you can read more about under the section Upcoming Changes to the API.

This document will be updated once it's possible to retrieve this data using the new API.

If you would like to be notified once this is possible, please see the section Keeping Up-To-Date below.

Authentication

For all the endpoints used in this guide, you will need to authenticate as an HID user, using a Bearer token.

More detailed guidance on how to do this will be provided at a later date, but for now you can use the /jsonwebtoken endpoint with the production server, detailed at https://api.humanitarian.id/docs/#/.

Data Structure

There are a few key data entity types made available in the API endpoints presented in this document that it's important to be able to make sense of to correctly join all the data together.

  • Plan

    E.g. a single Humanitarian Response Plan that is in the HPC.tools system.

    We expect that most people following this guide will be working with respect to a specific plan.

  • Governing Entity

    A Cluster or Sector within a plan.

    These are usually also aligned with a global cluster (which are shared across all plans in HPC.tools), allowing for aggregated analyses to be produced across multiple plans.

  • Plan Entity

    These are entities that help structure the log-frame in a useful way for data organization and data-entry.

    Usually one of the following:

    • A Humanitarian Condition for a Plan
    • A Strategic Objective for a Plan
    • A Specific Objective for a Plan, usually aligned with a Strategic Objective
    • A Sector Objective for a Governing Entity
    • A Sector Activity for a Governing Entity

    Exactly what entities are made available via these endpoints, and how they relate to one another, depends on the internal configuration of a plan.

  • Attachment

    An attachment is a bit of data that can be associated with either:

    • A plan
    • A Governing Entity
    • A Plan Entity

    There are a number of different types of attachments, including:

    • Cost attachments

      (which may or may not be used to calculate the plan's requirements, depending on the costing configuration of the plan)

    • Caseloads

      • Metrics relating to people, such as:
        • Total population
        • People affected
        • People in need
        • People targeted by plan
        • etc...
      • May or may not include a disaggregation, breaking down high-level metrics into locations and demographics.
    • Indicators

      • More general than caseloads, and can be used to track many other things. For examples of the sort of things that are tracked, visit the Indicator Registry.
    • others (TODO)...

  • Monitoring Period

    A period of time for which certain metrics (i.e. caseloads and indicators that have been specified as attachments) should be measured for a particular plan.

    Exactly how many monitoring periods there are for a plan, and which attachments need to be measured, depends on the configuration of the plan in RPM.

    For example, it's common for plans to have 12 (monthly) periods, or 4 (quarterly) periods.

  • Measurement

    Collected monitoring data is called measurements. A measurement is a bit of data that includes measurement metrics (Reached, Cumulative Reach, Covered...etc.) , for a specific attachment, such as caseloads or indicators, in a specific monitoring period.

    Each attachment that needs to be monitored will have an associated measurement entity for each of the monitoring periods that have been "opened" in a plan.

    If the attachment includes disaggregated data, then the measurement may also include data disaggregated in the same way.

Getting the High-Level Data & Structure for a Plan

The URL to use for getting this data is:

https://api.hpc.tools/v2/plan/:planId?content=measurements&disaggregation=false

It requires the ID of the plan we want to get the data for. For details of how to list plans and their IDs, take a look at the guide Getting All Plans for a Given Year.

We specifically include the following query parameters:

  • content=measurements: This indicates to the API that we're interested in getting all of the measurement data across the plan.

    As a result, the API also includes all of the structure (Governing Entities and Plan Entities) in addition to all of the Attachments associated with the plan, so that measurements can be included hierarchically in the appropriate place.

  • disaggregation=false: This tells the API to exclude disaggregated data.

    If we include disaggregated data, the API requests can often take a very long time to request, and either fail or be extremely bloated. Instead, if you are interested in getting disaggregated data, we recommend getting it in chunks that we explain later.

So for example, to get all of the plan structure for the Nigeria 2021 HRP, you can use the URL:

https://api.hpc.tools/v2/plan/1032?content=measurements&disaggregation=false

TODO: explain the result

Getting chunks of measurement data (including disaggregated data)

As explained previously, it is often too much to request all disaggregated data for all measurements across all monitoring periods of a plan at once.

As an alternative, there are various different ways of requesting for subsets of the measurement data that will result in significantly reduced response sizes.

Format of measurement data

In all the strategies listed below, the format of the response is the same, and is just filtered in different ways.

TODO: This document will be updated with an explanation of the format.

Requesting all measurement data for a single monitoring period

Using the ID of one of the monitoring periods returned by the request above (each of which is associated with a single plan), you can request all measurements within that period:

https://api.hpc.tools/v2/measurement?planReportingPeriodId=<monitoring period id>

This will return all measurements, regardless of which attachment it's associated with, or which entity its attachment is associated with.

Requesting all measurement data for a set of attachments

Using the IDs of one or more attachments returned by the request above, you can get the monitoring data for all monitoring periods for a specific set of attachments:

https://api.hpc.tools/v2/measurement?attachmentId=<comma-separated IDs>

E.g, to get all measurements for the attachments 21717 and 22012, use the following request:

https://api.hpc.tools/v2/measurement?attachmentId=21717,22012

Requesting all measurement data directly attached to a plan

You can request all of the measurements for attachments that are directly attached to a plan (and not to a governing entity or plan entity).

https://api.hpc.tools/v2/measurement?objectType=plan&objectId=<plan ID>

e.g:

https://api.hpc.tools/v2/measurement?objectType=plan&objectId=1032

Requesting all measurement data associated with a specific Sector / Cluster

You can request all of the measurements for attachments that are directly attached to a plan's governing entity (i.e. a sector / cluster):

https://api.hpc.tools/v2/measurement?objectType=governingEntity&objectId=<governing entity ID>

e.g:

https://api.hpc.tools/v2/measurement?objectType=governingEntity&objectId=5992

Requesting all measurement data associated with a specific Plan Entity

You can request all of the measurements for attachments that are directly attached to a plan entity (e.g. a strategic objective or sector activity):

https://api.hpc.tools/v2/measurement?objectType=planEntity&objectId=<plan entity ID>

e.g:

https://api.hpc.tools/v2/measurement?objectType=planEntity&objectId=12282

Using Excel Workbook to extract all Monitoring periods for selected plan

We have created an Excel workbook that can be easily configured to automatically download and present monitoring data for your plan in a useful manner.

PowerQuery is used throughout the workbook to download, and re-organize the data provided by the API into useful tables / sheets.

[Download the Excel Workbook](excel/Humanitarian operation planning and monitoring data.xlsx)

You can download the Excel file here: [Plan metrics and monitoring.xlsx](excel/Plan metrics and monitoring.xlsx)

What's inside the Excel Workbook

The workbook presents the following sheets:

  • **0.2 Monitoring Periods: **
  • List all monitoring periods as configured in RPM
  • 1 - Monitoring Totals:
  • All caseloads plan and monitoring caseload metrices including PiN, targets, Baselines and others and indicators for all entities including plan, clusters, strategic objectives, specific objectives, cluster objectives and cluster activities from RPM
  • 2 - Monitoring Disaggregated:
  • All caseloads plan and monitoring caseload metrices including PiN, targets, Baselines and others and indicators for all entities including plan, clusters, strategic objectives, specific objectives, cluster objectives and cluster activities from RPM disaggregated by category and locations

Configuring and Using the Excel Workbook

Follow same steps used to setup and refresh the Bridge Tool under the section Configuring and Using the Excel Workbookhere

⚠️ **GitHub.com Fallback** ⚠️