On‐demand Production (ODPRIP) API Reference - dlr-eoc/prosEO GitHub Wiki

On-demand Production (ODPRIP) API Reference

For a full description of the On-demand Production Interface Delivery Point (ODPRIP) API see the ESA document "Production Interface Delivery Point Specification" (ESA-EOPG-EOPGC-IF-3). The sections below provide an abridged summary of this document.

Introduction

ODPRIPs allow clients to trigger the generation of a higher level user product (typically Level 1 or 2) from an archived lower level product (typically Level-0), for those higher level products not available online on the rolling archive. The ODPRIP processes the request, requests and retrieves the applicable archived product from a Long-term Archive, applies the requested processing and makes the output product available for download by the client.

More specifically, this is achieved by creating a processing order, which specifies the product to be used as input, the workflow to apply, and the options if any for customising the workflow. In the context of the On-Demand Processing, a workflow corresponds to a specific version of a product processor with a specific basic configuration applied to it (a "Configured Processor" in prosEO terms).

The nominal scenario can be described in the following steps:

STEP 1: Workflow Query

Following the initial identification of the input product to which on-demand processing is to be applied, the ODPRIP Client may query the ODPRIP Service for relevant Workflows to apply, if this is not already known. The query response provides information on the identified Workflows, including the unique identifier, name, textual description and input and output products types. A set of user options may also be included within a Workflow, which constrain how the requested processing may be modified and override the default options of the Workflow. The default set of options applied are also provided.

STEP 2: Processing Order Submission

Having selected the input product and the Workflow to be triggered plus applicable options, the Client submits a request to create a processing order towards the ODPRIP Service. As well as the parameters just listed, the ProductionOrder may also include an optional priority and an optional endpoint through which the Client can be notified about output product download readiness.

The ODPRIP Service returns a response which, in addition to the input parameters, includes a unique identifier for the particular ProductionOrder, a status (initially queued), a status message and other information.

STEP 3: Order Processing

Following receipt of a processing order, the ODPRIP Service generates a request to retrieve the input product from a suitable Long-term Archive Service. Following retrieval of the input product, the ODPRIP applies the processing and options identified in the processing order and applicable workflow. The output product, once available, is then ingested and published for retrieval via the PRIP API.

STEP 4: Output Product Retrieval

If a notification endpoint was provided, the ODPRIP Service will send an Output Product Download Readiness Notification containing the Product ID of the output product. The Client is then free to submit a product download request. Alternatively, if no endpoint was provided, the Client polls the ODPRIP Service to determine the status of a processing order. Once the status has been updated to completed, the output product can be identified and can be downloaded via the PRIP API.

Entity Description

The data entities used in the interface are simplified versions of the ProcessingOrder and Workflow entities in the prosEO data model.

The Workflow ODPRIP entity contains the following properties:

  • Id: A locally unique identifier for the workflow instance within the ODPRIP
  • Name: Short name of the workflow
  • Description: Textual description of the workflow, including details of the processor version and configuration applicable
  • InputProductType: Product type of the input product
  • OutputProductType: Product type of the output product
  • WorkflowVersion: Version number applicable to the workflow (by convention usually the version number of the processor)
  • WorkflowOptions: A list of client-selectable processing options, specified by:
    • Name: The name of the option
    • Description: Textual description of the option
    • Type: Type of the option (e. g. String)
    • Default: Default value of the option if available
    • Value: An array of all possible values for this option, if it is an enumeration

The ProductionOrder ODPRIP entity contains the following properties:

  • Id: A locally unique identifier for the processing order within the ODPRIP
  • Status: The order status (one of queued, in_progress, completed, failed, cancelled)
  • StatusMessage: Text message providing additional information on the returned status
  • OrderOutputSize: Actual size in bytes of the output product composing the order (only in status completed)
  • SubmissionDate: Date and time at which the processing order was received by the ODPRIP
  • EstimatedDate: Estimated date and time when the product will be available for download
  • CompletedDate: Date and time when the product was available for download
  • EvictionDate: Date when the Product related to the order will be removed from the ODPRIP
  • Priority: Priority of the ProductionOrder. It is an integer from 1-100, where 100 is the highest priority
  • InputProductReference: Complex type used to unambiguously identify the input product; it contains at least one of:
    • Reference: Identifier of the product, typically the product file name
    • ContentDate: The sensing range period, consisting of
      • Start: Sensing start time
      • Stop: Sensing stop time
  • WorkflowId: Unique identifier for the workflow to be applied to the processing order
  • WorkflowName: Short name of the workflow
  • WorkflowOptions: A (possibly empty) list of applicable options from the workflow, each given as:
    • Name: Name of the workflow option
    • Value: Requested value for the option
  • NotificationEndpoint: URI used by the ODPRIP for product download readiness notifications, should these be required (optional)
  • NotificationEpUsername: The username associated with the endpoint URI provided; mandatory if endpoint requires authentication
  • NotificationEpPassword: The password associated with the endpoint URI provided; mandatory if endpoint requires authentication

The output product notification contains the following properties:

  • ProductId: The unique identifier of the product being notified
  • ProductName: The data file name of the product being notified
  • ProductionOrderId: The unique identifier of the ProductionOrder entity being notified
  • NotificationDate: Date and time at which the notification was generated

All times are in UTC in the format YYYY-MM-DDThh:mm:ss.sssZ.

Workflow and Processing Order Request Examples

Workflow Query

Example Request

The following query shows the request for, as an example, the list of processing services (Workflows) supported for a given type of parent (input) product, in this case Sentinel-2 ‘S2MSI1C’ products. It ‘expands’ the result to show the WorkflowOptions.

GET http://<service-root-uri>/odata/v1/Workflows?filter=InputProductType eq 'S2MSI1C'&expand=WorkflowOptions

Example Response

In this example response, a single workflow containing a single option is returned:

{
  "@odata.context": "$metadata#Workflows/$entity",
  "value": [
    {
      "Id": "2b17b57d-fff4-4645-b539-91f305c83z91",
      "Name": "S2_L1C_L2A",
      "Description": "Product processing from Sentinel-2 L1C to L2A. Processor V2.3.6",
      "InputProductType": "S2MSILC",
      "OutputProductType": "S2MSI2A",
      "WorkflowVersion": "1.2",
      "WorkflowOptions": [
        {
          "Name": "ImageFormat",
          "Description": "Image Format option for the production output: GEOTIFF, JPG",
          "Type": "String",
          "Default": "GEOTIFF",
          "Value": [ "GEOTIFF", "JPG" ]
        }
      ]
    }
  ]
}

Processing Order Submission

Example Request

The following example request shows the Client requesting creation of a ProductionOrder via the OData.CSC.Order method. The mandatory InputProductReference is supplied, as well as the WorkflowId, WorkflowName and WorkflowOptions and Priority fields (default values are used if the latter two fields are not supplied).

POST "http://<service-root-uri>/odata/v1/ProductionOrder/OData.CSC.Order"
{
  "InputProductReference":
    {
      "Reference": "S2B_MSIL1C_20191025T085939_N0208_R007_T37VCC_20191025T112031.zip",
      "ContentDate":
        {
          "Start": "2016-01-17T10:34:51.922Z",
          "End": "2016-01-17T10:35:18:872Z"
        }
    }
  "Priority": 50,
  "WorkflowId": "6c18b57d-fgk4-1236-b539-12h305c26z89",
  "WorkflowName": "S2_L1C_L2A",
  "WorkflowOptions": [
    {
      "Name": "ImageFormat",
      "Value": "JPG"
    }
  ],
  "NotificationEndpoint": "https://myservice/notification",
  "NotificationEpUsername": "myserviceuser",
  "NotificationEpPassword": "mypassword"
}

Example Response

The following response is an example of what would be sent in the case that the ProductionOrder request in ii-i is successfully accepted. The response describes the newly created ProductionOrder:

{
  "@odata.context": "$metadata#OData.CSC.Order",
  "value": [
    {
      "Id": "2b17b57d-fff4-4645-b539-91f305c26x53",
      "Status": "queued",
      "StatusMessage": "request is queued for processing",
      "SubmissionDate": "2019-03-27T13:55:12.390Z",
      "EstimatedDate": "2019-03-27T14:02:51.390Z",
      "Priority": 50,
      "InputProductReference":
        {
          "Reference": "S2B_MSIL1C_20191025T085939_N0208_R007_T37VCC_20191025T112031.zip",
          "ContentDate":
            {
              "Start": "2016-01-17T10:34:51.922Z",
              "End": "2016-01-17T10:35:18:872Z"
            }
        },
      "WorkflowId": "6c18b57d-fgk4-1236-b539-12h305c26z89",
      "WorkflowName": "S2_L1C_L2A",
      "WorkflowOptions": [
        {
          "Name": "ImageFormat",
          "Value": "JPG"
        }
      ]
      "NotificationEndpoint": "https://myservice/notification"
    }
  ]
}

Processing Order Retrieval

Example Requests

This request retrieves a particular ProductionOrder by its Id. The response is a single ProductionOrder.

GET https://<service-root-uri>/odata/v1/ProductionOrders('Id')

This request retrieves all current processing orders whose status is completed. This request can be used to identify output products ready for download.

GET https://<service-root-uri>/odata/v1/ProductionOrders?$filter=Status eq OData.CSC.JobStatus'completed'

Example Response

The following shows the ProductionOrder response returned for a status completed:

{
  "@odata.context": "$metadata#ProductionOrder/$entity",
  "value": [
    {
      "Id": "2b17b57d-fff4-4645-b539-91f305c26x53",
      "Status": "completed",
      "StatusMessage": "requested output product is available",
      "OrderOutputSize": 23444841,
      "SubmissionDate": "2019-03-27T13:55:12.390Z",
      "EstimatedDate": "2019-03-27T14:02:51.390Z",
      "CompletedDate": "2019-03-27T14:12:12.400Z",
      "EvictionDate": "2019-04-01T14:12:12.400Z",
      "Priority": 50,
      "InputProductReference":
        {
          "Reference": "S2B_MSIL1C_20191025T085939_N0208_R007_T37VCC_20191025T112031.zip",
          "ContentDate":
            {
              "Start": "2016-01-17T10:34:51.922Z",
              "End": "2016-01-17T10:35:18:872Z"
            }
        },
      "WorkflowId": "6c18b57d-fgk4-1236-b539-12h305c26z89",
      "WorkflowName": "S2_L1C_L2A",
      "WorkflowOptions": [
        {
          "Name": "ImageFormat",
          "Value": "JPG"
        }
      ],
      "NotificationEndpoint": "https://myservice/notification"
    }
  ]
}

Processing Order Output Product Notification

TODO

Product Download

TODO