Product Query and Download Guide (PRIP API Reference) - dlr-eoc/prosEO GitHub Wiki
This page introduces the Production Interface Delivery Point (PRIP) API and shows, how it can be used to query products and to download them. For a full description of this API see the ESA document "Production Interface Delivery Point Specification" (ESA-EOPG-EOPGC-IF-3). The sections below provide an abridged summary of this document.
PRIPs are the first point of retrieval for newly published Sentinel data products and, as such, they are co-located with the production service. They make new products available in the agreed packaged unit, with the products only being published once the product package is fully available. The API and standard use scenario allow authorised users to discover, with a simple set of filters, the list of products which need to be retrieved in batches, for example since the last check, and then to perform the downloads.
The nominal use scenario for discovery and download of products from a PRIP consists of the following steps:
- In the first step, the PRIP user uses the API to request a list of products currently available for download from the PRIP by sending a GET Products List Query. As PRIP users are in general systematic downloaders of all or a subset of Sentinel data products, these requests are sent regularly to ensure that a complete picture of all relevant published products is maintained at user-level. The query can include a filter by the content of the product name, and by publication time.
- In response, the PRIP uses the GET Products List Response to provide the user with the requested list of products. For each product on the list, a limited set of product property metadata is provided.
- Following receipt of the query response, the PRIP user can optionally check if any of the products reported in the list have already been retrieved. Then, Product Download Requests are sent to the PRIP via the API for the required products. Products will be downloaded in the pre-defined packages for the particular mission and product type. Please note that requests may be limited by a quota mechanism, e.g. limiting the user a maximum number of connections in parallel or a maximum download volume per time unit (not yet implemented in prosEO).
The Product
and Attributes
entities used in the PRIP API represents a combination of the information found in prosEO's Product
, ProductParameter
and ProductFile
classes. All parameters of a product will be given as attributes in the Product List Response, but some properties of the Product
class will also be provided as attributes in the response body.
The PRIP Product
and embedded Attributes
entities contain the following properties:
-
Id
: A universally unique identifier (UUID). The Id is a local identifier for the product instance within the PRIP, assigned by the Production system. -
Name
: The name of the data file -
ContentType
: The MIME type of the product (alwaysapplication/octet-stream
in prosEO, since prosEO does not differentiate the MIME types of data files) -
ContentLength
: Actual size in bytes (B) of the downloadable file -
OriginDate
: Timestamp recording when the entire downlinked raw data for the downlink session containing the data take is available at the acquisition interface point -
PublicationDate
: Publication date and time of the data file (time at which the file becomes visible to the user) -
EvictionDate
: Date when the data file will be removed from the rolling archive -
Checksum
: List of the known checksums for the product’s physical date, each consisting of:-
(Algorithm
: The checksum algorithm (at least an MD5 checksum will be given) -
Value
: The checksum value (may use uppercase or lowercase characters in hexadecimal values!) -
ChecksumDate
: The date and time when the checksum was computed
-
-
ContentDate
: The sensing range period, consisting of:-
Start
: Sensing start time -
End
: Sensing stop time
-
-
ProductionType
: Type of production, which produced the product, one ofsystematic_production
,on-demand default
oron-demand non-default
-
Footprint
(deprecated): Geographical product footprint as OData LineString or Polygon in string format -
GeoFootprint
: Geographical product footprint as structured object -
Attributes
: Additional product attributes (including all prosEO product parameters), each consisting of:-
Name
: Name of the attribute -
ValueType
: Attribute data type -
Value
: Attribute value
-
The attributes available for a product are mission-specific and usually governed by some sort of product entity description. Please see the mission-specific documentation on which attributes to expect for any given mission product.
Retrieval of products from the PRIP involves a two-step approach:
- Querying the product catalogue,
- Downloading one or more of the products found in the catalogue query response.
The sections below show examples for a catalogue query, a catalogue query response and a product download.
For a systematic retrieval of processed products, a catalogue query by publication date and possibly by product type. A query combining these two elements (assuming a prior retrieval of products published up to 2024-02-01T01:23:45.678Z) and expanding the product attributes could look like this:
https://<hostname>:<port>/proseo/prip/odata/v1/Products?$filter=PublicationDate gt 2024-01-01T01:23:45.678Z and Attributes/OData.CSC.ValueTypeAttribute/any(att:att/Name eq 'productType' and att/Odata.CSC.StringAttribute/Value eq 'PTM_L1B_P1')&$expand=Attributes
Other examples for filter criteria:
-
$filter=startswith(Name,'PTM_TEST_PTM_L1B')
returns all products, where the filename starts withPTM_TEST_PTM_L1B
. Other alternatives arecontains
andendswith
, however these are not recommended for regular use, as they incur a high load on the database. -
$filter=ContentDate/Start gt 2019-05-15T00:00:00.000Z and ContentDate/End lt 2019-05-16T00:00:00.000Z
selects all products with a sensing time interval within the given calendar day 2019-05-15. In the same manner queries for any of the properties given above may be constructed.
Filtering can use the comparison operators eq
(equals), lt
(less than), le
(less or equal), gt
(greater than), ge
(greater or equal) and in
(comparison with a set of values given as something like ('AUX_WAV','AUX_ICE','AUX_WND')
). Filter criteria can be combined using the and
, or
and not
logical operators and parentheses to override the operator precedence.
In addition to filtering other options for managing the returned data are available:
-
$expand=Attributes
: Includes the product attributes (without this option only the regular properties will be available). -
$orderby=PublicationDate desc
: Order the output by the given property, ordering direction can be eitherasc
(ending) ordesc
(ending) -
$top=100
: Return only the first 100 records; this option should always be used to avoid very large return sets (possibly resulting in a429 Too many requests
HTTP status code). -
$skip=400
: Skip the first 400 records; when used in conjunction with$top
this option enables paging through the result set. -
$count=true
: When given, the query response will contain the number of all records matching the selection criteria, independent of the setting of the$top
and$skip
settings, if present.
The metadata for a single product can be retrieved using the query syntax .../odata/v1/Products(<product UUID>)
instead of using the filter condition $filter=Id eq <product UUID>
. In contrast to the filtering syntax this query will not return a list of products with
one element, but a single object.
The example filtering query shown above might yield a response like the following:
{
"@odata.context": "$metadata#Products(Attributes())",
"value": [
{
"Id": "0ec8d282-0a57-439b-b697-96e6430a7e54",
"Name": "PTM_TEST_PTM_L1B_P1_20191104T090000_20191104T104110_03000_2_0.1.0_20240131T151219.nc",
"ContentType": "application/octet-stream",
"ContentLength": 104,
"OriginDate": "1970-01-01T00:00:00Z",
"PublicationDate": "2024-01-31T15:12:34.572Z",
"EvictionDate": "2024-03-01T15:12:19.356Z",
"Checksum": [
{
"Algorithm": "MD5",
"Value": "659CBC82EF7A9F5255AB169039D58A41",
"ChecksumDate": "2024-01-31T15:12:34.126Z"
}
],
"ContentDate": {
"Start": "2019-11-04T09:00:00.2Z",
"End": "2019-11-04T10:41:10.3Z"
},
"ProductionType": "on-demand default",
"Attributes": [
{
"@odata.type": "#OData.CSC.DateTimeOffsetAttribute",
"Name": "beginningDateTime",
"ValueType": "DateTimeOffset",
"Value": "2019-11-04T09:00:00.2Z"
},
{
"@odata.type": "#OData.CSC.DateTimeOffsetAttribute",
"Name": "endingDateTime",
"ValueType": "DateTimeOffset",
"Value": "2019-11-04T10:41:10.3Z"
},
{
"@odata.type": "#OData.CSC.StringAttribute",
"Name": "productType",
"ValueType": "String",
"Value": "PTM_L1B_P1"
},
[...]
{
"@odata.type": "#OData.CSC.StringAttribute",
"Name": "processingLevel",
"ValueType": "String",
"Value": "L1B"
}
]
},
{
"Id": "8c7625a8-09b6-41c6-8eba-9f7dc6b78659",
"Name": "PTM_TEST_PTM_L1B_P1_20191104T104110_20191104T122220_03001_2_0.1.0_20240131T151312.nc",
[...]
},
[...]
]
}
Querying for just one product using the syntax .../odata/v1/Products(0ec8d282-0a57-439b-b697-96e6430a7e54)
would yield:
{
"@odata.context": "$metadata#Products/$entity",
"Id": "0ec8d282-0a57-439b-b697-96e6430a7e54",
"Name": "PTM_TEST_PTM_L1B_P1_20191104T090000_20191104T104110_03000_2_0.1.0_20240131T151219.nc",
"ContentType": "application/octet-stream",
"ContentLength": 104,
"OriginDate": "1970-01-01T00:00:00Z",
"PublicationDate": "2024-01-31T15:12:34.572Z",
"EvictionDate": "2024-03-01T15:12:19.356Z",
"Checksum": [
{
"Algorithm": "MD5",
"Value": "659CBC82EF7A9F5255AB169039D58A41",
"ChecksumDate": "2024-01-31T15:12:34.126Z"
}
],
"ContentDate": {
"Start": "2019-11-04T09:00:00.2Z",
"End": "2019-11-04T10:41:10.3Z"
},
"ProductionType": "on-demand default"
}
The query response may contain any of the following HTTP status codes:
-
200 OK
: Query executed successfully -
400 Bad request
: The OData syntax of the query was incorrect -
401 Unauthorized
: The given user credentials (or OAuth2 token) are invalid or the user is not authorized to access the PRIP API -
404 Not found
: The given URL does not exist (e. g.Products
misspelt), or a query by product UUID was attempted and no product with the given UUID exists. -
429 Too many requests
: The maximum number of records allowed for a single retrieval as configured for the PRIP API is exceeded. -
500 Internal server error
: An internal error occurred in the PRIP service, details can be found in the PRIP service log.
Once the UUID of the desired product has been determined, a product download can be initiated with the request:
https://<hostname>:<port>/proseo/prip/odata/v1/Products(<product UUID>)/$value
This will return a 200 OK
HTTP status code and a data stream with the requested product file, or any of the following HTTP
status codes:
-
400 Bad request
: The OData syntax of the query was incorrect -
401 Unauthorized
: The given user credentials (or OAuth2 token) are invalid or the user is not authorized to access the PRIP API -
404 Not found
: The given URL does not exist (e. g.Products
misspelt), or no product with the given UUID exists. -
429 Too many requests
: The user's download quota for the PRIP API is exceeded (not yet implemented). -
500 Internal server error
: An internal error occurred in the PRIP service, details can be found in the PRIP service log.
The request may use the HTTP Range header to request the product file in individual chunks (e. g. by specifying Range: bytes=0-1023
).
Only byte ranges are allowed. Using small ranges is discouraged, as depending on the backend storage each request may result in
reading the file from the start up to and including the requested range, thereby multiplying the system load on the backend.