This page introduces aspects to consider when creating a data processor intended for use with prosEO and when integrating existing data processors into prosEO.

• The Job Order File
• Processor Design Consideration
• Creating a prosEO Wrapper for Integration

The application examples presented below are based on the Sample Processor and Sample Wrapper provided in the prosEO source code in the directory <project root>/samples/sample-processor and <project root>/samples/sample-wrapper, respectively.

The main link between the prosEO Production Planner and any data processor being run by prosEO is the Job Order File (JOF). Any processor to be run within the prosEO framework is required to adhere to ESA's Generic IPF Interface Specification (MMFI-GSEG-EOPG-TN-07-0003 or its variant GMES-GSEG-EOPG-TN-09-0016; see Configure Processor Classes, Versions and Configurations). The two reference documents present slightly different versions of the XML structure of a Job Order File, which will be detailed below.

First let's have a look at a simple Job Order File, as it would be fed into the Sample Processor:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<Ipf_Job_Order>
  <Ipf_Conf>
    <Processor_Name>PTML2</Processor_Name>
    <Version>2.0.0</Version>
    <Stdout_Log_Level>INFO</Stdout_Log_Level>
    <Stderr_Log_Level>INFO</Stderr_Log_Level>
    <Test>false</Test>
    <Breakpoint_Enable>false</Breakpoint_Enable>
    <Processing_Station>prosEO Test Mission localhost</Processing_Station>
    <Config_Files>
      <Conf_File_Name>/usr/share/sample-processor/conf/ptm_l2_config.xml</Conf_File_Name>
    </Config_Files>
    <Sensing_Time>
      <Start>20191104_090000200000</Start>
      <Stop>20191104_104110300000</Stop>
    </Sensing_Time>
    <Dynamic_Processing_Parameters>
      <Processing_Parameter>
        <Name>Threads</Name>
        <Value>10</Value>
      </Processing_Parameter>
      <Processing_Parameter>
        <Name>logging.root</Name>
        <Value>notice</Value>
      </Processing_Parameter>
      <Processing_Parameter>
        <Name>Processing_Mode</Name>
        <Value>OPER</Value>
      </Processing_Parameter>
    </Dynamic_Processing_Parameters>
  </Ipf_Conf>
  <List_of_Ipf_Procs count="1">
    <Ipf_Proc>
      <Task_Name>ptm_l2</Task_Name>
      <Task_Version>2.0.0</Task_Version>
      <List_of_Inputs count="1">
        <Input>
          <File_Type>PTM_L1B_P1</File_Type>
          <File_Name_Type>Physical</File_Name_Type>
          <List_of_File_Names count="1">
            <File_Name>/proseo/data/cache/2025/8/28/16/54/PTM_TEST_PTM_L1B_P1_20191104T090000_20191104T104110_03000_2_2.0.0_20250828T165438.nc</File_Name>
          </List_of_File_Names>
        </Input>
      </List_of_Inputs>
      <List_of_Outputs count="1">
        <Output>
          <File_Type>PTM_L2_B</File_Type>
          <File_Name_Type>Physical</File_Name_Type>
          <File_Name>/proseo/data/processor/364176106/PTM_TEST_PTM_L2_B_20191104T090000_20191104T104110_03000_99_2.0.0_20250828T165452.nc</File_Name>
        </Output>
      </List_of_Outputs>
    </Ipf_Proc>
  </List_of_Ipf_Procs>
</Ipf_Job_Order>

Basically a Job Order File consists of a configuration section and a list of one or more processor declarations (depending on the number of tasks defined in the selected processor). It is important to understand, where the elements in the Job Order File come from in prosEO:

As mentioned above, ESA has two documents specifiying the format of Job Order Files with the following differences to the tag names:

The JOF structure is the same for both variants. The applicable variant can be selected by setting the jobOrderVersion attribute of the Processor object to either MMFI_1_8 or GMES_1_1.

There is a newer "Generic Processor ICD" by ESA (ESA-EOPG-EEGS-ID-0083, issue 1.3), but its specifications have not yet been implemented in prosEO.

In order to minimize the integration effort for a data processor, the following recommendations should be kept in mind when designing a processor:

1. Use the Output/File_Type JOF element rather than the Processor_Name element to select the applicable processing algorithm. The Processor_Name element uniquely identifies the processor executable, not individual algorithms within it.
2. The values for Stdout_Log_Level, Stderr_Log_Level and Breakpoint_Enable currently cannot be configured in prosEO, so the default values (INFO and false, respectively) shall be acceptable for the processor.
3. Static data comes in two flavours: Configuration files and static input files. Configuration files are assumed to be comparatively small and may be included in the processor container image, they go into the Config_Files element. "Static input files" may be much larger (e. g. a global Digital Elevation Model), so they should be kept outside of the container image (they can be linked into the container via bind mounts on a common NFS share) and should be defined in the corresponding section of the Configuration object.
4. Dynamically selected input files are provided as Physical file paths, never as directories.
5. If multiple input files are selected for any given product type (File_Type), they will be combined in a single Input element with several file paths in the List_of_File_Names.
6. A processing order requesting multiple output product classes will be split into several job steps, each with exactly one output product class. Therefore the JOF contains exactly one Output element per task (but if several tasks are generated, they all get the same output product specification).
7. The processor may choose between the JOF standards given in either ESA specification (MMFI-GSEG-EOPG-TN-07-0003 or GMES-GSEG-EOPG-TN-09-0016), but the standards must not be mixed or element tags changed.

Not adhering to these recommendations will complicate the processor integration. Except for recommendation (1), which can be handled by the processor configuration, all deviations have to be addressed in the processor wrapper.

For any data processor to be run in prosEO, a wrapper component is required, which provides the link between the (processor-agnostic) prosEO framework and the (prosEO-agnostic) externally developed data processor. (It would of course be possible to develop a processor in such a way that it would integrate seamlessly with prosEO without the need for a separate wrapper, but then the processor could not be run in other environments any more.)

The diagram below shows how the wrapper links the data processor (termed the "Instrument Processing Facility" or IPF by ESA) to the Production Planner on one side and to the Storage Manager on the other:

The IPF Wrapper fulfils the role of the "Management Layer" in ESA's "Generic IPF Specifications" (MMFI-GSEG-EOPG-TN-07-0003). In contrast to the other prosEO components it is highly mission- and processor-specific. Its main task is to communicate with the Storage Manager component to fetch the required input files (and the Job Order File itself) from the backend storage (which typically is not a POSIX file system, but some S3 object storage) and to push the output files to that backend storage. It also communicates with the Ingestor component to update the output product metadata and with the Production Planner to notify it of its completion (successful or otherwise).

One mandatory task of the wrapper is to translate the JOF as received from the Production Planner into a JOF recognizable by the data processor, because the Production Planner sends a somewhat enhanced version of the JOF to the wrapper. For the example described above, the input and output file lists as received from the Production Planner might look like this:

      <List_of_Inputs count="1">
        <Input>
          <File_Type>PTM_L1B_P1</File_Type>
          <File_Name_Type>Physical</File_Name_Type>
          <List_of_File_Names count="1">
            <File_Name FS_Type="S3">s3://backend/2025/8/28/16/54/PTM_TEST_PTM_L1B_P1_20191104T090000_20191104T104110_03000_2_2.0.0-SNAPSHOT_20250828T165438.nc</File_Name>
          </List_of_File_Names>
        </Input>
      </List_of_Inputs>
      <List_of_Outputs count="1">
        <Output Product_ID="56">
          <File_Type>PTM_L2_B</File_Type>
          <File_Name_Type>Physical</File_Name_Type>
          <File_Name FS_Type="POSIX">PTM_TEST_PTM_L2_B_20191104T090000_20191104T104110_03000_99_2.0.0-SNAPSHOT_20250828T165452.nc</File_Name>
        </Output>
      </List_of_Outputs>

You will notice that:

• The File_Name element of the input file has an additional FS_Type attribute with the value S3, and the file path is not a valid POSIX file path, but an S3 URL. This URL is sent to the Storage Manager to retrieve the input product, and in the final JOF the string is replaced by the actual file path in a POSIX file system, where the product has been downloaded to. The (as per the JOF schema invalid) FS_Type attribute is removed.
• The Output element has a Product_ID attribute, which is stored locally by the wrapper and used to find the correct product metadata record to update, when processing is completed. In the final JOF, the attribute is removed.
• The Output/File_Name element only contains a file name, but not a directory path. The absolute path is determined by the wrapper and will be added, when the final JOF is generated.

Since the communication with Storage Manager, Ingestor and Production Planner and the modifications for the final JOF are standardized, a convenience class is provided, which handles all that: The BaseWrapper class. Any concrete wrapper for a specific data processor will extend this base class and will fill in the three extension points provided by it as required.

The flow of control of the BaseWrapper (and hence any wrapper) is as follows (see BaseWrapper.java):

1. Check environment variables (acting as invocation parameters),
2. Fetch the Job Order File from the Storage Manager,
3. Create a Job Order document from the Job Order File,
4. Extension hook (optional): Modify Job Order document for processor operation,
5. Fetch input files and remap file names in Job Order,
6. Extension hook (optional): Modify fetched data and Job Order document for processor,
7. Create Job Order File in file system for container context,
8. Execute data processor,
9. Extension hook (optional): Perform processor-specific updates to the Job Order document,
10. Push Processing Results to prosEO Storage,
11. Register pushed products with prosEO Ingestor,
12. Cleanup temporary files/directories,
13. Report success or failure to Production Planner.

In the case of a "well-behaved" data processor (i. e. one conforming to all rules set out above), no further implementation in the processor-specific wrapper is required. Alas, this is very rarely the case, therefore a typical wrapper will need to do some adaptations in one or more of the extension hooks (steps 4, 6 and 9 above). The basic structure of a processor-specific IPF Wrapper is this:

package de.dlr.proseo.samplewrap;

import java.io.ByteArrayOutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import de.dlr.proseo.basewrap.BaseWrapper;
import de.dlr.proseo.model.enums.JobOrderVersion;
import de.dlr.proseo.model.joborder.JobOrder;

public class SampleWrapper extends BaseWrapper {

    private static Logger logger = LoggerFactory.getLogger(SampleWrapper.class);

    @Override
    protected void preFetchInputHook(JobOrder jobOrderDoc) throws WrapperException {
        ...
    }

    @Override
    protected void postFetchInputHook(JobOrder jobOrderDoc) throws WrapperException {
        ...

        // Save the final JOF in the log for post-mortem analysis
        ByteArrayOutputStream jof = new ByteArrayOutputStream();
        jobOrderDoc.writeXMLToStream(jof, false, JobOrderVersion.valueOf(ENV_JOBORDER_VERSION));

        logger.info("Updated Job Order file:\n" + jof.toString());
    }

    @Override
    protected void postProcessingHook(JobOrder joWork) throws WrapperException {
        ...
    }
}

While not strictly necessary it is recommended practice to log the final Job Order File before invocation of the processor for error diagnosis.

Things to do in these methods (and actually observed in the wild) include:

• For preFetchInputHook:
  • Check availability of all required input,
  • Remove unnecessary input (e. g. a selection rule may return several input products of a specific product type, or several selection rules may provide products of alternative product types, and only one of them must be fed to the processor);
• For postFetchInputHook:
  • Change the File_Type values in the Job Order document (some processors have their own ideas about how the product types are named),
  • Add more output products (remember that prosEO only knows about one, the "primary" output product; everything else is considered "windfall output"),
  • Replace Sensing_Time/Start and Sensing_Time/Stop by 0000000_000000000000000 and 999999_999999999999999, respectively (yes, there are processors which need that),
  • Any other modification of the contents of the Job Order document the processor might require;
• For postProcessingHook:
  • Retrieve the actual output files (because they are not named as prescribed in the JOF),
  • Ingest any "windfall output" (it is not handled by the BaseWrapper code),
  • Extract additional metadata from the output product file(s) and update the product metadata accordingly,
  • If a single output file is required: Create a zipped archive of all the output files,
  • Transfer some output files directly to an external pickup point, where they need to be provided as soon as practicable,
  • Generate processing orders for further processing of the output files (this should become obsolete once proper functionality for automatic order generation is added to prosEO).

Of course this list is by no means complete, and much more complicate integrations have been required over time. But you get the gist ... And if anything goes wrong, log an error message and throw a WrapperException.

To support some of the more common tasks listed above, the BaseWrapper class provides some convenience methods for its subclasses (see BaseWrapper Javadocs):

• protected RestProduct retrieveProductMetadata(InputOutput output) throws WrapperException
• protected void updateProductMetadata(InputOutput output, RestProduct productMetadata) throws WrapperException
• protected IngestorProduct createIngestorProduct(Path outputFilePath)
• protected void ingestProduct(IngestorProduct product)
• protected String extractProseoMessage(HttpResponseInfo responseInfo)

In addition, some of the BaseWrapper's environment variables are accessible for the processor-specific wrappers:

The IPF Wrapper container image is built from the container image of the data processor (IPF) like so (note that in our sample environment the data processor is a Java program, too, which is unusual):

FROM localhost:5000/proseo-sample-processor:1.2.0
WORKDIR /usr/share/proseo-sample-wrapper

ADD target/proseo-sample-wrapper-jar-with-dependencies.jar proseo-sample-wrapper.jar

# Timeout if other prosEO services do not respond within 10 min
ENV HTTP_TIMEOUT=600

ENV PROCESSOR_SHELL_COMMAND="java -jar /usr/share/sample-processor/proseo-sample-processor.jar"

ENTRYPOINT ["java", "-jar", "/usr/share/proseo-sample-wrapper/proseo-sample-wrapper.jar", "de.dlr.proseo.samplewrap.SampleWrapper"]