Workflows - 52North/IlwisCore GitHub Wiki

A workflow composes existing operations and integrates into Ilwis transparently as normal operation. That is, a workflow instance can be used as a normal operation and can also be combined within other workflows. A Workflow can be created as a re-usable processing template or just can be run on pre-defined data to produce certain output.

More Details

There are two kind of operations: a) SingleOperation and b) WorkflowOperation.

Ilwis ships lots of pre-defined operations. Each operation has its own implementation, for example resample, binarymathraster, etc. (there are further operations in other modules, e.g. the rasteroperations module. In short, there is a straight interface so that further operations can be added easily. However, when complex operations and modelling comes into play almost always particular operations may be missing.

Besides implementing custom operations, Ilwis shares the workflow concept which gives all needed flexibility to solve any complex spatial-temporal problem by composing existing operations and workflows. The Ilwis UI provides capabilities to create and edit workflows visually.

Workflow API

A Workflow implements the OperationMetadata interface, so it integrates into Ilwis transparently. It is responsible to create the formal operation description and registering itself in the mastercatalog. The commandhandler then sees and can treat a workflow instance just as a normal operation, like for example the predefined resample operation. From the workflow's metadata the WorkflowOperationImplementation executes the workflow via the commandhandler.

A workflow is created as a bi-directed graph (yes, loops are allowed) at the backend. Each operation is described as a WorkflowNode with corresponding WorkflowNodeProperties. These nodes can be linked via WorkflowEdges. An edge links a particular operation output to some particular input of another operation, which can be the same (via ilwis loop/if operations).

There are three different ways to assign input to a workflow node:

  1. assign external data (an explicit input assignment). Required parameters are given an input assignment automatically, either update them or just override
  2. linking an operation's output to an operation's input (an implicit input assignment)
  3. no assignment at all (optional parameters)

Assigned inputs can take values from the beginning. If this is the case for an input node the workflow treats it as a node with constant input. The input parameter won't be part of the formal syntax of the workflow operation then.

Example

Here's an c++ code example how to create an NDVI caculation (NDVI=(NIR-VIS)/(NIR+VIS)) as an Ilwis Workflow. The intended workflow can be depicted as follows:

workflow_diagram.png

Step-by-step instruction:

  1. Create Workflow instance and set its operation's metadata (longname, keywords)

    OperationResource operation({"ilwis://operations/ndvi"}, itWORKFLOW);
Ilwis::IWorkflow ndviWorkflow(operation);
ndviWorkflow->setLongName("NDVI Workflow");
ndviWorkflow->setKeywords({"operation, workflow, ndvi"});

  2. Start adding operations needed to calculate the NDVI. Operations to add, substract or divide raster pixels are implemented by the BinaryMathRaster class. A valid executable operation to add two rasters would be for example binarymathraster(raster1,raster2,add). As for each operation the binary operand will be the same, we assign it as constant operation input.

    The actual OperationImplementation is identified by its global ilwis id, so we add it as node property.

    QUrl binaryRaster = QUrl("ilwis://operations/binarymathraster");
quint64 binaryOperationId = mastercatalog()->url2id(binaryRaster, itSINGLEOPERATION);

WorkflowNodeProperties dividentProperties;
dividentProperties.id = binaryOperationId;
WorkflowNode ndviDividentNode = ndviWorkflow->addOperation(dividentProperties);
SPAssignedInputData difference = ndviWorkflow->assignInputData({ndviDividentNode, 2});
difference->value = "substract"; // constant assignment

    The {ndviDividentNode, 2} constructs an InputAssignment implicitely, which is a simple typedef for std::make_pair(ndviDividentNode, 2).

    Do the same to calculate NDVI divisor and the actual ratio:

    // ndvi divisor
WorkflowNodeProperties divisorProperties;
divisorProperties.id = binaryOperationId;
WorkflowNode ndviDivisorNode = ndviWorkflow->addOperation(divisorProperties);
SPAssignedInputData sum = ndviWorkflow->assignInputData({ndviDivisorNode , 2});
sum->value = "add"; // constant assignment

// ndvi ratio
WorkflowNodeProperties ratioProperties;
ratioProperties.id = binaryOperationId;
WorkflowNode ndviRatioNode = ndviWorkflow->addOperation(ratioProperties);
SPAssignedInputData ratio = ndviWorkflow->assignInputData({ndviRatioNode, 2});
ratio->value = "divide"; // constant assignment

  3. Now we want to link the operations to get a workflow calculation. We need to 1) create the edge and 2) define which output index maps to what input index.

    WorkflowEdgeProperties divisorRatioEdgeProperties;
divisorRatioEdgeProperties.outputIndexLastOperation = 0;
divisorRatioEdgeProperties.inputIndexNextOperation = 0;
ndviWorkflow->addOperationFlow(ndviDividentNode, ndviRatioNode, divisorRatioEdgeProperties);

WorkflowEdgeProperties dividentRatioEdgeProperties;
dividentRatioEdgeProperties.outputIndexLastOperation = 0;
dividentRatioEdgeProperties.inputIndexNextOperation = 1;
ndviWorkflow->addOperationFlow(ndviDivisorNode , ndviRatioNode, dividentRatioEdgeProperties);

  4. We declare expected input data for the divident via

    SPAssignedInputData nirInput = ndviWorkflow->assignInputData({ndviDividentNode, 0});
SPAssignedInputData visInput = ndviWorkflow->assignInputData({ndviDividentNode, 1});

    As divisor and divident operations share the same inputs, we can declare this by

    ndviWorkflow->assignInputData({ndviDivisorNode , 0}, nirInput);
ndviWorkflow->assignInputData({ndviDivisorNode , 1}, visInput);

  5. Set readable syntax names

    nirInput->inputName = "NIR";
visInput->inputName = "VIS";

  6. Create the metadata (registers itself to be known by the mastercatalog)

    ndviWorkflow->createMetadata();
qDebug() << ndviWorkflow->source()["syntax"]; // ndvi(NIR,VIS)

  7. Execute by passing in both raster bands, near infrared (NIR) and visual red (VIS):

    IRasterCoverage vis;
IRasterCoverage nir;
vis.prepare(makeInputPath("b2.tif"));
nir.prepare(makeInputPath("b3.tif"));

QString ndvi = QString("ndvi_out=ndvi(%1,%2)").arg(vis->name()).arg(nir.name());

ExecutionContext ctx;
SymbolTable symbolTable;
bool ok = commandhandler()->execute(executeString, &ctx, symbolTable);
if ( !ok) {
    qDebug() << "workflow execution failed.";
}

  8. Get the result from the SymbolTable

    Symbol actual = symbolTable.getSymbol("ndvi_out");

// alternatively get it from mastercatalog
IRasterCoverage raster("ilwis://internalcatalog/ndvi_out");
QString outFile = QDir::homePath() + "/ndvi_out.tiff"; // in user's home
qDebug() << "write ndvi result to " << outFile;
raster->connectTo(outFile, "GTiff","gdal",Ilwis::IlwisObject::cmOUTPUT);
raster->createTime(Ilwis::Time::now());
raster->store();

Implementation Details

workflow_class_diagram.png

  • Workflow: Provides a means to compose operations and/or workflows. Creates formal metadata description and registration of these at the mastercatalog.
  • WorkflowOperationImplementation: Implementation of the Ilwis operation interface to execute a workflow instance. It recursively follows the execution branch(es) starting from each output node until a root node and executes it node by node. Then doing the same for the next output node. Already executed branches are skipped.