ExplicitSuppliers - ObjectVision/GeoDMS GitHub Wiki

The ExplicitSuppliers property is used to configure items that need to be updated before the item for which the property is configured is updated.

An example is a parameter with a list of files in a folder, needed to determine which files need to be read, that needs to be updated with a batch command.

This parameter can be configured as ExplicitSupplier for the item that reads the file list.

ExplicitSuppliers can be combined with a semicolon (;) delimiter.

Since GeoDMS 20.18 an IntegrityCheck that applies to an ExplicitSupplier — configured on the supplier itself, on one of its ancestors, or reached through the supplier's own ExplicitSuppliers — also guards the declaring item: the check is woven into its calculation, and a failing check fails the declaring item and everything computed from it. Before, such a check was only evaluated beside the declaring item and its verdict did not travel to consumers. A configuration whose supplier check silently failed can therefore start failing on the first run after upgrading — which is what the check was configured for. See IntegrityCheck.

Since GeoDMS 20.20.0 the ExplicitSuppliers of an item that is read from a gdal.vect or str storage are part of the identity of that read: the read waits for them like a calculation waits for its arguments, and a change in a supplier causes a new read. See StorageManager.

The interest an item holds on its ExplicitSuppliers is supplier interest, and supplier interest lives until the declaring item itself reaches the Committed state. Everything an ExplicitSupplier needed in order to be produced therefore stays in memory until the declaring item is finished, not until that supplier is finished.

For a single supplier that is exactly what you want. For a driver item that lists many independent suppliers it means the peak memory of the run grows with the number of suppliers instead of with the largest one: a driver that writes N storages holds the sources of all N until the last of them is done, even though each storage was already written to disk minutes earlier.

The remedy is a driver per supplier, so that every supplier has a commit of its own to be released at:

// grows with the whole set: nothing is released until the last fileset is done
parameter<string> generate_all := 'OK'
,	ExplicitSuppliers = "= AsList('xml/pand/PerFileSet/fs_' + string(fileset/Values) + '/pand', ';')";

// grows with one fileset: each driver releases its own the moment it commits
Template VerwerkFileSet
{
	parameter<uint32> fileset_rel;
	parameter<string> fileset_expr := 'xml/pand/PerFileSet/fs_' + string(fileset_rel) + '/pand';
	parameter<string> result       := 'OK', ExplicitSuppliers = "= fileset_expr";
}

container drivers :=
	for_each_ne(
		'd_' + string(fileset/Values)
		,'VerwerkFileSet(' + string(fileset/Values) + ')'
	);

parameter<string> generate_all := 'OK'
,	ExplicitSuppliers = "= AsList('drivers/d_' + string(fileset/Values) + '/result', ';')";

No chaining between the drivers is needed. The suppliers of one item are updated one after another, each completely before the next is started, so independent drivers already give one supplier at a time.

That also means ExplicitSuppliers is not a way to ask for concurrency: listing suppliers together makes them run in sequence, not side by side. To run independent calculations side by side, chain them with do and name that chain before the storage in each driver; the Do page shows the file set example of this configuration.

Measured on the BAG 2.0 import of BAG-Tools, configuration BAG20_Xml2FSS, over ten filesets of 169 MB of XML each: peak memory 3449 MB with one driver for all filesets against 556 MB with a driver per fileset, which is what a single fileset costs, at the same run time. On the full national extract the first form needs memory proportional to the whole extract and the second does not. See ObjectVision/GeoDMS#1259.

See Update-Mechanism and For_each.