IntegrityCheck - ObjectVision/GeoDMS GitHub Wiki

In modelling, especially when complexity rises, errors are easily made.

The GeoDMS contains ways of assisting modellers in tracking down and solving errors, think for instance of unit metric consistency.

Another useful feature is the configuration of integrity checks for data items.

An IntegrityCheck is used to check if an (intermediate) result meets certain requirements, for instance that all values need to be within a certain range or that no missing data may occur.

If an IntegrityCheck fails, the data of the resulting items can usually still be requested in a table view, in order to find out what is wrong. The data can not be exported.

Since GeoDMS 20.14.0 an IntegrityCheck is enforced for every item below the item carrying it: requesting a nested item also validates the checks of its ancestors, and a failing ancestor check fails the requested item. In older versions such a check only fired when the checked item was itself requested; reading an item underneath it delivered the data without any validation. A configuration that relied on that silence can start failing on the first run after upgrading — which is the intended effect: the check was configured to gate exactly that data. A typical example is the version guard shown under Recent version below, which now actually protects every item of the configuration.

What counts as "below" is tree position, not item kind. A container and a domain unit are the usual carriers, but an attribute can have sub-items too — a total broken up into its components, or a categorical attribute with further categorisations underneath it — and a check on such an attribute guards those sub-items in exactly the same way.

Since GeoDMS 20.18 this also covers items read from a storage. An item with a StorageName and no calculation rule used to escape the weaving: a check on it or on its ancestors was only evaluated in the validation phase, after the data had been read. The guard now travels with every use of the imported data, so a calculation or table that consumes such an item fails when the check fails, exactly as for calculated items. Requesting the imported item itself still validates in the validation phase after reading — the data of a failed item can then still be inspected, as described above.

Since GeoDMS 20.18 a check also guards the items that declare the checked item — or an item below it — as an ExplicitSuppliers. Declaring a supplier means "evaluate this first", and the checks that apply to that supplier — its own, its ancestors', and, transitively, those of its own ExplicitSuppliers — now apply to the declaring item as well: they are woven into its calculation like ancestor checks, and a failing one fails the declaring item and everything computed from it.

parameter<uint32> run_step := ..., IntegrityCheck = "this == 0";
parameter<string> result   : StorageName = "...", ExplicitSuppliers = "run_step";

Here a failing run_step check fails result in every use, even though result's own definition never mentions run_step. In older versions such a check was only evaluated beside the declaring item, in the validation phase: a table view of the declaring item did report the failure, but the check was not scheduled as part of its calculation and did not travel to its consumers. Each guard is deduplicated: an item that also references the supplier in its expression already carries the guard through that reference and is not wrapped again.

Note what this does not change: the check gates the delivery of the declaring item's result, not the start of its calculation — the condition and the item's own expression are still evaluated as sibling suppliers.

Two rules to keep in mind when writing such a check:

• The check may not refer to items inside the subtree it guards: that closes a cycle through the guarded items and is reported as a circular dependency. Let it refer to items outside the guarded subtree, for instance a parameter configured next to it. This is what makes the break-up case awkward: total with sub-items comp1 and comp2 cannot carry IntegrityCheck = "total == total/comp1 + total/comp2", because the components would have to be calculated in order to check them and carry that same check themselves. Since 20.14.0 this is refused whichever item is requested, where before it was refused only when total itself was requested. Moving it up to the parent does not help — that parent contains the same items. Express such a cross-item assertion as a separate boolean item that carries the check on itself:

  parameter<bool> total_adds_up := total == total/comp1 + total/comp2
  ,	IntegrityCheck = "total_adds_up == True";
  

  The boolean has no sub-items of its own, and referring to the item that carries the check is the supported self-reference, so nothing reaches into a guarded subtree.

• The condition is calculated once, not once per guarded item, and each guard is enforced once per calculation, no matter how often the guarded items reference each other. When the check fails, the failure is reported on the item(s) directly under the guard; items computed from those fail because their source failed, and their failure reason names that source. So when tracking down a "Validation (Integrity Check) Failed" error, follow the named source item to the check that actually fired.

• The checks travel with the calculation rather than beside it: an ancestor's condition is folded into the calculation of the items below it, so it is scheduled and computed like any other supplier, once, and its verdict reaches every consumer of the guarded item.

If you write an expression in the IntegrityCheck, you can use 'this' to refer to itself. This is preferred.

container ReadOpbrengsten_perOP  := 
	for_each_neidv(OP/name
		, 'ReadOpbrengsten_perOP_UNCHECKED/'+OP/name
		, '(all(IsNull(this)))'
		, AdminDomain
		, EUR
	);

attribute<city> city_rel (neighborhood) := rlookup(city_code, regions/city/city_code)
, IntegrityCheck = "IsDefined(city_rel)";

The attribute uses the rlookup function to find the relation to the regions/city domain unit.
The configured IntegrityCheck checks if for each neighborhood a city is found.

In such an IntegrityCheck it is allowed to refer to the item for which the IntegrityCheck is configured, the GeoDMS has a workaround to work with such self references.

container root: IntegrityCheck = "GeoDmsVersion() >= 7.123"

This IntegrityCheck at the root container of the configuration checks if the configuration is opened in a GeoDMS version 7.123 or later.

Since 20.14.0 this guard actually protects the whole configuration: every item under the root carries it, so an older version is refused whatever item is requested. Before 20.14.0 it only fired when the root container itself was requested, which a model run never does.

An IntegrityCheck can not be configured on an item that is written to an MMD storage: writing such an item is refused, naming the storage holder to configure the check on instead. See MMD for what the storage records about the units its attributes refer to.

• an integrity check can also be used in a for_each loop.