Data Registry - ngageoint/opensphere-desktop GitHub Wiki

The data registry is a centralized location in which the application stores data. It is implemented as a two level cache, consisting of in-memory storage, and an H2 database. The registry is accessible via the System Toolbox, and when retrieving data, it first asks for data in memory, and in case of cache miss, then makes use of the H2DB, and if no data is available in the H2DB, then uses an Envoy to query external data sources. The External Source Request is optional, and can be disabled. When attempting to retrieve remote data, the registry searches for an Envoy that can query data of the requested type, and if no envoys are found, no results are returned. Results are aggregated if more than one envoy is found.

The data registry stores data models, which are produced by io.opensphere.core.api.Envoy instances and used by io.opensphere.core.api.Transformer instances, as well as other plug-in components.

Storing Data

To add models to the registry, call DataRegistry.addModels(CacheDeposit). Each deposit has an expiration date. After the expiration, the models will no longer appear in query results, and the registry is free to delete the data. To make models only last for the current session, specify an expiration date of CacheDeposit.SESSION_END. If the models should never expire, simply construct a Date with a large number, like Long.MAX_VALUE. Each deposit also has a critical flag. If a deposit is critical, it will not be eligible for deletion when the database is cleaned.

The data registry can store objects either in-memory only, or persistently to disk. Objects that are in-memory only will obviously not appear in the registry when the application is restarted, regardless of expiration date. Objects stored persistently may also be cached in-memory for quick retrieval.

Clients of the data registry decide how their data should be represented in the registry. For low-volume simple data, it may be easiest to simply store the data models wholesale. If all the models within a category will always be retrieved, no additional properties are needed.

To do this simplest case, follow these steps:

  1. Construct a DataModelCategory for the models.
  2. Build a PropertyDescriptor. The property name in the property descriptor can be any string. The property type should be the class of the data models.
  3. Create one of the following:
    • SimplePersistentCacheDeposit.SimplePersistentCacheDeposit(DataModelCategory, PropertyDescriptor, Collection, Date) if your models are serializable and you want them to be stored persistently.
    • SimpleSessionOnlyCacheDeposit.SimpleSessionOnlyCacheDeposit(DataModelCategory, PropertyDescriptor, Collection) if your models are not serializable or you do not want them to be stored persistently.
  4. Call DataRegistry.addModels(CacheDeposit) with your CacheDeposit.

The call to DataRegistry.addModels(CacheDeposit) will return an array of unique numeric IDs generated by the registry, one for each input object. These IDs may be used later to update or retrieve the objects.

Models added in this manner will only be query-able by their ID or DataModelCategory and serialized representation. It may be necessary for data models to be query-able by some other property, such as a string key. To associate additional properties with the models, follow these steps:

  1. Build a PropertyDescriptor for each property. Each property should have a different property name. Property types may be duplicated, except in the case of Geometry instances.
  2. Create a PropertyAccessor for each property that will extract the property values from the model objects. In most cases, SerializableAccessor can be used. For more complicated cases, see also TimeSpanAccessor, GeometryAccessor, and PropertyArrayAccessor.
  3. If you still want to store your data model serialized, you'll also need to create an accessor for the data model itself. See SerializableAccessor.getHomogeneousAccessor() for a way to easily create it.
  4. Since more than one accessor will be needed, use a DefaultCacheDeposit rather than the simpler versions suggested above.

Reading Data

There are several ways to read data from the registry, both synchronously, and asynchronously, and also a way to access data without invoking the Envoys to retrieve data in case of a cache miss.

To retrieve values from the registry, the simplest case is to get all values that match a DataModelCategory and a PropertyDescriptor. These objects can be the same as what was provided in the DataRegistry.addModels(CacheDeposit) call. Components of the DataModelCategory may also be null to indicate a wild-card. For example, it may not matter what the source of the objects is, so the source in the DataModelCategory could be null. To do a query like this, do the following:

  1. Use the constructor SimpleQuery.SimpleQuery(DataModelCategory, PropertyDescriptor) with your property descriptor to create a Query object.
  2. Depending on the type of access desired, use one of the following methods:
    • For asynchronous loading access:
      • Call DataRegistry.submitQuery(Query) with the DataModelCategory and the Query object.
      • Call SimpleQuery.getResults() to get the query results.
    • For asynchronous non-loading access (no data load will be attempted in case of a cache miss)::
      • Call DataRegistry.submitLocalQuery(Query) with the DataModelCategory and the Query object.
      • Call SimpleQuery.getResults() to get the query results.
    • For synchronous loading access:
      • Call DataRegistry.performQuery(Query) with the DataModelCategory and the Query object.
      • Call SimpleQuery.getResults() to get the query results.
    • For synchronous non-loading access (no data load will be attempted in case of a cache miss):
      • Call DataRegistry.performLocalQuery(Query) with the DataModelCategory and the Query object.
      • Call SimpleQuery.getResults() to get the query results.
  3. To get a value of one property that goes with a specified value of another property, use KeyValueQuery.

To do more complex queries, see the subclasses of io.opensphere.core.cache.matcher.PropertyMatcher<T>. One or more matchers may be passed to SimpleQuery.SimpleQuery(DataModelCategory, PropertyDescriptor, java.util.List).

Also see io.opensphere.core.data.util.DefaultQuery to do ordered or row-limited queries.

Subscribing to Registry Changes

To receive asynchronous notification of registry changes, create a new io.opensphere.core.data.DataRegistryListener<T>, and use it to create a subscription within the EventManager (accessed through the system Toolbox). When creating the subscription, do so using an appropriate io.opensphere.core.cache.util.PropertyDescriptor<T> that describes the data that you are interested in. To make things easier, an empty implementation of the DataRegistryListener interface is provided as io.opensphere.core.data.DataRegistryListenerAdapter<T>, allowing you to implement only necessary methods. In the DataRegistryListener implementation, process the data using the valuesAdded, valuesRemoved, and valuesUpdated methods.

Example of Creating the Subscription:

public static final PropertyDescriptor<WPSCapabilitiesType> WPS_GET_CAPABILITIES = new PropertyDescriptor<>(
        "wpsGetCapabilties", WPSCapabilitiesType.class);

private final WpsGetCapabilitiesDataRegistryChangeListener myDataRegistryListener;            
            
public WpsMantleController(Toolbox pToolbox)
{            
    myDataRegistryListener = new WpsGetCapabilitiesDataRegistryChangeListener(
        this::processCapabilitiesDocumentAdded, this::processCapabilitiesDocumentRemoved);

    DataModelCategory category = new DataModelCategory(null, WpsDataTypeInfo.SOURCE_PREFIX,
            WpsRequestType.GET_CAPABLITIES.getValue());
    myToolbox.getDataRegistry().addChangeListener(myDataRegistryListener, category,
            WPS_GET_CAPABILITIES);
}

Tips and Tricks

When registering with the Data Registry for change notification, ensure that your listener does not get garbage collected, as the registry maintains the listener list as a set of weak references, which go away if the parent object gets de-scoped.

In other words, don't do this:

public WpsMantleController(Toolbox pToolbox)
{
    ...
    
    WpsGetCapabilitiesDataRegistryChangeListener dataRegistryListener =
        new WpsGetCapabilitiesDataRegistryChangeListener(this::processTypeAdded);
        
    ...
    
    pToolbox.getDataRegistry().addChangeListener(dataRegistryListener, category,
        WpsPropertyDescriptors.WPS_GET_CAPABILITIES);
}

Do this instead:

private WpsGetCapabilitiesDataRegistryChangeListener myDataRegistryListener;

public WpsMantleController(Toolbox pToolbox)
{
    ...
    
    myDataRegistryListener = new WpsGetCapabilitiesDataRegistryChangeListener(this::processTypeAdded);
    
    ...
    
    pToolbox.getDataRegistry().addChangeListener(myDataRegistryListener, category,
        WpsPropertyDescriptors.WPS_GET_CAPABILITIES);
}

Note the difference in the example. In the first example, the dataRegistryListener field will be garbage collected (contrary to typical Java behavior), because the data registry maintains weak references. In order for the instance to remain, a local handle must be preserved, forcing the second example to use a member field to preserve the instance.