Query Concepts - learn-tibco-cep/tutorials GitHub Wiki

BE applications can read/query data from cache and/or a persistent store by using a few different approaches. We describe common practices for read operations in this article.

Read Concepts by Primary Key

The most common read operation is to read a concept from cache or persistent store by its primary key. For concepts that use extId as the implicit primary key, you can call the catalog function Cluster.DataGrid.CacheLoadConceptByExtIdByUri(). It is used by this tutorial in many functions, e.g., onUpdateConcept.

In principle, you can read multiple concepts of a given set of primary keys by a single call to the plural version of the catalog function Cluster.DataGrid.CacheLoadConceptsByExtIdByUri(). Unfortunately, this function does not work in the current version of BE 6.2.2. Thus, you will have to load individual concepts in an iteration loop, which is not optimal for the best performance.

For concepts that specify explicit primary keys in the CDD file, you may read a concept by calling a similar catalog function Cluster.DataGrid.CacheLoadConceptByKeysByUri(). Unfortunately, there is no catalog function that you can use to load multiple concepts by a single invocation. Thus, you also have to load individual concepts in an iteration loop.

When a persistent store is specified by an application deployment, you can also use catalog functions to retrieve concepts by primary-keys. When ActiveSpaces is used as the store, you can load concepts by calling CEP Store functions Store.get() or Store.getAll(). Note, however, these functions do not work for concepts that use extId as the primary key implicitly.

When a RDBMS is used as the store, you can load DBConcepts by calling RDBMS functions Database.get() or Database.getAll().

BQL Query

The BusinessEvents Query Language (BQL) is still supported for backward compatibility of the BE platform. This tutorial uses BQL in a few functions to demonstrate how it can be used to read concepts from the Ignite cache.

For example, the function queryModifiedConceptIds executes a BQL by using the code similar to the following.

String sql = "select x@extId from /Concepts/Book as x where x.last_modified >= /#DateTime/parseString(\"2023-03-01 13:00:00\", \"yyyy-MM-dd HH:mm:ss\")";
Object list = Query.Util.executeInQuerySession("query-class", sql, null, false, -1);

This BQL query would return a list of primary-keys, i.e., extId of concepts that are modified after a specified date time.

Such query, however, has following performance limitations.

  • It requires a query agent of the specified agent class, i.e., query-class in this example, to execute the query;
  • Even though the query wants only primary-keys to be returned, the query agent will actually fetch and cache all fields of the resulting concepts in the processor memory of the query agent;
  • In the current BE release 6.2.2, BQL does not seem to use a pre-defined index in the query execution, e.g., the required index on last_modified for this query.

These problems would lead to bad performance, as well as risk of running out of memory in the query agent. Thus, we do not recommend the use of BQL, except for simple queries migrated from an old BE application that expects only small result-sets.

Native Query on Ignite Cache

A better approach is to use native Ignite query when the application is deployed with Ignite cache. For example, the function queryLimitedconceptIds executes a native Ignite query as follows.

String sql = "native-query: select _extId from be_gen_concepts_book where revision > 0 limit 1000";
Object list = Query.Util.executeInQuerySession("query-class", sql, null, false, -1);

Note that the query string has a prefix mark of native-query:, and the query must use the native column name, e.g., _extId and native table name, e.g., be_gen_concepts_book.

The execution of native queries also requires a query agent of a specified agent class, e.g., query-class in this example. However, it does not have the other performance issues of the BQL.

Note, however, that cache queries are executed on only data that are previously loaded in the cache, which may not be all the data in Ignite backing store. If an application must store large amount of data, and the deployment environment does not have enough memory resource, you may have to configure the CDD to use limited cache. In this case, a query on Ignite cache may not return all the data that the application may expect.

Query on Persistent Store

When a persistent store, e.g., ActiveSpaces, is configured in the CDD, you may use the Store catalog functions to execute store queries. For example, the function queryLimitedconceptIds implements a store query as follows.

String sql = "select extid from d_book where revision > 0 limit 1000";
Object iter = Store.query("http://localhost:8181", sql, null, null, null);
while (Collections.Iterator.hasNext(iter)) {
    Object row = Collections.Iterator.next(iter);
    String value = Store.Item.getString(row, "extid");
}

This query uses the table and column names in the persistent store, which are different from the definitions in Ignite cache.

Compared with the cache query in previous sections, the store query does not require a query agent, and the result-set does not suffer the limitation of partially cached data.

However, there is a known issue in the current release of BE 6.2.2 regarding the handling of column aliases in the query. For example, the following code will fail when extracting the count from the alias c.

String sql = "select count(*) as c from d_book";
Object iter = Store.query("http://localhost:8181", sql, null, null, null);
Object row = Collections.Iterator.next(iter);
long count = Store.Item.getLong(row, "c");

Query Agent

A query agent provides a couple of special features that help trigger activities based on realtime data.

Entity Aggregation

The CEP Query catalog functions include a set of aggregation functions under Datagrid.Aggregate, which can be used to generate notifications based on a specified threshold of an aggregated value.

For example, the function onAggregateQuery in this tutorial demonstrates how the filterCount or groupCount of a specified concept may be evaluated on-demand by a query agent. The query agent may send out a notification event based on the aggregation result.

Continuous Query

A query agent may start a continuous query at the startup of the agent, or by a scheduler, or upon receipt of an event. Whenever a dataset update happens, it will invoke a callback function, where a specified action will be performed.

In this tutorial, the function continuousCount implements a continuous query that specifies a callback function bqlCallback. When a query agent starts, it is configured to start 3 continuous queries as implemented in the startup function startContinuousQuery. Therefore, you can check the logs printed by the callback function whenever a new concept is created.

More details about continuous query are documented in Query Developer's Guide.