Data Persistence - learn-tibco-cep/tutorials GitHub Wiki

In this tutorial, we demonstrate how application data can be stored in various kinds of persistent stores. The BusinessEvents platform supports many options for data persistence, and you can select or change a database at the application deployment time and so the performance requirements of the application are satisfied.

This tutorial uses the Open Library Data to demonstrate the CRUD (Create, Read, Update, and Delete) operations on cached and/or persisted data in a BE application. Thus, as your hardware resource allows, you may use this application to benchmark the application performance and scalability with tens of millions of real-world records of books, works, and authors.

Prerequisites

This tutorial will not describe the details about how to create individual BE application artifacts from scratch. Beginner BE developer may refer to the previous tutorial, Get Started, to learn about the steps for creating key BE artifacts in BE studio. Since this project uses Apache Ignite cluster as data cache, The previous tutorial on Ignite Cache performance is also helpful to understand the deployment and configuration of BE application with Ignite cache.

Installation and Configuration of Persistent Stores

In a BE application, the cache and persistent store are configured in a Cluster Deployment Descriptor (CDD) file. This tutorial includes multiple CDD files, each of which can be used to run the same application with a different kind of persistent data store.

Depending on the hardware specification of the application runtime environment, you may adjust the following properties in the CDD file for the best performance of a BE application that uses cache and/or persistent stores.

• Agent.agent-class.threadcount = 24. It specifies the number of threads used by an inference engine to write data to a distributed cache. The default value of 2 is usually too small for high data volumes.
• Agent.agent-class.dbthreadcount = 10. It specifies the number of threads used by an inference engine to write data to a persistent store with or without a distributed cache. The default value of 2 is also too small for high data volumes.
• Agent.agent-class.dbOpsQueueSize = 40. It specifies the size of queue for pending database write jobs. This number should be increased to 4-10 times of the dbthreadcount.

More tunable properties are described in the BusinessEvents Configuration Guide.

The following linked articles will describe steps to install and configure each type of cache and persistent option for deploying this tutorial.

• Apache Ignite data grid with no persistence
• Apache Ignite data grid with PostgreSQL as the backing store
• Apache Ignite data grid with ActiveSpaces 4.x as the backing store
• Apache Ignite data grid with Cassandra 4.1 as the backing store
• ActiveSpaces 4.x as the persistent store with no cache cluster

Choose a Persistence Strategy

The BusinessEvents platform supports more data persistence approaches, although this tutorial covers only the above 5 configurations. To choose an appropriate implementation for data persistence, you need to consider the following factors.

• Performance requirements of the data persistence implementation
• The ease of management of the production environment
• The development effort required to implement CRUD operations required by the application

The article Select a Persistence Store describes my personal strategy that may help you to make a correct choice for a cache/store for your application.

Implementation of CRUD Operations

The BusinessEvents platform provides build-in utilities that enables application developers to easily implement CRUD database operations, which includes Create, Query, Update, and Delete operations for single data record and bulk of data records. The following linked articles describe the implementation and practical tips used by the tutorial.

• Create or Update concepts
• Query concepts
• Delete concepts

Start and Test the Application

The README describes the steps to start BE engine nodes for this tutorial application. Third-party persistent store services are described in the linked articles in the previous section.

This tutorial implemented all performance test activities as HTTP services, and thus you can perform all tests by using an HTTP client, e.g., curl.

The script start is a sample used to start multiple required BE engine processes.

The script request packages a few requests for basic performance tests. For example, you may test the performance of batched load and updates by using the following steps.

# Load some Open Library data from OL data dump files 
./request load
# Verify resulting data counts
./request

# batch update some books and authors
./request update
# verify resulting data counts again
./request

# batch delete and revoke changes in the previous step
./request revoke
# verify resulting data counts again
./request

Practical Tips

In each of the above linked pages, a list of BE implementation tips are explained to help readers to understand the design of the corresponding application component. These tips will help BE application developers to design and implement cleaner and more efficient applications.

Source Code

The complete BE project described in this tutorial can be downloaded from Books. The Project README also summarizes the implementation of data CRUD operations and explains how to start the application nodes and run unit tests.

You may clone the Books repository, and import it into your BE studio workspace as an Existing TIBCO BE Studio Project.

 

Next Step: