Ignite Cache and Performance Test - learn-tibco-cep/tutorials GitHub Wiki

Most applications in production will require data grid cache and/or persistent data store. One of the great benefits of the BE platform is that it manages the data cache and persistence transparently. The platform supports various kind of data grid and database systems for cache and data persistence. The Apache Ignite is one of the supported options.

This page describes how the Async Service can be configured to run with an Ignite distributed cache. All configurations for the Ignite cache is specified in a CDD file, DemoCache.cdd.

CDD Configuration for Ignite Cache

Following are configurations in the CDD to support performance testing with Ignite distributed cache.

  • In Cluster Configuration, choose Clustered management mode, and use Apache Ignite as the Cluster Provider.
  • In Agent Class Configuration for the inference class, set Thread Count of shared queue appropriately based on the performance requirement. This value specifies the number of threads used to process inbound messages. The default value is 10. For the performance test of this tutorial, it is set to 64.
  • In Agent Class Configuration for the inference class, set the post RTC threads appropriately based on the performance of the network and cache nodes. The value is set by the property Agent.inference-class.threadcount, where inference-class is the name of the agent class. The default value is 2. For the performance test of this tutorial, it is set to 10.
  • In Processing Units Configuration, set global variables to use cluster locks, e.g., Agent/LockLocal = false, and to use shared FTL endpoints for inbound messages.

Required Code Update

The source code of this tutorial already supports both unclustered in-memory mode and clustered Ignite cache mode. The only API difference is for the data loading operation. When running in memory-only mode, it calls Instance.getByExtIdByUri(). When running in cache mode, this API is switched to Cluster.DataGrid.CacheLoadConceptByExtIdByUri(). Both are supported by this tutorial as shown in the function onHandlerExit.

Build and Execute Performance Test

Clone source project, and import it into BE studio workspace, $WS, as a Existing TIBCO BE Studio Project. Build the enterprise archive, e.g., $WS/Demo.ear.

Start Ignite cache engines

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n cache-1 -u cache -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

Optionally, start more cache engines for load distribution:

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n cache-2 -u cache -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

Start inference engines

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n default-1 -u default -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

Optionally, start more inference engines for load distribution:

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n default-2 -u default -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

Start mock service

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n mock -u mock -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

Start test driver

$BE_HOME/bin/be-engine --propFile $BE_HOME/bin/be-engine.tra -n test -u test -c ${WS}/AsyncService/DemoCache.cdd Demo.ear

As described in README, you may adjust the request rate of the test driver by setting the variables of Test/RequestInterval and Test/Publishers.

Test Result

We run one Ignite cache engine and one inference engine on 2 separate servers. We set the test driver to use 10 publishers and each publisher publish requests with interval of 20 ms, and thus the driver ingests requests at a rate of approximately 500/s. The mock service is configured to return 1-3 responses for each request, and thus it generates service responses at a rate of approximately 1000/s. Counting 1 client response per request, we generated a total FTL message rate of 2000/s during this test.

The test driver printed out the following performance stats, which shows that the system handled the actual client requests at rate of 466.88/s. The actual elapsed time of the process is 69.413 ms, which includes the message delay of 50 ms deliberately added by the mock service. 0.74% of the requests timed out due to race conditions when service response is received before the message handler is written in the cache. This condition is left unhandled simply to test the implementation of timeout rules.

[Complete-ServerElapsed] reset: 31317, elapsed: 68041 ms, rate: 466.880/s, count: 450, avg: 61.356, max: 181.000, min: 3.000
[Complete-ClientElapsed] reset: 31317, elapsed: 68041 ms, rate: 466.880/s, count: 450, avg: 69.413, max: 195.000, min: 7.000
[Timeout-ServerElapsed] reset: 0, elapsed: 68041 ms, rate: 0.735/s, count: 50, avg: 10098.900, max: 10389.000, min: 10005.000
[Timeout-ClientElapsed] reset: 0, elapsed: 68041 ms, rate: 0.735/s, count: 50, avg: 10114.380, max: 10449.000, min: 10007.000

More detailed performance stats are printed out by the inference engine as follows, which shows the fact that FTL message delays or Ignite cache read time are all within 4 ms. The mock service adds a delay of 32.38 ms per service response, which sums to a total delay of 64.76 ms for 2 responses per client request. This calculation demonstrates how the Stats utility can be used to print out detailed performance data and help identify system bottlenecks.

[ClientRequestDelay] reset: 30641, elapsed: 67493 ms, rate: 468.123/s, count: 954, avg: 3.655, max: 20.000, min: 3.000
[SendServiceRequest] reset: 30643, elapsed: 67494 ms, rate: 468.145/s, count: 954, avg: 0.089, max: 1.000, min: 0.000
[ServiceElapsed] reset: 61088, elapsed: 67494 ms, rate: 932.320/s, count: 1838, avg: 32.381, max: 139.000, min: 0.000
[ServiceEnd2End] reset: 61089, elapsed: 67494 ms, rate: 932.335/s, count: 1838, avg: 40.650, max: 175.000, min: 2.000
[AcquireHandlerLock] reset: 61231, elapsed: 67494 ms, rate: 934.379/s, count: 1834, avg: 1.283, max: 56.000, min: 0.000
[RetrieveLockedHandler] reset: 61176, elapsed: 67494 ms, rate: 933.579/s, count: 1835, avg: 2.295, max: 58.000, min: 0.000
[RetrieveExitHandler] reset: 0, elapsed: 67494 ms, rate: 0.741/s, count: 50, avg: 2.400, max: 4.000, min: 1.000
[ServiceResponseDelay] reset: 61189, elapsed: 67494 ms, rate: 933.727/s, count: 1832, avg: 4.002, max: 56.000, min: 3.000
[Complete-HandleElapsed] reset: 30556, elapsed: 67494 ms, rate: 466.797/s, count: 950, avg: 56.848, max: 177.000, min: 3.000
[Complete-ExpectedResponses] reset: 30557, elapsed: 67494 ms, rate: 466.812/s, count: 950, avg: 1.934, max: 3.000, min: 1.000
[Timeout-HandleElapsed] reset: 0, elapsed: 67494 ms, rate: 0.741/s, count: 50, avg: 10098.900, max: 10389.000, min: 10005.000
[Timeout-ExpectedResponses] reset: 0, elapsed: 67494 ms, rate: 0.741/s, count: 50, avg: 1.960, max: 3.000, min: 1.000

Tips

This exercise of performance testing indicates that both FTL messaging and Apache Ignite cache perform very well when used by a BE application. However, the Ignite cache is not persisted in this test. If data persistence is turned on, the performance would depend largely on the speed of disk I/O of the storage device.

Besides, the following practical tips would help the performance tuning when Ignite is used as BE cache.

Configure Ignite not to read from backup node

By default, Ignite uses PRIMARY_SYNC as the cache synchronization mode, which means that reading from a backup node may return stale data. To avoid such problem in a BE application, you should turn off the Ignite flag Read from Backup in the CDD file.

Increase thread pool for cache writes

Cache writes in a BE application happen in post RTC. If cache writes become a bottleneck of the system, you may verify the problem by checking the WorkPool threads in a jstack thread dump. If most WorkPool threads are waiting on cache writes, you can improve the system performance by setting the property Agent.AgentClass.threadcount to a larger value, where AgentClass is the name of the inference agent class in the CDD.