6. Benchmark and Applications - OpenMPDK/SMDK GitHub Wiki
The following is an example to apply CXL memory tiering to STREAM benchmark tool using SMDK compatible API library (libcxlmalloc.so).
You need to build STREAM tool in SMDK before running the script. (run_stream_c_heap.sh)
$ cd /path/to/SMDK/lib
$ ./build_lib.sh bm
The test script below is to use STREAM benchmark tool on CXL memory with SMDK to check the memory bandwidth; COPY, SCALE, SUM, and TRIAD.
Command lines
$ cd /path/to/SMDK/lib/stream/
$ ./run_stream_c_heap.sh <-e | -n> [-a N]
(Example) $ ./run_stream_c_heap.sh -e
Options
| Options | Desc. | Default |
|---|---|---|
| -e | Gives CXL memory a priority. | -e |
| -n | Gives DRAM a priority. | |
| -a <n> | Binds STREAM processes/threads to specific CPU node. (This can only be set to a value of 0 or 1 for now.) |
none |
Example
/path/to/SMDK/lib/stream$ sudo ./run_stream_c_heap.sh -n
use_exmem:true,exmem_size:32768,normal_size:32768,maxmemory_policy:remain,priority:normal,:
*** use_adaptive_interleaving is disabled
g_arena_pool[0].nr_arena=20
g_arena_pool[0].type_mem=normal
......
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
......
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: ......
Solution Validates: avg error less than 1.000000e-13 on all three arrays
Since Intel MLC executable binary file is already in /path/to/SMDK/lib/mlc, you do not need to download or install it.
You can find sample script for MLC in /path/to/SMDK/lib/mlc. (mlc_run.sh)
Please refer to the example of executing the script below.
Command lines
$ cd /path/to/SMDK/lib/mlc/
$ ./mlc_run.sh # run with root privileges
Example
/path/to/SMDK/lib/mlc# ./mlc_run.sh
Intel(R) Memory Latency Checker - v3.10
Command line parameters: --latency_matrix
Using buffer size of 2000.000MiB
Measuring idle latencies (in ns)...
Numa node
Numa node 0 1 2 3
0 ......
Note: Since SMDK supports bandwidth aggregation of CXL memory, the performance value measured in a node consisting of two or more CXL devices is aggregated performance.
(Ref.) MLC configurations
For specific analysis through MLC, you can specify the options below in this script.
| Options for MLC | Desc. |
|---|---|
| --latency_matrix | prints a matrix of local and cross-socket memory latencies. |
| --bandwidth_matrix | prints a matrix of local and cross-socket memory bandwidths. |
| --peak_injection_bandwidth | prints peak memory bandwidths for various read-write ratios with all local accesses. |
| --idle_latency | prints the idle memory latency of the platform. |
| --loaded_latency | prints the loaded memory latency of the platform. |
| --c2c_latency | prints the cache to cache data transfer latency of the platform. |
| [-e] [-r] [-X] [-v] |
-e: h/w prefetcher state will not be modified -r: random access to get latencies(needed when prefetchers can't be disabled by MLC) -X: only one thread from each core used -v: verbose printout |
For more information about MLC configuration, please see --help option.
Note: The explanations described below presuppose that all the applications required are installed on your system. We provide script to build some applications such as Redis, Memcached, Memtier, etc. See the script build_lib.sh at /path/to/SMDK/lib.
The following is an example to apply CXL memory tiering to Memcached application using SMDK compatible API library (libcxlmalloc.so). We are providing source code of Memcached and build script to install it in your system.
You can build Memcached in SMDK through the commands below.
$ cd /path/to/SMDK/lib
$ ./build_lib.sh memcached
After the build is finished, you can find Memcached executable file at /path/to/SMDK/lib/memcached-1.6.9.
Please note that it is not necessary to use the Memcached source included in SMDK to run the examples below, but the scripts described below are based on the assumption that you use the Memcached in SMDK.
The following Memcached scripts can be found at /path/to/SMDK/src/app/memcached.
This script is to operate Memcached server based on CXL memory and SMDK allocator library.
Command lines
$ cd /path/to/SMDK/src/app/memcached/
$ ./memcached_run.sh <-e | -n> [-a N]
(Example) $ ./memcached_run.sh -e
Options
| Options | Desc. | Default |
|---|---|---|
| -e | Gives CXL memory a priority. | -e |
| -n | Gives DRAM a priority. | |
| -a <n> | Binds Memcached server processes/threads to specific CPU node. (This can only be set to a value of 0 or 1 for now.) |
none |
(Ref.) Memcached configurations
You can see that this script runs Memcached by $ /path/to/memcached -m 4096 -c 1400 -u root. The table below shows the execution options of Memcached.
| Options for Memcached server | Desc. |
|---|---|
| -p, --port=<num> | TCP port to listen on. (default: 11211) |
| -m, --memory-limit=<num> | item memory in megabytes. (default: 64) |
| -c, --conn-limit=<num> | max simultaneous connections. (default: 1024) |
| -t, --threads=<num> | number of threads to use. (default: 4) |
| -B, --protocol=<name> | protocol - one of ascii, binary, or auto. (default: auto-negotiate) |
| -I, --max-item-size=<num> | adjusts max item size. (default: 1m, min: 1k, max: 1024m) |
| -o, --extended | comma separated list of extended options. |
| - ext_page_size | (external storage) size in megabytes of storage pages. (default: 64) |
| - ext_wbuf_size | (external storage) size in megabytes of page write buffers. (default: 4) |
| - ext_threads | (external storage) number of IO threads to run. (default: 1) |
| - ext_item_size | (external storage) store items larger than this (bytes, default 512) |
Note: For more information about Memcached configuration, refer to --help option.
This script allows you to access the currently running Memcached server through telnet protocol, to obtain statistic information, etc.
After running the Memcached server instance as shown below, you can execute this script to perform some simple actions.
Memcached server
/path/to/SMDK/src/app/memcached$ sudo ./memcached_run.sh -e
use_exmem:true,exmem_size:16384,normal_size:16384,maxmemory_policy:remain,priority:exmem
*** use_adaptive_interleaving is disabled
g_arena_pool[0].nr_arena=20
g_arena_pool[0].type_mem=normal
......
Memcached client (memcached_monitor)
/path/to/SMDK/src/app/memcached$ sudo ./memcached_monitor.sh
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get mykey
END
set mykey 0 60 4
good
STORED
get mykey
VALUE mykey 0 4
goodEND
Memtier(memtier_benchmark) is a CLI-based benchmark tool developed by Redis Labs for load generation and benchmarking NoSQL key-value databases.
It supports the generation of workloads for both Redis and Memcached, so you can check the actual benchmark performance number using Memtier after running the Memcached or Redis server.
Like Memcached, SMDK provides source code and build script to build and use Memtier in your system.
$ /path/to/SMDK/lib
$ ./build_lib.sh memtier
The following script is at /path/to/SMDK/src/app/memtier, and it is based on the assumption that you use the Memtier included in the SMDK.
memtier_bench.sh
Options
| Options | Desc. | Default |
|---|---|---|
| --app | Protocol to use; Redis or Memcached. | N/A |
| --data_size | Object data size. | 4096 |
| --pipeline_num | Number of concurrent pipelined requests. | 1 |
| --thread_num | Number of threads. | 1 |
| --client_num | Number of clients per thread. | 1 |
| --call_num | Number of total requests per client. | 1 |
| --key_max | Key ID maximum value. | 42949673 |
| --key_pattern | Set:Get pattern. | P:P |
| --ratio | Set:Get ratio. | 1:1 |
Example: running memtier_benchmark with Memcached
Before executing the script memtier_bench.sh, Memcached server should be run in your system first. For running Memcached server, please refer to memcached_run.sh section above.
[Memtier_benchmark client]
/path/to/SMDK/src/app/memtier$ sudo ./memtier_bench.sh --app memcached --data_size 128 --pipeline_num 1 --thread_num 5 --client_num 10 --call_num 100 --key_max 5000 --key_pattern P:P --ratio 2:1
......
Start SET/GET test case
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
......
5 Threads
10 Connections per thread
100 Requests per client
ALL STATS
=========================================================================
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
-------------------------------------------------------------------------
......
The following is an example to apply CXL memory tiering to Redis server application using SMDK compatible API library (libcxlmalloc.so). We are providing vanilla source code of Redis and build script to install it in your system.
You can build Redis in SMDK through the commands below.
$ cd /path/to/SMDK/lib
$ ./build_lib.sh redis # build Redis with libc
After the build is finished, you can find Redis executable files at /path/to/SMDK/lib/redis-6.2.1.
Please note that it is not necessary to use the Redis source included in SMDK to run the examples below, but the scripts described below are based on the assumption that you use the Redis in SMDK.
The following Redis scripts can be found at /path/to/SMDK/src/app/redis.
Command lines
$ cd /path/to/SMDK/src/app/redis/
$ ./redis_run.sh <-e | -n> | [-a N]
(Example) $ ./redis_run.sh -e
Options
| Options | Desc. | Default |
|---|---|---|
| -e | Gives CXL memory a priority. | -e |
| -n | Gives DRAM a priority. | |
| -a <n> | Binds Redis server processes/threads to specific CPU node. (This can only be set to a value of 0 or 1 for now.) |
none |
(Ref.) Redis configurations
You can see that this script runs Redis server by $ /path/to/redis-server $REDISCONF --loglevel verbose.
The table below shows the execution options of redis-server.
| Options for Redis server | Desc. |
|---|---|
| --bind=<IP address> | Address Redis listens for connections. (default: all) |
| --protected-mode=<yes|no> | When enabled, Redis server only accept the clients from specified in 'bind' option. (default: yes) |
| --port=<num> | Accept connections on the specified port. (default: 6379) |
| --save=<seconds> <changes> | Redis will save the DB if both the given number of seconds and the given number of write operations against the DB occurred. (default: "") |
| --maxclients=<num> | Set the max number of connected clients at the same time. (default: 10000) |
| --maxmemory=<num> | Set a memory usage limit to the specified amount of bytes. When the memory limit is reached Redis will try to remove keys according to the eviction policy selected by '--maxmemory-policy'. (default: "") |
| --maxmemory-policy=<policy> | volatile-lru -> Evict using approximated LRU, only keys with an expire set. allkeys-lru -> Evict any key using approximated LRU. volatile-lfu -> Evict using approximated LFU, only keys with an expire set. allkeys-lfu -> Evict any key using approximated LFU. volatile-random -> Remove a random key having an expire set. allkeys-random -> Remove a random key, any key. volatile-ttl -> Remove the key with the nearest expire time. (minor TTL) noeviction -> Don't evict anything, just return an error on write operations. (default: noeviction) |
| --io-threads=<num> | Number of I/O threads that handle Redis clients socket reads and writes. |
| --io-threads-do-reads=<yes|no> | When '--io-threads' is larger than 1 and this option enabled, I/O threads to reads and protocol parsing. Otherwise I/O threads only do the write(2) syscall and transfer the client buffers to the socket. |
Note: You can run Redis server easily with a separate configuration file. (e.g., $ ./redis-server conf/redis.conf) Please check /path/to/SMDK/src/app/redis/redis.summary.conf and --help option for more information.
This script allows you to access the currently running Redis server through telnet protocol, to obtain statistic information, etc. See the example of running redis_monitor.sh script below.
[Redis-server]
/path/to/SMDK/src/app/redis$ sudo ./redis_run.sh -e
use_exmem:true,exmem_size:1024,normal_size:1024,maxmemory_policy:remain,use_auto_arena_scaling:true,priority:exmem
*** use_adaptive_interleaving is disabled
g_arena_pool[0].nr_arena=20
g_arena_pool[0].type_mem=normal
......
[Redis-monitor]
/path/to/SMDK/src/app/redis$ sudo ./redis_monitor.sh
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set mykey value
+OK
get mykey
$5
value
This is a script for executing the redis-cli tool provided by Redis basically, which is a simple program that allows to send commands to Redis, and read the replies sent by the server. You can find more information about redis-cli by $ /path/to/Redis/src/redis-cli --help.
This is a script for executing the redis-benchmark tool provided by Redis basically, which is to measure Set/Get performance of Redis server. You can find more information about redis-benchmark by $ /path/to/Redis/src/redis-benchmark --help.
Command lines
$ cd /path/to/SMDK/src/app/redis/
$ ./redis_bench.sh [options...]
(Example) $ ./redis_bench.sh --value_size 128 --conn_num 10 --call_num 100
Options
| Options | Desc. | Default |
|---|---|---|
| --value_size <byte> | Size(B) of the value to be set/get. | 262144 |
| --conn_num <n> | Number of clients. | 1 |
| --call_num <n> | Number of requests. | 10000 |
Result
Data size: 128
Number of clients: 10
Number of requests: 1000
SET:
throughput summary: ......
GET:
throughput summary: ......
Like Benchmark Memcached with Memtier section above, the test script below presupposes that you use Memtier included in SMDK, and Redis server instance is running in your system.
For running Redis server, please refer to redis_run.sh section above. For installing Memtier benchmark tool, please refer to Benchmark Memcached with Memtier section above.
run_imdb_benchmark.sh
This is another script for easier use of Memtier, both for Memcached and Redis (/path/to/SMDK/src/app/memtier). We wrote some examples in the script and commented on them, so you can uncomment them then run the script, or write your own command with the following rules: run_memtier_bench NTHREAD NCLIENT NCALL NSIZE KEYMAX KEYPATTERN RATIO
[inside run_imdb_benchmark.sh]
……
##############################################################
# Redis
##############################################################
run_memtier_bench redis 5 10 100 128 5000 "P:P" "2:1"
#run_memtier_bench redis 24 50 3495 4096 4194000 "P:P" "1:0"
......
[script running]
/path/to/SMDK/src/app/memtier$ sudo ./run_imdb_benchmark.sh
......
Start SET/GET test case
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
......
5 Threads
10 Connections per thread
100 Requests per client
ALL STATS
=========================================================================
Type Ops/sec Hits/sec Misses/sec Latency KB/sec
-------------------------------------------------------------------------
......