We use the lineitem table in TPC-H dataset, and set the length of comment text field in the dbgen lineitem to 512 (from 27). So the average row length of the lineitem table is 615 bytes, in which, the key is the combination of the first 3 fields printed into decimal string from integers.

The total size of the dataset is 554,539,419,806 bytes, with 897,617,396 lines. The total size of the keys is 22,726,405,004 bytes, the other part is values.

TPC-H dbgen generates raw data of strings, which we use directly in our test, without any transformation on the data.

	Server A
CPU Number	2
CPU Type	Xeon E5-2680 v3
CPU Freq.	2.5 GHz
CPU Actual Freq.	2.5 GHz
Cores per CPU	8 cores with 16 threads
Cores in total	16 cores 32 threads
CPU Cache	30M
CPU bogomips	4988
Memory	64GB
Memory Freq.	DDR4 2133Hz
SSD Capacity	2TB x 1
SSD IOPS	20000 (Network virtual SSD, not local SSD)

	RocksDB	TerarkDB
Level Layers	4	4
Compression	Level 0 doesn’t compress Level 1~3 use Snappy	All level uses Terark Compression
Compact Type	Level based compaction	Universal compaction
MemTable Size	1G	1G
Cache Size	16G	Do not need
MemTable Number	3	3
Write Ahead Log	Disabled	Disabled
Compact Threads	2~4	2~4
Flush Threads (MemTable Compression)	2	2
Terark Compress threads	NA	12
Working Threads	25 in total 24 Random Read, 1 Write	25 in total 24 Random Read, 1 Write
Target_file_size_base	Default (64M)	1G
Target_file_size_multiplier	1 (SST size in RocksDB do not influence performance)	5

For both of them, we disable all write speed limits:

• options.level0_slowdown_writes_trigger = 1000;
• options.level0_stop_writes_trigger = 1000;
• options.soft_pending_compaction_bytes_limit = 2ull<<40
• options.hard_pending_compaction_bytes_limit = 4ull<<40

	RocksDB (DB Cache 16GB)	TerarkDB
0~2 minutes	Read OPS 1M Write OPS 140K (Memory is enough)	Read OPS 1.69M Write OPS 120K (Memory is enough)
2~15 minutes	Read OPS 690K Write OPS 90K (Compaction started, memory pressure increases, but still enough to use)	Read OPS 850K Write OPS 80K (Memory is enough)
15~30 minutes	Read OPS down to around 10K Write OPS around 90K (Memory runs out)	Read OPS fluctuates sharply between 500K~1M, Write OPS around 60K. CPU usage close to 100%, IOWait close to 0. (Memory is sufficient, but the compression thread in the background starts to compete with the read threads.)
30~60 minutes	Read OPS declines all the time. Write OPS around 110K	Read OPS fluctuates between 350K ~ 980K, with average 500K.CPU usage starts to go down. (Compaction thread hits the read performance)
60~120 minutes	Read OPS declines all the time.	Read OPS fluctuates sharply, with average 450K. Write OPS keeps at around 55K.
170 minutes	550G data all written, with average write OPS 88K. SSD usage starts to decline.
173 minutes	Read OPS declines to lower than 10K
3 hours 30 minutes		550G data finishes writing (The compression thread in the background impacts read performance greatly.)
3.5~11 hours	Read OPS declines to lower than 5000	Read OPS keeps at around 50K (Data being compressed gradually, more data can be loaded into memory)
8 hours 40 minutes	Full Compaction finishes, data size about 213GB	Read OPS increases to around 60K
11 hours	Read OPS keeps at around 3K (Data doesn’t fully fit in memory)	Read OPS grows and keeps at around 180K
>30 hours		Data compression completes, Read OPS keeps at 1.85M (Compaction completes, data size is 47GB after compression, can be fully loaded into memory)