TickTockDB v.s. InfluxDB, max cardinality comparison in RaspberryPI4 (ARM 64 bit) - ytyou/ticktock GitHub Wiki

Table of Contents

1. Introduction

2. IoTDB-benchmark Introduction

3. Experiment Settings

4. Resource consumption comparison

4.1 CPU

4.2 IO Util

4.3 Write and read bytes rate

4.4 Memory

5. Response time comparison

5.1 Average write response time

5.2 Average read response time

5.3 P999 write response time

5.4 P999 read response time

6. Conclusion

1. Introduction

In our previous wikis, we have evaluated how much cardinality that TickTockDB and InfluxDB can handle in RPI-zero-Wireless (ARMv6, 32bit OS) and OrangePI zero2. In this experiment, we want to see how they perform in RaspberryPI4.

In our tests, we let clients send data points to a TSDB periodically (one data point every 10 seconds per time series). We measure how many time series a TSDB can handle in max. We think it is close to normal scenarios in which there should be certain intervals between two consecutive operations from a client. For example, CPU data are collected once every 10 seconds.

2. IoTDB-benchmark Introduction

We select IoTDB-benchmark for performance evaluation. Please refer to README and the introduction in the previous wiki for details.

3. Experiment settings

3.1. Hardware

The figure shows a RaspberryPI4, a Single Board Computer (SBC) with

  • Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz,
  • 4GB LPDDR4-3200 SDRAM,
  • 802.11 b/g/n wireless LAN,
  • 128GB v30 extreme SD card (SanDisk), (we used exactly the same SD card in OrangePI zero2 test.)
  • OS: Debian GNU/Linux 11 (bullseye),
  • Cost: $55 in Canakit.com (board only, not including the SD card)

We run IoTDB-benchmark on an Ubuntu laptop with 12 cores AMD Ryzen5 5600H cpu, 20GB of memory. We try to minimize network bottleneck by connecting the laptop with the RaspberryPI4 by a network cable directly. We assign static IPs to RaspberryPI4 and the laptop by running, e.g., in RaspberryPI4

sudo ip ad add 10.0.0.4/24 dev eth0

3.2. Software

  • TickTockDB
  • Version: 0.11.6
  • Config: tt.conf
  • Most configs are default except the followings. You can call config.sh to find out.
ylin30@raspberrypi:~/ticktock $ ./admin/config.sh
{
  "tsdb.timestamp.resolution": "millisecond"
}

For comparison purpose, we pick InfluxDB since it is the most popular TSDB.

  • Influxdb
  • IoTDB-benchmark
  • Version: main
  • Sample config:
  • Important settings in the config:
  • Read-Write ratio: reads(10%) and writes(90%).
  • Loop: 2160 and 10-seconds interval (which keeps each test running for 6 hours(=2160*10s))
  • Number of sensors per device: 10
  • We scale up loads by increasing the number of devices from 1k to 250k.
  • We bind each client to 100 devices (except 50 devices/client for the case of 250k devices since reads are too slow already). So we will update CLIENT_NUMBER and DEVICE_NUMBER in config.properties for each test.

The above configs will simulate a list of clients collecting a list of metrics (DEVICE_NUMBER * 10 sensors per device) every 10 seconds, and sending the metrics to TickTockDB/InfluxDB. Note that we use InfluxDB line write protocol in both TickTockDB and InfluxDB since the protocol is more concise than both OpenTSDB plain put protocol and InfluxDB v1 batch writes. Essentially, the line write protocol can send multiple data points in just one line, e.g., you can send cpu.usr, cpu.sys, and cpu.idle of cpu 1 in host rpi in one line.

cpu,host=rpi,id=1 usr=10,sys=20,idle=70 1465839830000

4. max cardinality: Resource consumption comparison

In our tests, clients send a device's sensor data in one operation at every 10 seconds. Each sensor's data is a unique time series, so the cardinality is number of devices * 10 sensors/device. Note that it is not a backfill case in which data points are sent back to back right after a previous write is finished. The write throughput is fixed in our test setup. So it doesn't make sense to compare throughputs in both TickTockDB and InfluxDB. Instead, we compare how much OS resources TickTockDB and InfluxDB consumed at this load. The lower OS resources a TSDB consumes, the better the TSDB is.

4.1. CPU

The above figures show cpu.idle metric during tests. The higher, the better.

At the load of 10k devices, the cardinality is 100k (=10k devices * 10 sensors/device). Cpu.idle was 50-60% for InfluxDB, while it was about 97% for TickTockDB. The situation is similar at the load of 20k devices (i.e., cardinality of 200k). Obviously, TickTockDB consumes much less CPU than InfluxDB.

The left part (in blue) is 3 tests on InfluxDB. Higher the load is, more CPU was consumed. We stopped the test of InfluxDB at 20k devices since it has already taken 22820.24 seconds (longer than the planned 6 hours/21600 seconds) to finish. There were 1917 failed operations out of 38866383 due to timeout (10s by default). Cpu.idle was 30%. We considered its max cardinality as 150k (i.e., 15k devices * 10 sensors/device).

The right part (in red) is 6 tests on TickTockDB. CPU consumption is proportional to loads. We stopped the test of TickTockDB at 250k devices (i.e., 2.5M cardinality). It took 22887.38 seconds to finish. There were no failed operations but we did see some writes took longer than 10s. Cpu.idle was about 17%.

Note that we used sync writes in IotDB-benchmark clients. Async writes would be faster than sync according to our evaluation. We did try async instead of sync writes in IotDB-benchmark clients, the 2.5M cardinality load was finished in time. Nevertheless, let's consider TickTockDB's max cardinality as 2M.

4.2. IO Util

At the load of 10k devices (i.e., 100k cardinality), InfluxDB's IO util was up to 80% while TickTockDB's IO util was almost negligible at this load. It is similar at the load of 20k devices.

The max IO util consumed by InfluxDB was 90% at the load of 15k devices. Interestingly when the load was increased to 20k devices, IO util was decreased. We think it was because InfluxDB couldn't finish more and more requests within 10 seconds. Hence, clients actually sent less requests at the load of 20k devices.

IO util consumed by TickTockDB grew when loads were increased. But it was less than 20% even at the load of 250k devices (i.e., 2.5M cardinality).

4.3 Write and read bytes rate

Write bytes rate pattern is similar to IO util pattern, not surprisingly.

At the load of 10k devices, InfluxDB's write bytes rate was up to 4.5MB/sec, and TickTockDB's write byte rate about 50kB/sec, respectively.

The max write bytes rate was about 5MB/sec for InfluxDB at the load of 15k devices. It was less than 2MB/sec most of time for TickTockDB at the load of 250k devices.

Note that we used a v30 SanDisk SD card. We tested write and read bytes rate using dd, they are 21.2MB/s and 44.1MB/s, respectively. So write bytes rate was still far from saturation though IO util for InfluxDB was close to saturation.

ylin30@rpi4:~ $ dd if=/dev/zero of=./test bs=512k count=4096
4096+0 records in
4096+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 101.088 s, 21.2 MB/s
ylin30@rpi4:~ $
ylin30@rpi4:~ $ ls -hl ./test
-rw-r--r-- 1 ylin30 ylin30 2.0G Apr  8 00:47 ./test
ylin30@rpi4:~ $ sudo sysctl -w vm.drop_caches=3
vm.drop_caches = 3
ylin30@rpi4:~ $ dd if=./test of=/dev/zero bs=512k count=4096
4096+0 records in
4096+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 48.6926 s, 44.1 MB/s
ylin30@rpi4:~ $

Both TickTockDB and InfluxDB read bytes rate were very small. We think it is because most of data were still in cache.

4.4 Memory

At the load of 10k devices, RSS memory of InfluxDB grew up to 415MB. TickTockDB's RSS memory was up to 105MB. TickTockDB consumes less RSS memory than InfluxDB.

With higher loads, RSS memory consumption was higher in both InfluxDB and TickTockDB. When InfluxDB reached its max cardinality (i.e., 150k=15k devices * 10 sensors/device), RSS memory was less than 884MB. Note that RaspberryPI4 has 4GB memory in total. So memory is not the bottleneck for InfluxDB. There are plenty of memory left for InfluxDB.

TickTockDB used lots of memory. When TickTockDB reached its max cardinality (i.e., 2M=200k devices * 10 sensors/device), RSS was 2.27GB. With 2.5M cardinality, RSS of TickTockDB was 2.85GB. It is close to 4GB. Generally OS will reserve some memory and applications should not use more than half to 3/4 of total memory. We consider TickTockDB takes good advantage of memory while InfluxDB doesn't.

5. Response time comparison

How fast an TSDB write and read data points is important. Let's look at response time in both InfluxDB and TickTockDB. Note that it is response time per operation (not per data point)[^1][^2].

[^1]:In IotDB-benchmark, a write operation is for a device and there are 10 data points (i.e., 10 sensors' data). In normal DevOps scenarios with collector framework like TCollector or Telegraf, an operation may contain more than 10 data points as collector frameworks would combine data points collected by different collectors (like cpu collector and memory collector) in one request. Generally, more data points in a request, less write response time per data point in average.

[^2]:Also note that the config of 10 sensors/device determines how efficient write line protocol performs. In DevOps scenarios, we can't put cpu metrics and memory metrics in one line according to write line protocol since CPU ad memory metrics are quite different and with different tags. But collector frameworks can still send two lines in one request. Unfortunately IotDB-benchmark doesn't support this.

5.1. Average write response time

Higher the load is, longer the write response time is for both InfluxDB and TickTockDB.

At 100k cardinality, average write response time per operation for InfluxDB was 29.48ms while 2.65ms for TickTockDB. TickTockDB is 10x faster than InfluxDB.

At 200k cardinality, average write response time per operation for InfluxDB was 64.70ms. At 10x higher cardinality (i.e., 2M), it was only 23.45ms for TickTockDB.

5.2. Average read response time

We used 5 kinds of reads in IotDB-benchmark. Their response time patterns are similar so we just present you AGG_RANGE here.

Aggregation query with time filter: select func(v1)... from data where device in ? and time > ? and time < ?.

At 100k cardinality, average read response time for InfluxDB was 11.67ms while 23.25ms for TickTockDB. InfluxDB is faster than TickTockDB at this load in average.

However, at 200k cardinality, InfluxDB was slower than TickTockDB in read, 60.27ms v.s., 30.05ms, respectively.

Read response time kept going up while loads were increased in both InfluxDB and TickTockDB. At 2.5M cardinality, TickTockDB spent up to 350ms in read in average.

5.3. P999 write response time

Now let's look at P999 write response time. The p999 write response time is much larger than the average in both InfluxDB and TickTockDB.

At 100k cardinality, InfluxDB's p999 write operation took 2063.04ms. Recall that its average write only took 29.48ms above. It was a 66x increase. TickTockDB's p999 write took 42.99ms, compared with its average write response time 2.65ms. It was a 16x increase.

When InfluxDB reached its max cardinality 150k, its p999 write took 2548.94ms. When TickTockDB reaches max cardinality 2M, its p999 write took 732.54ms, less than 1 second.

5.4. P999 read response time

At 100k cardinality, InfluxDB's p999 read operation took 1283.96ms. Recall that its average read only took 11.67ms above. It was a 109x increase. TickTockDB's p999 read took 112.74ms, compared with its average write response time 23.25ms. It was a 5x increase.

When InfluxDB reached its max cardinality 150k, its p999 read took 2531.69ms. When TickTockDB reaches max cardinality 2M, its p999 read took 16926.35ms which is too long to be acceptable.

6. Conclusion

  • We compared TickTockDB with InfluxDB in RaspberryPI4 (ARMv8, 64 bit OS) in terms of max cardinality. Instead of using backfill scenarios, we simulated normal scenarios that a list of clients send a list of time series (100 devices per client and 10 sensors per device) at every 10 seconds interval.
  • InfluxDB's max cardinality is 150K (i.e., 15k devices and 10 sensors/device). TickTockDB's max cardinality is 2M (i.e., 200k devices and 10 sensors/device).
  • At the same cardinality load (10k devices), TickTockDB consumes much less OS resources than InfluxDB in CPU, IO, and memory.
  • In RaspberryPI4, CPU and IO util are the bottlenecks for InfluxDB. IO util might be even worse.
  • In RaspberryPI4, TickTockDB's consumption of OS resources are more balanced than InfluxDB. At its max cardinality 2M, there were still 40-50% cpu.idle and 1.75-2GB memory available. We considered 2M as its max cardinality just because clients couldn't finish all requests within the planned time range. If we used async instead of sync writes in IotDB-benchmark clients, 2.5M cardinality could have been finished in time.
  • TickTockDB is very efficient in IO. Its IO util was only about 10% at 2M cardinality while InfluxDB already 80% at 100k cardinality.
  • TickTockDB's p999 read response time is too long (about 17 seconds) at its max cardinality although it is acceptable at lower loads. We will optimize its read performance in our future releases.