Table of Contents

1 Installation

1.1 Runtime Requirements

1.2 Running TickTock inside docker container

1.3 Install TickTock from a binary package

1.4 Compile TickTock from source

1.5 Verify TickTock running successfully

2 Configuration

2.1 Format of the configuration file

2.2 List of supported configurations

2.3 Set configuration options in TickTock docker

2.4 Performance tuning

3 Writing data to TickTock

3.1 Using TCollector

3.2 Using Telegraf

3.2.1 Using Opentsdb protocol for write

3.2.1.1 HTTP protocol

3.2.1.2 TCP protocol (suggested)

3.2.1.3 Restart Telegraf

3.2.2 Using Influxdb line protocol for write (suggested)

3.2.2.1 HTTP protocol

3.2.2.2 TCP protocol

3.2.2.3 Restart Telegraf

3.2.2.4 Query examples

3.3 Doing it yourself

4 Querying from TickTock

4.1 Querying TickTock on command line

5 Visualizing data using Grafana

The quickest way to get TickTock up and running is to run TickTock inside a Docker container. You can also install TickTock from a binary package (coming soon). Compiling from the source is another option.

To run TickTock, you will need,

• A Linux system (e.g. Ubuntu, CentOS, RaspberryPI OS Buster & Bullseye, etc). We tested both Ubuntu, CentOS, RPI. Other Linux systems are not tested yet.
• glibc 2.28 or up (Buster/Bullseye with 2.28 or higher are tested)
• zlib 1.2.11 or up

Here's the list of shared object dependencies, from ldd.

$ ldd ticktock
    linux-vdso.so.1 (0x00007ffe701a4000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6ae9b28000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6ae9946000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6ae97f7000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6ae97dc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6ae97b9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6ae95c7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6ae9bc9000)

We strongly suggest you to update the open files setting in your OS from 1024(default) to a large number, e.g., 1048576. In ubuntu, do this:

ylin30@raspberrypi:~/ticktock $ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 1384
max locked memory           (kbytes, -l) 65536
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 1384
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
ylin30@raspberrypi:~/ticktock $ sudo vim /etc/security/limits.conf
# Add these sample lines to the file. The followings are mine. Replace ylin30 with your account name.
*                soft    nofile          1048576
*                hard    nofile          1048576
ylin30           soft    nofile          1048576
ylin30           hard    nofile          1048576
# End of file
ylin30@raspberrypi:~/ticktock $ sudo vim /etc/sysctl.conf
# Add this line
fs.file-max=1048576
fs.nr_open = 1048576
ylin30@raspberrypi:~/ticktock $ sudo shutdown -r now

Note that you need to restart after modifying /etc/security/limits.conf in order for the new setting to take effect.

You need to install Docker Engine first (see instructions). You can run Docker Engine on any platform that supports it. Once the Docker Engine is installed, simply run

docker run -d --name ticktock -p 6181-6182:6181-6182 -p 6181:6181/udp ytyou/ticktock:latest

The first time you run this command, it will pull down the official TickTock image from Docker Hub before running it.

1. Download the latest (v0.20.0 on 4/21/2024) package here

2. Extract the file, e.g, ticktock-rpi-32bit.tar.gz, into a folder, ticktock/

   pi@raspberrypi:~/tmp $ gunzip ticktock-rpi-32bit.tar.gz
   pi@raspberrypi:~/tmp $ tar xvf ticktock-rpi-32bit.tar.gz
   

3. The structure of the ticktock is as:

   pi@raspberrypi:~/tmp/ticktock $ ls -l
   total 296
   drwxr-xr-x 2 pi pi   4096 Feb 17 20:15 admin         // A list of scripts for admin
   drwxr-xr-x 2 pi pi   4096 Feb 17 10:33 bin           // binaries
   drwxr-xr-x 2 pi pi   4096 Feb 17 20:26 conf          // a default config, conf/tt.conf
   drwxr-xr-x 2 pi pi  36864 Feb 17 20:28 data          // data dir in default config, conf/tt.conf
   -rw-r--r-- 1 pi pi  35149 Feb 17 20:16 LICENSE
   drwxr-xr-x 2 pi pi   4096 Feb 17 20:28 log           // log dir in default config, conf/tt.conf
   drwxr-xr-x 2 pi pi   4096 Feb 17 20:15 scripts       // A list of helpful scripts 
   drwxr-xr-x 2 pi pi   4096 Feb 17 20:17 tools         // A python file to collect metrics sent to Prometheus
   pi@raspberrypi:~/tmp/ticktock $
   

4. Run ticktock:

   pi@raspberrypi:~/tmp/ticktock $ bin/tt -c conf/tt.conf
   

Note that the default data dir is ./data and log dir is ./log. You may also override the configs in conf/tt.conf by specifying configs in the command line. E.g.,

   pi@raspberrypi:~/tmp/ticktock $ bin/tt -c conf/tt.conf --tsdb.timestamp.resolution millisecond --tsdb.data.dir /tmp

5. Gracefully stop ticktock:

We STRONGLY suggest to gracefully shutdown ticktock using <ticktock>/admin/stop.sh, although TT can still survives with kill -9 or host losing power suddenly. Without graceful shutdown, TT may lose data up to 5 minutes (default) in caches. Besides, there might be some loss of in-memory settings critical to TT.

ylin30@pi5:~/ticktock $ ./admin/stop.sh
Exiting now.
ylin30@pi5:~/ticktock $ Start shutdown process...
Shutdown process complete

[1]+  Done                    ./bin/tt -c conf/tt.conf --tsdb.timestamp.resolution millisecond --http.server.port 6182,6183

You will need

• A Linux system (Ubuntu or CentOS)
• Git
• Make
• GCC
• Zlib 1.2.11 or later

On Ubuntu (latest version tested: Ubuntu 20.04.3 LTS), install these tools with

apt update
apt install -y git zlib1g-dev
apt install -y build-essential

On CentOS (latest version tested: 8), do this

yum install -y git zlib-devel
yum groupinstall -y "Development Tools"

Now clone the source code from GitHub, and build it using either Makefile.ubuntu or Makefile.centos, depending on your platform,

git clone https://github.com/ytyou/ticktock.git
cd ticktock
make -f Makefile.<os> all

If all goes well, you can now run TickTock,

bin/tt -c conf/tt.conf [-d]

Use the option -d to run TickTock as daemon.

For ubuntu:

apt install -y curl

For centos:

yum install -y curl

curl -XPOST 'http://localhost:6182/api/admin?cmd=ping'

If TickTock is healthy, it should response with pong.

curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'

curl -v 'http://localhost:6182/api/query?start=1600000000&m=avg:testM1'

or:

curl -v -g 'http://localhost:6182/api/query?start=1600000000&m=avg:testM1{host=foo}'

Note the additional option "-g" is to encode '{' and '}' in curl.

If you are interested in TickTock's logs, you can go to Deleveoper Guides. The default log location for ytyou/ticktock is /var/lib/ticktock/log/ticktock.log.

Use the -c command line option to tell TickTock the location of your configuration file. The configuration file conf/tt.conf contains all the supported configurations, together with their default values. If you need to change a configuration to something other than its default, simply uncomment it (remove the semicolon at the beginning of the line), and update the value.

Each line in the configuration file specifies one configuration, with the format

<configuration.name> = <value>

Value can be numbers, boolean (true or false), or string. Please do NOT quote strings, or any other type of values, even when they contain spaces. Sometimes numbers are appended with a unit, e.g. 2s, which means 2 seconds.

Here's a complete list of supported configurations, with descriptions.

You can set configuration options when running a TickTock docker, e.g.,

docker run -d --name ticktock -h ticktock -p 6182:6182 -p 6181:6181 --cpus 1 -m 8GB --memory-reservation 6GB ytyou/ticktock:latest --tsdb.timestamp.resolution millisecond

The command above has one additional configuration option setup. Note that they must be after "ytyou/ticktock:latest".

• --tsdb.timestamp.resolution millisecond

• Timestamp of data points in PUT are in the millisecond format. Timestamps in GET can be either millisecond (13 digits) or seconds (10 digits).

(note: some uncommon config options are not supported as argument in command line. You have to explicitly change them in config files. You can run <tt>/admin/config.sh to see what configs have been changed from their default values, after TT successfully runs.)

Now you can write a data point in json format:

curl -v -XPOST 'http://localhost:6182/api/put' -d '[{"metric":"testM1","value":2.33266,"timestamp":1514779680000,"tags":{"host":"foo"}}]'

You can query the result with:

curl -v -s 'http://127.0.0.1:6182/api/query?start=1514779680&m=avg:1m-avg:testM5'

or

curl -v -s 'http://127.0.0.1:6182/api/query?start=1514779680000&m=avg:1m-avg:testM5'

or you can write a data point in plain format:

curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM2 1633412175 123 host=foo'

query with:

curl -v -s 'http://127.0.0.1:6182/api/query?start=1633412175&m=avg:1m-avg:testM2'

For maximum performance, the total number of requests processing threads should be the same as, or slightly above, the number of cores of the host. For example, if the host has 8 cores, then tcp.listener.count = 2 and tcp.responders.per.listener = 4.

If, for some reason, TickTock was terminated abnormally, you will lose the last few minutes of data that can only be restored if you have append log enabled. Exactly how much data will be lost depends on the tsdb.flush.frequency setting, which has a default value of 5 minutes. A smaller value here will mean less data will be lost in the event of abnormal termination, but at a cost. Our experiment shows that anything less than 1 minute will have a significant impact on performance.

Use OpenTSDB's official TCollector, or collectd's write_tsdb plugin, or InfluxDB's Telegraf, or any other agent that is compatible with OpenTSDB. Remember to change the port number from 4242 to 6181 (or whatever port number you are using).

! WARNING: (At least until v0.12.1-beta)

! Please make sure your write requests are in correct formats of Opentsdb put or Influx line protocol.
! We trust clients to send requests in correct formats. 
! If not, clients should have found the problems during testing. 
! The consequence of write requests in weird formats might result TickTockDB's meta data 
! in bad states and crashing TickTockDB.

TCollector is the default collector framework for OpenTsdb. To use TCollector to collect metrics, simply download it and run,

git clone https://github.com/ylin30/tcollector.git
cd tcollector
./tcollector start --host <ticktock-host> --port <ticktock-port> (default: 6181)

We suggest our forked version in https://github.com/ylin30/tcollector since the office tcollector (https://github.com/opentsdb/tcollector) has bugs, especially not supporting python3. We also add detailed usage guides in our tcollector repo.

Telegraf is a collector framework developed by Influxdb. It collects metrics in different data format (e.g., Influxdb input line protocol, CSV, Json, collectd, prometheus remote write, Grok, MQTT etc) through hundreds of input plugins and third party APIs, and sends metrics to different databases (e.g., Influxdb, Opentsdb, Graphite, Datadog, Kafka etc).

To install and start Telegraf, please follow the official instructions here.

TickTock can work with Telegraf as an output database as Opentsdb. You just need to add a plugin section outputs.opentsdb in telegraf.conf. (You'd better just copy the sample config provided in installation).

ylin30@yi-IdeaPad:~$ sudo cp /etc/telegraf/telegraf.conf.sample /etc/telegraf/telegraf.conf

Sample config is shown below. You can use most of the default settings in the sample telegraf.conf, except host and port.

ylin30@yi-IdeaPad:~$ sudo vim /etc/telegraf/telegraf.conf 
...
# Configuration for OpenTSDB server to send metrics to
[[outputs.opentsdb]]
    ## prefix for metrics keys
    prefix = "telegraf."

    ## DNS name of the OpenTSDB server
    ## Using "opentsdb.example.com" or "tcp://opentsdb.example.com" will use the
    ## telnet API. "http://opentsdb.example.com" will use the Http API.
    host = "http://127.0.0.1"

    ## Port of the OpenTSDB server
    port = 6182

**Note that all writes are in Json format. E.g., ***

{"metric":"telegraf.diskio_read_time","timestamp":1678407970,"value":4780,"tags":{"host":"yi-IdeaPad","name":"loop11"}}
,{"metric":"telegraf.diskio_reads","timestamp":1678407970,"value":43,"tags":{"host":"yi-IdeaPad","name":"loop11"}}
,{"metric":"telegraf.diskio_merged_writes","timestamp":1678407970,"value":0,"tags":{"host":"yi-IdeaPad","name":"loop11"}}
,{"metric":"telegraf.diskio_io_time","timestamp":1678407970,"value":4592,"tags":{"host":"yi-IdeaPad","name":"loop11"}}
,{"metric":"telegraf.diskio_writes","timestamp":1678407970,"value":0,"tags":{"host":"yi-IdeaPad","name":"loop11"}}

Json format is much heavier than the plain put format which is supported in Telegraf with TCP. See the section below.

Sample config is shown below. You can use most of the default settings in the sample telegraf.conf, except host and port.

ylin30@yi-IdeaPad:~$ sudo vim /etc/telegraf/telegraf.conf 
...
# Configuration for OpenTSDB server to send metrics to
[[outputs.opentsdb]]
    ## prefix for metrics keys
    prefix = "telegraf."

    ## DNS name of the OpenTSDB server
    ## Using "opentsdb.example.com" or "tcp://opentsdb.example.com" will use the
    ## telnet API. "http://opentsdb.example.com" will use the Http API.
    host = "tcp://127.0.0.1"

    ## Port of the OpenTSDB server
    port = 6181

**Note that all writes are in plain put format. E.g., ***

put telegraf.cpu_usage_system 1678408370 0.000000 cpu=cpu10 host=yi-IdeaPad
put telegraf.cpu_usage_softirq 1678408370 0.000000 cpu=cpu10 host=yi-IdeaPad
put telegraf.cpu_usage_irq 1678408370 0.000000 cpu=cpu10 host=yi-IdeaPad
put telegraf.cpu_usage_steal 1678408370 0.000000 cpu=cpu10 host=yi-IdeaPad
put telegraf.cpu_usage_guest 1678408370 0.000000 cpu=cpu10 host=yi-IdeaPad

Don't forget to restart your telegraf service after modifying telegraf.conf.

sudo systemctl restart telegraf

You can see metrics sent by Telegraf with the prefix "telegraf." in the example config above.

[Yi-MBP ticktock (dev2)]$ curl -s 'http://127.0.0.1:6182/api/suggest?type=metrics&q=telegraf' 

["telegraf.cpu_usage_guest","telegraf.cpu_usage_guest_nice","telegraf.cpu_usage_idle",...,"telegraf.system_uptime"]

TickTockDB supports InfluxDB line protocol and Telegraf since v0.11.3-beta. You just need to add an output plugin in telegraf.conf. (You'd better just copy the sample config provided in installation).

ylin30@yi-IdeaPad:~$ sudo cp /etc/telegraf/telegraf.conf.sample /etc/telegraf/telegraf.conf

To let Telegraf send metrics in HTTP protocol, you need to add outputs.http plugin. You can use most of the default settings except updating the url to http://<TickTockDB>:<http port, default 6182>/api/write.

ylin30@yi-IdeaPad:~$ sudo vim /etc/telegraf/telegraf.conf 
...
# A plugin that can transmit metrics over HTTP
[[outputs.http]]
    ## URL is the address to send metrics to
    url = "http://10.0.0.2:6182/api/write"

To let Telegraf send metrics in TCP protocol, you need to add outputs.socket_writer plugin. You can use most of the default settings except updating the address to tcp://<TickTockDB>:<tcp port, default 6180>.

 # # Generic socket writer capable of handling multiple socket types.
 [[outputs.socket_writer]]
 #   ## URL to connect to
     address = "tcp://10.0.0.2:6180"

Don't forget to restart your telegraf service after modifying telegraf.conf.

sudo systemctl restart telegraf

You can query the metrics sent by Telegraf.

[yi-IdeaPad ~]$ curl 'http://10.0.0.2:6182/api/query?start=1m-ago&m=avg:mem\{_field=buffered\}'
[{"metric":"mem","tags":{"_field":"buffered","host":"yi-IdeaPad"},"aggregateTags":[],"dps":{"1678404420":451432448.0,"1678404430":451436544.0,"1678404440":451436544.0,"1678404450":451440640.0,"1678404460":451444736.0}}]

Please note the keyword _field which refers to a field_key in the line protocol. The syntax of the line protocol is:

<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]

E.g.,

mem,host=yi-IdeaPad active=6915649536i,low_total=0i,slab=851771392i,vmalloc_chunk=0i,available=18955440128i,free=3993980928i,used=1474494464i,available_percent=91.03309480336719,cached=14902673408i,high_total=0i,huge_pages_total=0i,sreclaimable=638746624i,swap_cached=1097728i,dirty=40960i,huge_page_size=2097152i,vmalloc_used=72032256i,inactive=8697843712i,mapped=299307008i,shared=8257536i,vmalloc_total=35184372087808i,write_back=0i,committed_as=5844856832i,swap_total=2147479552i,write_back_tmp=0i,total=20822581248i,used_percent=7.081228049676236,buffered=451432448i,commit_limit=12558770176i,high_free=0i,huge_pages_free=0i,page_tables=19402752i,swap_free=2126630912i,low_free=0i,sunreclaim=213024768i 1678404420000000000

Please refer to Usage Examples.

TickTockDB supports InfluxDB line protocol since v0.11-beta. The syntax is:

<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]

For example:

cpu,host=rpi,id=1 usr=10,sys=20,idle=70 1465839830

TickTockDB provides an endpoint at <ticktockDB>:<port>/api/write for the line protocol. For the port, you can either use 6182 for HTTP or 6180 for TCP (note that the TCP for Opentsdb plain put protocol is 6181). For example:

// Write two measurements (test.cpu and test.mem) in two lines.
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl -XPOST 'http://localhost:6182/api/write' -d $'test.cpu,host=rpi,id=1 usr=10,sys=20,idle=70 1465839830\ntest.mem,host=rpi,id=1 available=1024,free=512,buffer=512 1465839830'

In order to query the above metrics, you can still use Opentsdb query format. However, the fields (e.g., usr, sys, idle in test.cpu) are considered a specific tag (_field) in Opentsdb query.

// Query test.cpu and _field=usr
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl 'http://localhost:6182/api/query?start=1465839830&m=avg:test.cpu\{_field=usr\}'
[{"metric":"test.cpu","tags":{"_field":"usr","host":"rpi","id":"1"},"aggregateTags":[],"dps":{"1465839830":10.0}}]
// Query test.cpu and _field=sys
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl 'http://localhost:6182/api/query?start=1465839830&m=avg:test.cpu\{_field=sys\}'
[{"metric":"test.cpu","tags":{"_field":"sys","host":"rpi","id":"1"},"aggregateTags":[],"dps":{"1465839830":20.0}}]

Please note the keyword {_field=usr} and {_field=sys}. It is reserved to identify different fields in the line protocol. You can try two scripts in <ticktock>/admin/ dir.

ylin30@raspberrypi:~/ticktock.0.11.1/admin $ ./write.sh
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ ./query3.sh
[{"metric":"test.measurement","tags":{"_field":"field2","host":"host1","sensor":"sensor1"},"aggregateTags":[],"dps":{"1677959400":2.0}}]
[{"metric":"test.measurement2","tags":{"_field":"field4","host":"host1","sensor":"sensor2"},"aggregateTags":[],"dps":{"1677959400":4.0}}]
ylin30@raspberrypi:~/ticktock.0.11.1/admin $

If you are interested in send data to TickTockDB yourself, the TCP Server inside TickTockDB accepts data in the following format by default (default configuration option: --http.request.format=plain):

put <metric-name> <timestamp> <value> <tag1>=<val1> <tag2>=<val2> ...

One data point per line (ended with \n), where <timestamp> is Unix timestamp (number of seconds/milliseconds since Epoch), and <value> can be either integer or floating point number. Supported floating point number formats include,

• Decimal floating point numbers, e.g. 88, +1.23, -0.45, 15e16
• Hexdecimal floating point numbers, e.g. 0x12, -0x1afp-2

Do not put quotes anywhere, even around metric names, tag names, or tag values. Metric name, tag names, and tag values all cannot contain spaces.

Metric names, tag names, and tag values all has to be ASCII encoded. TickTockDB does not support unicode, yet.

Example:

curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'

Note:

• The Data point will be stored in data files located in a data directory specified as tsdb.data.dir (default: /tmp) in conf/tt.conf.
• There is a data file (1633392000.1633478400.0) and a metadata file (1633392000.1633478400.meta) in /tmp for the date of the written data point at 1633412175. Each file has a prefix as ., where:

• fromSecond: the first second of a day
• toSecond: the first second of the next day

• There may be more than 1 data file for a day if there are more data points than a data file size. They will be postfixed as ".1", ".2", etc. The data file size is determined by tsdb.page.count (each page 4kB) in conf/tt.conf. There will be only meta file for a day no matter how may data files in a day.
• You can specify the granularity you want to start a new meta file as tsdb.rotation.frequency (default: 1d).

If you want to use json format in PUT requests as Opentsdb APIs, you can run TickTock with configuration --http.request.format=json. Please refer to section 2.3 above.

We try to use the same query syntax as that of OpenTSDB as much as possible, although we do not support all the syntax that OpenTSDB support. Even so, as reference, you can check out OpenTSDB's documentation.

Currently we do not support functions and expressions in query. We hope to add those supports in the future.

Example:

curl -v 'http://localhost:6182/api/query?start=1600000000&m=avg:testM1'

To use Grafana to consume TickTock data, treat TickTock as if it is OpenTSDB. When creating data source in Grafana, select the type OpenTSDB, and specify the URL to point to the HTTP Server port (default 6182). An example URL would be: http://localhost:6182. Note that we are currently compatible with OpenTSDB 2.2, as far as Grafana is concerned.

Once the data source is created, you can proceed to create your favorite dashboard using the data source.

If you are a Docker user, you can check out our demo container (refer to our front page for details).