Usage Examples - ytyou/ticktock GitHub Wiki
Table of Contents
1.2 Support Opentsdb protocol: two data point formats, plain and json
2.2 Write a data point with two tags in Opentsdb plain put protocol
2.3 Write two data points with two tags in Opentsdb plain put protocol
2.9 Write and read several data points in InfluxDB line protocol
For writes, TickTock supports three protocols, TCP, UDP, and HTTP, at the same time. We will provide examples in the rest of this wiki.
- TCP:
A reliable streaming protocol providing once and only once semantics. You don't need to worry about if your data will be lost or received more than once in server side. However, keep in mind that TCP write requests will be returned immediately without waiting for server responses. The write requests might fail to be processed by TickTock even though they were received by TickTock server. So we call TCP write async write. You can update port in config,
tcp.server.port = 6181,6180
The first one
6181is for Opentsdb plain put protocol and the second one6180is for InfluxDB line protocol. If you use Opentsdb plain put protocol to send writes, use 6181 (i.e.,<ticktock>:6181/api/put). If you use InfluxDB line protool for writes, use 6180 (i.e.,<ticktockDB>:6180/api/write).
- UDP:
An unreliable connectionless protocol. Compared with TCP, UDP is faster but less reliable without once and only once semantics, which means that requests might be received multiple times or not at all by TickTock server. As TCP, UDP requests will be returned immediately without waiting for response.
UDP is disabled by default. To enable UDP, you need to set explicitly in config,udp.server.enabled = true. It uses same port as TCP in config,tcp.server.port = 6181
- HTTP
HTTP is built on top of TCP. Unlike TCP, HTTP requests will not be returned until TickTock server finishes applying the requests, successfully or failed. We call HTTP writes sync writes. It is slower than TCP as it is blocking. Default port: 6182, as controlled by a setting,
http.server.port = 6182.
For reads, TickTock only supports HTTP. Please refer to examples below in this wiki.
If you are interested in send data to TickTock yourself, the TCP Server inside TickTock accepts
data in the following format by default (default configuration option: --http.request.format=plain):
put <metric-name> <timestamp> <value> <tag1>=<val1> <tag2>=<val2> ...
One data point per line (Note: must be ended with \n), where <timestamp> is Unix timestamp
(number of seconds/milliseconds since Epoch. Default is second, as controlled by a setting,tsdb.timestamp.resolution = second), and <value> can be either integer or
floating point number. Supported floating point number formats include,
- Decimal floating point numbers, e.g.
88,+1.23,-0.45,15e16- Hexdecimal floating point numbers, e.g.
0x12,-0x1afp-2
Do not put quotes anywhere, even around metric names, tag names, or tag values. Metric name, tag names, and tag values all cannot contain spaces.
Metric names, tag names, and tag values all has to be ASCII encoded. TickTock does not support unicode, yet.
If you want to use json format in PUT requests as Opentsdb APIs, you can run TickTock with configuration --http.request.format=json. Please refer to User Guide section 2.3.
A simple write in json is like this:
curl -v -XPOST 'http://localhost:6182/api/put' -d '[{"metric":"testM1","value":2.33266,"timestamp":1514779734,"tags":{"host":"foo"}}]'
Compared with plain format:
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1514779734 2.33266 host=foo'
We strongly recommend our default format, plain, due to its simplicity and that Json consumes much more processing power.
TickTockDB supports InfluxDB line protocol since v0.11-beta. The syntax is:
<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]
For example:
cpu,host=rpi,id=1 usr=10,sys=20,idle=70 1465839830
TickTockDB provides an endpoint at <ticktockDB>:<port>/api/write for the line protocol. For the port, you can either use 6182 for HTTP or 6180 for TCP (note that the TCP for Opentsdb plain put protocol is 6181).
Data point metric name: testM1, tag: host=foo, timestamp: 1633412175, value=123
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'
Note:
- The Data point will be stored in data files located in a data directory specified as
tsdb.data.dir(default: /tmp) in conf/tt.conf.- There is a data file (1633392000.1633478400.0) and a metadata file (1633392000.1633478400.meta) in /tmp for the date of the written data point at 1633412175. Each file has a prefix as ., where:
- fromSecond: the first second of a day
- toSecond: the first second of the next day
- There may be more than 1 data file for a day if there are more data points than a data file size. They will be postfixed as ".1", ".2", etc. The data file size is determined by
tsdb.page.count(each page 4kB) in conf/tt.conf. There will be only meta file for a day no matter how may data files in a day.- You can specify the granularity you want to start a new meta file as
tsdb.rotation.frequency(default: 1d).
Data point metric name: test.cpu.usr, tag: host=foo cpu=1, timestamp: 1633412175, value=123
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put test.cpu.usr 1633412175 123 host=foo cpu=1'
Note: please use \n between two put requests, and the $'...' option ('...' does not work as it will treat \n as two chars \ and n instead of NEWLINE).
curl -v -XPOST 'http://localhost:6182/api/put' -d $'put test.cpu.usr 1633412175 123 host=foo cpu=1\nput test.cpu.sys 1633412175 123 host=foo cpu=1\n'
curl -v 'http://localhost:6182/api/query?start=1600000000&m=avg:testM1'
Note the avg, time series aggregator. It will aggregate all data points of the related time series at each timestamp to a single data point. In this example, testM1 has only 1 time series (with tag host=foo), so you won't see the difference. But let say, if we insert two data points:
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 125 host=goo'
Metric testM1 will have two data points from two time series (host=foo and host=goo) at timestamp 1633412175. The above query m=avg:testM1 will return the average value as 124 (i.e, (123+125)/2).
The aggregator options are:
- avg
- sum
- min
- max
- p50
- p90
- p99
- p999
- count
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1\{host=foo\}'
Note the end=1633412176. It can be a future epoch time. If you don't specify end (as in Example 2.4 above), it will be set to the current time by default.
Also note the backslash in \{ and \} are escape chars for curl commands in terminal windows. You don't need such backslash chars in binary commands such as python, java, or Postman command. Please look at a python query example below in https://github.com/ytyou/ticktock/wiki/Usage-Examples/#33-write-with-http:
url = "http://%s:%s/api/query?start=1633412100&m=avg:1m-avg:http.cpu.usr{host=foo}" % (HOST, PORT)
res = requests.get(url)
Also note that there is no & in between m=avg:testM1 and \{host=foo\}'. If you add a & in between, then anything after & won't be considered as part of query parameters m=aggregator:downsampler:metric{tag1=val1,tag2=val2...}. The results would be like no tags is specified.
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1600000060&m=avg:1m-avg:cpu.usr'
Note the 1m-avg, a downsample aggregator. It will aggregate all data points of the related time series in 1 minute to a single data point. It is different from the time series aggregator avg as explained in Example 2.4. Let say there are 4 data points within 1 minute (from 1600000000 to 1600000060) of cpu.usr:
- 2 data points from host=foo, one at 1600000000 with value=50 and the other at 1600000030 with value=70,
- 2 data points from host=goo, one at 1600000000 with value=30 and the other at 1600000030 with value=50,
The query avg:1m-avg:cpu.usr will return a single value 50 at timestamp 1600000000. It will downsample 1m-avg of
-
host=fooas (1600000000, 60) in which 60==(50+70)/2, and -
host=gooas (1600000000, 40) in which 40==(30+50)/2.
Then it will calculate the avg of host=foo and host=goo as (1600000000, 50) in which 50=(60+40)/2.
You can use, e.g.,
-
1h-avg, average of all data points in 1 hour -
1d-avg, average of all data points in 1 day -
1w-avg, average of all data points in 1 week -
10m-avg, average of all data points in 10 minutes -
10m-count, count of all data points in 10 minutes -
10m-max, max value of all data points in 10 minutes
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1600000060&m=avg:1m-avg:cpu.usr\{host=goo,cpu=1\}'
2.6.2 READ a metric cpu.usr with downsample aggregator, and multi tags, and result timestamp in millisecond.
If TickTock supports milliseconds in tt.conf, you can get result timestamps in millisecond by specifying 'msResolution=true' in your query:
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1600000060&m=avg:1m-avg:cpu.usr\{host=goo,cpu=1\}&msResolution=true'
To query existing metric names starting with 'test':
curl -v 'http://localhost:6189/api/suggest?type=metrics&q=test'
It will return, e.g.,
["test.cpu.sys","test.cpu.sys1","test.cpu.usr","test.cpu.usr1"]
To query by POST with detailed parameters in request body:
[Yi-MBP ticktock ]$ curl -XPOST http://localhost:6182/api/query -d '{
"start": 1356998400,
"end": 1641159300,
"queries": [
{
"aggregator": "avg",
"metric": "cpu.usr",
"tags": {
"host": "*",
"cpu": "*"
}
},
{
"aggregator": "avg",
"metric": "cpu.sys",
"tags": {
"host": "*"
}
}
]
}'
TickTockDB provides an endpoint at <ticktockDB>:<port>/api/write for the line protocol. For the port, you can either use 6182 for HTTP or 6180 for TCP (note that the TCP for Opentsdb plain put protocol is 6181). For example:
// Write two measurements (test.cpu and test.mem) in two lines.
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl -XPOST 'http://localhost:6182/api/write' -d $'test.cpu,host=rpi,id=1 usr=10,sys=20,idle=70 1465839830\ntest.mem,host=rpi,id=1 available=1024,free=512,buffer=512 1465839830'
// Query test.cpu and _field=usr
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl 'http://localhost:6182/api/query?start=1465839830&m=avg:test.cpu\{_field=usr\}'
[{"metric":"test.cpu","tags":{"_field":"usr","host":"rpi","id":"1"},"aggregateTags":[],"dps":{"1465839830":10.0}}]
// Query test.cpu and _field=sys
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ curl 'http://localhost:6182/api/query?start=1465839830&m=avg:test.cpu\{_field=sys\}'
[{"metric":"test.cpu","tags":{"_field":"sys","host":"rpi","id":"1"},"aggregateTags":[],"dps":{"1465839830":20.0}}]
Please note the keyword _field. It is reserved to identify different fields in the line protocol. You can try two scripts in <ticktock>/admin/ dir.
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ ./write.sh
ylin30@raspberrypi:~/ticktock.0.11.1/admin $ ./query3.sh
[{"metric":"test.measurement","tags":{"_field":"field2","host":"host1","sensor":"sensor1"},"aggregateTags":[],"dps":{"1677959400":2.0}}]
[{"metric":"test.measurement2","tags":{"_field":"field4","host":"host1","sensor":"sensor2"},"aggregateTags":[],"dps":{"1677959400":4.0}}]
ylin30@raspberrypi:~/ticktock.0.11.1/admin $
We provide several simple python examples below. If you want to use Python to talk with TickTock, we recommend tcollector, OpenTSDB official collector. If you use Python3, please use our forked version in https://github.com/ylin30/tcollector. The official TCollector has bugs with Python3 and doesn't work.
This example writes two data points in plain format with TCP request. The code is checked in source code dir as ticktock/api-examples/python/tcp_plain_writer.py.
#!/usr/bin/env python3
import socket
import sys
import time
if len(sys.argv) != 3:
print("Usage: python tcp_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = int(sys.argv[2])
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM);
s.connect((HOST, PORT))
print('connect to', HOST, PORT)
put_data_point1 = 'put tcp.cpu.usr 1633412275 20 host=foo cpu=1';
put_data_point2 = 'put tcp.cpu.sys 1633412275 20 host=foo cpu=1';
# Note each PUT request must be ended with '\n'
req = put_data_point1 +'\n'+put_data_point2+'\n';
s.sendall(req.encode('utf-8'));
# We need to sleep a few seconds before close the socket in this example.
# Otherwise TickTock server might not be able to read data as the socket is closed too early.
time.sleep(5)
print("Done sending two put reqeuests:\n"+req);
s.close();
except socket.error as e:
print("Exception: %s", e)
This example writes two data points in plain format with UDP request. The code is checked in source code dir as ticktock/api-examples/python/udp_plain_writer.py. Note that you need to enable UDP write in TickTock config, udp.server.enabled = true
#!/usr/bin/env python3
import socket
import sys
import time
if len(sys.argv) != 3:
print("Usage: python udp_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = int(sys.argv[2])
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM);
s.connect((HOST, PORT))
print('connect to', HOST, PORT)
put_data_point1 = 'put udp.cpu.usr 1633412275 20 host=foo cpu=1';
put_data_point2 = 'put udp.cpu.sys 1633412275 20 host=foo cpu=1';
# Note each PUT request must be ended with '\n'
req = put_data_point1 +'\n'+put_data_point2+'\n';
s.sendall(req.encode('utf-8'));
# We need to sleep a few seconds before close the socket in this example.
# Otherwise TickTock server might not be able to read data as the socket is closed too early.
time.sleep(5)
print("Done sending two put reqeuests:\n"+req);
s.close();
except socket.error as e:
print("Exception: %s", e)
This example writes two data points in plain format with HTTP request. The code is checked in source code dir as ticktock/api-examples/python/http_plain_writer.py.
import sys
import requests
PY3 = sys.version_info[0] > 2
if PY3:
from urllib.request import Request, urlopen # pylint: disable=maybe-no-member,no-name-in-module,import-error
from urllib.error import HTTPError, URLError # pylint: disable=maybe-no-member,no-name-in-module,import-error
else:
from urllib2 import Request, urlopen, HTTPError, URLError # pylint: disable=maybe-no-member,no-name-in-module,import-error
if len(sys.argv) != 3:
print("Usage: python http_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = sys.argv[2]
try:
url = "http://%s:%s/api/put" % (HOST, PORT)
req = Request(url)
req.add_header('Content-type', 'application/text')
put_data_point1 = 'put http.cpu.usr 1633412175 20 host=foo cpu=1';
put_data_point2 = 'put http.cpu.sys 1633412175 20 host=foo cpu=1';
put_req = put_data_point1 +'\n'+put_data_point2+'\n';
response = urlopen(req, put_req.encode())
print("Received response:", response.getcode())
print("To query:");
url = "http://%s:%s/api/query?start=1633412100&m=avg:1m-avg:http.cpu.usr{host=foo}" % (HOST, PORT)
res = requests.get(url)
print(res.text)
print(res.url)
except HTTPError as e1:
print("HTTP Exception:", e1)
except URLError as e2:
print("URL Exception:", e2)
The example first writes two data points (HTTP in plain format) to TickTock, and then query the data point with HTTP request. The code is checked in source code dir here.
This example kicks off several threads to stress-test TickTock. Each thread writes data points continually to TickTock with TCP protocol. Data points are in plain format. The code is checked in source code dir as ticktock/tools/client_tcp.c.
This example inspects TickTock data files and metadata files. The code is checked in source code dir as ticktock/tools/inspect.cpp. If you build with 'make all', there will be a binary in <ticktock>/bin/inspect. Examples:
./inspect /var/ticktock/data/1633392000.1633478400.0
./inspect -a /var/ticktock/data/1633392000.1633478400.0
BTW, metadata files are plain text files. You can open them with, e.g., vim 1633392000.1633478400.meta.
This example backfills data to TickTock by reading append log files. The code is checked in source code dir as ticktock/tools/backfill.cpp. If you build with 'make all', there will be a binary in <ticktock>/bin/backfill.