Timestamp precision by examples - ytyou/ticktock GitHub Wiki
Table of Contents
1 Timestamp resolution feature
2 With tsdb.timestamp.resolution=second
3 With tsdb.timestamp.resolution=millisecond
TickTock supports both second (default) and millisecond precisions internally. You can specify it in config, e.g., tsdb.timestamp.resolution=millisecond.
In queries, TickTock accepts Unix (or POSIX) timestamp format. Unix times are the number of seconds that have elapsed since January 1st, 1970 at 00:00:00 UTC time. For example, 1727813339 represents Tuesday, October 1, 2024 20:08:59. Unix timestamps can support second, millisecond, microsecond, or nanosecond. TickTock only supports up to millisecond precision, to be consistent with OpenTSDB.
- Timestamps with 10 digits or less are treated as seconds.
- Timestamps with 11 to 13 digits are treated as milliseconds.
- Timestamps with 14 or more digits are not supported (A 400 BAD REQUEST status code will be returned by TickTock servers).
- A write request with multiple data points will be processed at TickTock's best effort. If there is a bad data point, the rest of data point would be inserted at TickTock's best effort, and 400 is returned as status code.
TickTock current version is v0.20.1. It also supports relative time format (<amount><time unit>-ago). Time unit can be one of the following:
- ms - Milliseconds
- s - Seconds
- m - Minutes
- h - Hours
- d - Days (24 hours)
- w - Weeks (7 days)
- n (or mon) - Months (30 days)
- yr - Years (365 days)
In the future, we will support absolute formatted time such as yyyy/MM/dd-HH:mm:ss.
We will show you write and read examples with different tsdb.timestamp.resolution settings in the following.
ylin30@pi5:~/ticktock $ ./bin/tt -c conf/tt.conf &
[1] 27320
ylin30@pi5:~/ticktock $ TickTockDB v0.20.1, Maintained by
Yongtao You ([email protected]) and Yi Lin ([email protected]).
This program comes with ABSOLUTELY NO WARRANTY. It is free software,
and you are welcome to redistribute it under certain conditions.
For details, see <https://www.gnu.org/licenses/>.
Writing to log file: /home/ylin30/ticktock/log/ticktock.log
TickTockDB is ready...
ylin30@pi5:~/ticktock $ ./admin/config.sh
{
}
ylin30@pi5:~/ticktock $
Note that the default resolution is second, so we don't need to change the default config (i.e., conf/tt.conf).
First, let's insert a data point with a 8 digits timestamp (metric name: testM1, tag: host=host1, timestamp: 12345678, value=8).
curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 12345678 8 host=host1"
Let's query it immediately after the insertion.
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678":8.0}}]
Note
- The query tries to retrieve all data points of metric testM4 since 12345678 Unix epoch time.
- The parameter
m=avg:testM4is in the format ofm=<aggregator>:<metric>. It doesn't specify any downsampling interval so data points in a time series would not be downsampled, though data points in different time series would be aggregated averagely (because ofavg:). Since there is only one time series (i.e., testM4 with a tag host=host1), the query is meant to return all data points explicitly since 12345678. ** Please refer to OpenTSDB query string format for details. - The int value 8 is stored as a floating point internally.
We can add a msResolution=true parameter to get millisecond timestamp precision in results.
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":8.0}}]
[yi-IdeaPad ~]$
You can see that the timestamp of the data point inserted has 3 more digits appended.
Next, we continued to insert two more data points with 9 and 10 digits. The results show that their timestamps were treated as seconds.
[yi-IdeaPad ~]$ curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 123456789 9 host=host1"
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":8.0,"123456789000":9.0}}]
[yi-IdeaPad ~]$
[yi-IdeaPad ~]$ curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 1234567890 10 host=host1"
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":8.0,"123456789000":9.0,"1234567890000":10.0}}]
[yi-IdeaPad ~]$
Next let's insert a data point with 11 digits timestamp (12345678901) and value as 11.
[yi-IdeaPad ~]$ curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 12345678901 11 host=host1"
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":11.0,"123456789000":9.0,"1234567890000":10.0}}]
[yi-IdeaPad ~]$
NOTE
- TickTock will overwrite previous inserted data points if a new data point with exactly the same timestamp arrives.
-
12345678901was treated as millisecond so the data point was at12345678.901second. Since TickTock server is running with timestamp precision as second (i.e.,tsdb.timestamp.resolution=secondby default), TickTock will store it at12345678second, which will overwrite the previously inserted data point at12345678.000.
We continued to insert 2 more data points with 12 and 13 digits timestamps. They are all treated milliseconds.
[yi-IdeaPad ~]$ curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 123456789012 12 host=host1"
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":11.0,"123456789000":12.0,"1234567890000":10.0}}]
[yi-IdeaPad ~]$
[yi-IdeaPad ~]$ curl -s -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 1234567890123 13 host=host1"
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":11.0,"123456789000":12.0,"1234567890000":13.0}}]
All requests with 14 or more digits timestamps would be rejected as BAD REQUEST (400 status code).
[yi-IdeaPad ~]$ curl -v -XPOST 'http://10.0.0.5:6182/api/put' -d "put testM4 12345678901234 14 host=host1"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 10.0.0.5:6182...
* TCP_NODELAY set
* Connected to 10.0.0.5 (10.0.0.5) port 6182 (#0)
> POST /api/put HTTP/1.1
> Host: 10.0.0.5:6182
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 39
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 39 out of 39 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host 10.0.0.5 left intact
If we issued another query, the result do not contain any data points with value as 14.
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM4&msResolution=true'
[{"metric":"testM4","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":11.0,"123456789000":12.0,"1234567890000":13.0}}][yi-IdeaPad ~]$
Note that there is only one single data point in a request in the above write examples. If you tried to insert multiple data points in a request, and the request has some data points in bad format (e.g., with 14 digits timestamps), then TickTock will try its best effort to insert all data points in good format, and still return a 400 status code. We say BEST EFFORT because sometimes a previous badly formatted data point might confuse TickTock in interpreting the rest of the request.
[yi-IdeaPad ~]$ curl -v -XPOST 'http://10.0.0.5:6182/api/put' -d $'put testM5 123456789 9 host=host1\nput testM5 1234567890 10 host=host1\nput testM5 12345678901 11 host=host1\nput testM5 12345678901234 14 host=host1'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 10.0.0.5:6182...
* TCP_NODELAY set
* Connected to 10.0.0.5 (10.0.0.5) port 6182 (#0)
> POST /api/put HTTP/1.1
> Host: 10.0.0.5:6182
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 146
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 146 out of 146 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host 10.0.0.5 left intact
[yi-IdeaPad ~]$
For example, the above write request tried to insert 4 data points (with 9, 10, 11, 14 digits timestamps). The returned code is 400 (Bad request). But the 3 data points with 9, 10, 11 digits timestamp are inserted correctly.
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM5&msResolution=true'
[{"metric":"testM5","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":11.0,"123456789000":9.0,"1234567890000":10.0}}]
[yi-IdeaPad ~]$
ylin30@pi5:~/ticktock $ ./admin/stop.sh
Exiting now.
ylin30@pi5:~/ticktock $ Start shutdown process...
Shutdown process complete
[1]+ Done ./bin/tt -c conf/tt.conf
ylin30@pi5:~/ticktock $ rm -rf ./data ./log
ylin30@pi5:~/ticktock $ ./bin/tt -c conf/tt.conf --tsdb.timestamp.resolution millisecond --http.server.port 6182,6183 &
[1] 27566
ylin30@pi5:~/ticktock $ TickTockDB v0.20.1, Maintained by
Yongtao You ([email protected]) and Yi Lin ([email protected]).
This program comes with ABSOLUTELY NO WARRANTY. It is free software,
and you are welcome to redistribute it under certain conditions.
For details, see <https://www.gnu.org/licenses/>.
Writing to log file: /home/ylin30/ticktock/log/ticktock.log
TickTockDB is ready...
ylin30@pi5:~/ticktock $ ./admin/config.sh
{
"http.server.port": "6182,6183",
"tsdb.timestamp.resolution": "millisecond",
}
ylin30@pi5:~/ticktock $
NOTE
- We strongly suggest to use
<ticktock>/admin/stop.shto gracefully shutdown TickTock. - By removing the data folders, you will have a completely new TickTock running.
- You can specify SOME parameters in command lines instead of updating them in config files.
Skipped for simplicity since it is same as Section 2.
The write part is same as section 2 above. The read results are different:
[yi-IdeaPad ~]$ curl -s 'http://10.0.0.5:6182/api/query?start=12345678&m=avg:testM5&msResolution=true'
[{"metric":"testM5","tags":{"host":"host1"},"aggregateTags":[],"dps":{"12345678000":8.0,"12345678901":11.0,"123456789000":9.0,"123456789012":12.0,"1234567890000":10.0,"1234567890123":13.0}}]
[yi-IdeaPad ~]$
The results show two data points in one second, i.e., "12345678.000" and "12345678.901", "123456789.000" and "123456789.012", and "1234567890.000" and "1234567890.123". It is because TickTock supports millisecond with tsdb.timestamp.resolution=millisecond internally. So it didn't cut off the last 3 digits in, e.g., 1234567890.123.
Skipped for simplicity since it is same as Section 2.
Skipped for simplicity since it is same as Section 2.