Tarantar - tsafin/tarantool GitHub Wiki

tarantar - tarantool xlog compression utility

Overview

Tarantar is used to make periodic snapshot files from a previously created snapshot and xlog (write-ahead log) files.

It uses the tarantool configuration file for its work.

Unlike tarantool, it tries to save memory by indexing only the key-part of a tuple and taking a reference to the tuple on disk.

Moreover, result key size is calculated as a sum of key types used for a tarantool composite index. And if any key has type of STR, or if the total key size is more than 20 bytes, then tarantar will take a SHA1 checksum from every key that is destined for such an index.

Due to the fact that a tarantool update operation can update the primary key, tarantar has to save a full updated tuple in-memory.

So, it's possible to assume that the total memory usage for an index used by tarantar is minimally 13 bytes and maximally 29 bytes + a correction for updated keys. For example, if a data set has 50M string keys, the total memory consumption ideally would be something near 1.6GB + a correction for updated keys.

The used indexing scheme is most beneficial for situations where the source data have small keys and large values.

How to use it

tarantar -c tarantool.cfg

Tarantool configuration file must be specified with configured snap_dir and wal_dir options.

Also, it is possible to put a hard limit on total memory consumption with --limit options, to set periodic intervals in which tarantar will try to take a new snapshot, and to specify start lsn number.