Vinyl Disk Layout - tsafin/tarantool GitHub Wiki

Tarantool 1.7.4 has the following disk layout:

├── <wal_dir>
    ├── 00000000000000000000.xlog
    ├── 00000000000000000047.xlog
    ├── 00000000000000000050.xlog
    ├── <wal_lsn>.xlog
    ├── 00000000000000000000.xctl
    ├── 00000000000000000050.xctl
    ├── <checkpoint_lsn>.xctl

├── <memtx_dir>
    ├── 00000000000000000000.snap
    ├── 00000000000000000050.snap
    └── <checkpoint_lsn>.snap

├── <vinyl_dir>
    └── 512 <!-- space_id
        ├── 0 <!-- primary key
        |    ├── 00000000000000000000.index
        |    ├── 00000000000000000000.run
        |    ├── 00000000000000000055.index
        |    ├── 00000000000000000055.run
        |    ├── <dump_lsn>.index
        |    └── <dump_lsn>.run
        ├── 1 <!-- secondary index
        |    ├── 00000000000000000000.index
        |    ├── 00000000000000000000.run
        |    ├── 00000000000000000032.index
        |    ├── 00000000000000000032.run
        |    ├── <dump_lsn>.index
        |    └── <dump_lsn>.run

• .xlog - write-ahead-log (common for all storage engines).
• .snap - consistent snapshot of all tuples from all Memtx spaces.
• .run - consistent snapshot of all tuples from a Vinyl range, like SST in LevelDB terminology. Contains tuples ordered by the key definition and grouped by pages.
• .index - contains the index of all pages in corresponding .run file and general information about this run.
• .xctl - physical journal of all operations with .run and .index files. .xlog, .snap, .index files will be stored in this journal in the future versions of Tarantool.

INDEX
0.13
Version: 1.7.4
Server: 39887eac-7447-4d74-bd54-485484b9887a

<FIXHEADER>
<run_info>
<page_info>
...
<page_info>
<EOF>

run_info is a xrow which contains general information about a Vinyl's run.

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = VY_INDEX_RUN_INFO = 100
• xrow body: map
  • VY_RUN_INFO_MIN_LSN = 1: unsigned = run_info->min_lsn
  • VY_RUN_INFO_MAX_LSN = 2: unsinged = run_info->max_lsn
  • VY_RUN_INFO_PAGE_COUNT = 3: unsinged = run_info->count
  • VY_RUN_INFO_BLOOM = 4: array
    • [0]: unsigned = bloom->table_size
    • [1]: unsigned = bloom->hash_count
    • [2]: raw = raw bloom filter table in little endian format

page_info is a xrow which contains information about a page in .run file.

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = VY_INDEX_PAGE_INFO = 101
• xrow body: map
  • VY_PAGE_INFO_OFFSET = 1: unsigned = page_info->offset;
  • VY_PAGE_INFO_SIZE = 2: unsigned = page_info->size;
  • VY_PAGE_INFO_UNPACKED_SIZE = 3: unsigned = page_info->unpacked_size;
  • VY_PAGE_INFO_ROW_COUNT = 4: unsigned = page_info->request_count
  • VY_PAGE_INFO_MIN_KEY = 5: array
  • VY_PAGE_INFO_PAGE_INDEX_OFFSET = 6: unsigned <!-- an offset to row index, see below

RUN
0.13
Version: 1.7.4
Server: 39887eac-7447-4d74-bd54-485484b9887a

<FIXHEADER> <!-- a page
<stmt>
..
<stmt>
<page_index>
...
<FIXHEADER>
<stmt>
..
<stmt>
<page_index>
<EOF>

stmt is a xrow which contains a single database operation in the format similar to WAL.

Current format:

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = IPROTO_REPLACE|IPROTO_UPSERT|IPROTO_DELETE
  • IPROTO_LSN: stmt->lsn
• xrow body: map
  • IPROTO_SPACE_ID: unsigned = key_def->space_id;
  • IPROTO_INDEX_ID: unsigned = key_def->id;
  • IPROTO_TUPLE: array -- REPLACE or UPSERT
  • IPROTO_KEY: array -- DELETE only
  • IPROTO_OPS: array -- UPSERT only

page_index - page index is a xrow which contains offsets for the current Vinyl page.

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = VY_RUN_PAGE_INDEX = 102
• xrow body: map
  • VY_PAGE_INDEX_INDEX = 1: page index in big endian

Current format:

VYMETA
0.13
Server: 39887eac-7447-4d74-bd54-485484b9887a

<FIXHEADER>
<xctl_request>
...
<FIXHEADER>
<xctl_request>
<EOF>

Proposed format:

XCTL
0.13
Version: 1.7.4
Server: 39887eac-7447-4d74-bd54-485484b9887a

<FIXHEADER>
<xctl_request>
...
<FIXHEADER>
<xctl_request>
<EOF>

Changes:

• Add Version: 1.7.4 instead of v13;
• Remove VClock: {};
• Rename VYMETA to XCTL.

Current format:

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = IPROTO_INSERT
• xrow body: map
  • IPROTO_TUPLE: array
    • 0: unsigned = record->type;
    • 1: map:
      • VY_LOG_KEY_INDEX_ID: unsigned = record->index_id
      • VY_LOG_KEY_RANGE_ID: unsigned = record->range_id
      • VY_LOG_KEY_RUN_ID: unsigned = record->run_id
      • VY_LOG_KEY_RANGE_BEGIN: tuple
      • VY_LOG_KEY_RANGE_END: tuple

Proposed format:

• xrow header: map
  • IPROTO_REQUEST_TYPE: unsigned = record->type
• xrow body: map
  • VY_XCTL_PATH: unsigned = record->path <!-- a relative path to the file
  • VY_XCTL_RUN_ID: unsigned = record->run_id
  • VY_XCTL_SPACE_ID: unsigned = record->space_id
  • VY_XCTL_INDEX_ID: unsigned = record->index_id
  • VY_XCTL_RANGE_ID: unsigned = record->range_id
  • VY_XCTL_RANGE_BEGIN: array
  • VY_XCTL_RANGE_END: array

Changes:

• Remove a map in an array in a map overengineering;
• Re-enumerate xrow body keys;
• Rename VY_LOG_KEY to VY_XCTL.