Flatbuffers - uccross/skyhookdm-ceph GitHub Wiki

Skyhook stores database table partitions into Ceph objects. The current data format for a partition is based on flatbuffers. Each partition is described by a ROOT_TABLE flatbuffer, which contains some metadata regarding the partition followed by a sequence of REC_TABLE flatbuffers. Each REC_TABLE has some metadata regarding the record (row here) followed by a single flexbuffer containing the row's data. The following diagram describes this data layout:

• data-layout-object

Furthermore, within an object, the data partition may be further subdivided into a sequence of sub-partitions. Hence, a single object may contain a sequence of ROOT_TABLE flatbuffers. The union of all REC_TABLE flatbuffers in an object is the entire data partition.

More information about flatbuffers:

• https://google.github.io/flatbuffers/.

Skyhook Flatbuffers experiments for SkyhookDB data layout within Ceph objects:

• Experimental results for various potential data layouts with flatbuffers/flexbuffers (Billy Lai).