Binary Bitstream Format Specification (.sec)

This document formalizes the binary bitstream layout, serialization protocol, and framing structures for SpaceEntropyCompression (.sec) files and network payloads. The format guarantees cross-platform byte determinism, constant-time header parsing, and random-access chunk decoding.

---

1. Top-Level Container Architecture

A .sec stream comprises a Global File Header, a series of self-contained Spatial Chunks, and an optional terminal Table of Contents (TOC) for random-access spatial querying:

+-----------------------------------------------------------------+
|                       Global File Header                        |
|                  Magic, Version, Flags, Global AABB             |
+-----------------------------------------------------------------+
|                        Spatial Chunk 0                          |
|             Header + Frequency Tables + ANS Bitstream           |
+-----------------------------------------------------------------+
|                        Spatial Chunk 1                          |
|             Header + Frequency Tables + ANS Bitstream           |
+-----------------------------------------------------------------+
|                              ...                                |
+-----------------------------------------------------------------+
|                        Spatial Chunk N-1                        |
+-----------------------------------------------------------------+
|                     Table of Contents (TOC)                     |
|           Chunk Offset Index, Spatial Bounds, CRC-32            |
+-----------------------------------------------------------------+

All multi-byte numeric fields are encoded in standard Little-Endian (LE) byte order.

---

2. Global File Header Specification

The global header occupies exactly 64 bytes, aligned to CPU cache-line boundaries:

#pragma pack(push, 1)
struct SECGlobalHeader {
    uint32_t magic;              // 0x31434553 ("SEC1" in ASCII)
    uint16_t version_major;      // Format version major (e.g., 1)
    uint16_t version_minor;      // Format version minor (e.g., 0)
    uint16_t profile_flags;      // Bitmask: [0: Lossless, 1: HasTensors, 2: HasReciprocal, 3: Hilbert, 4: Morton]
    uint8_t  dimension;          // Spatial dimension count (typically 3 or 4)
    uint8_t  coord_bit_depth;    // Coordinate bit depth (10, 16, 21, 32)
    uint64_t total_elements;     // Total spatial points / nodes in container
    uint32_t chunk_count;        // Total number of spatial chunks
    
    // Global Axis-Aligned Bounding Box (AABB) in double-precision float
    double   aabb_min[3];        // Minimum spatial bounds (X, Y, Z)
    double   aabb_max[3];        // Maximum spatial bounds (X, Y, Z)
    
    float    quant_scale;        // Quantization step scale factor (Δq)
    uint32_t toc_offset;         // File byte offset to Table of Contents (0 if streaming)
    uint32_t header_crc32;       // CRC-32 checksum of bytes 0..59
};
#pragma pack(pop)

Byte Offset	Field	Type	Description
0x00 - 0x03	magic	uint32	Must equal 0x31434553 ('S', 'E', 'C', '1')
0x04 - 0x07	version	uint16[2]	Major (1), Minor (0)
0x08 - 0x09	profile_flags	uint16	Bit 0: Lossless (1) or Bounded (0); Bit 1: Curvature Tensors included; Bit 3: Hilbert order
0x0A	dimension	uint8	Spatial manifold dimension ($d=3$ or $d=4$)
0x0B	coord_bit_depth	uint8	Quantization bits per coordinate ($16$ or $21$)
0x0C - 0x13	total_elements	uint64	Total coordinate entities stored
0x14 - 0x17	chunk_count	uint32	Total chunks in stream
0x18 - 0x2F	aabb_min[3]	double[3]	Global bounding box minimum ($X, Y, Z$)
0x30 - 0x47	aabb_max[3]	double[3]	Global bounding box maximum ($X, Y, Z$)
0x48 - 0x4B	quant_scale	float32	Quantization step size $\Delta q$
0x4C - 0x4F	toc_offset	uint32	File offset to Table of Contents
0x50 - 0x53	header_crc32	uint32	IEEE 802.3 CRC-32 checksum

---

3. Spatial Chunk Structure

Each chunk is self-contained and independently decodable without state dependencies on prior chunks:

#pragma pack(push, 1)
struct SECChunkHeader {
    uint32_t chunk_id;           // Zero-based sequential chunk index
    uint32_t point_count;        // Number of points in this chunk (typically 1024 - 16384)
    uint32_t compressed_bytes;   // Byte length of payload following this header
    uint16_t chunk_flags;        // Chunk-specific encoding flags
    uint16_t context_model_id;   // Probability model configuration ID
    
    // Chunk Local Bounding Box (Integer grid indices)
    uint32_t local_bbox_min[3];
    uint32_t local_bbox_max[3];
    
    uint32_t rANS_initial_states[4]; // Seed states for 4-way interleaved rANS decoder
    uint32_t payload_crc32;      // CRC-32 of compressed bitstream payload
};
#pragma pack(pop)

Chunk Payload Data Layout:

1. Probability Table Header (Quantized Frequencies):
   • Transmitted as run-length encoded symbol probabilities using a 12-bit total frequency sum ($M = 4096$).
2. rANS Packed Bitstream:
   • 4-way interleaved 16-bit word stream storing spatial Hilbert coordinate residuals ($\Delta h, \Delta x, \Delta y, \Delta z$).
3. Optional Curvature Tensor Stream:
   • Appended immediately if HasTensors flag is asserted, containing packed discrete metric operator indices ($q_{ij}$).

---

4. Table of Contents (TOC) & Spatial Indexing

To support fast spatial querying (e.g. view frustum culling, radius bounding queries) without scanning the entire file, the stream terminates with a TOC:

struct SECTOCEntry {
    uint64_t file_offset;        // Byte offset from start of file to chunk header
    float    bbox_min[3];        // Single-precision bounding box
    float    bbox_max[3];
    uint32_t point_count;        // Coordinate count in chunk
};

A client application loads the 64-byte file header, jumps directly to toc_offset, reads the compact TOC array, and selectively streams only the chunks intersecting the spatial region of interest.

---

5. Streaming & Error Resilience

• Stream Framing: For network socket streaming (TCP/QUIC) or IPC pipelines, the toc_offset is set to 0x00000000. The receiver processes chunks sequentially as they arrive over the wire.
• Integrity Validation: Every chunk header contains a payload_crc32. Corrupted packets are rejected before entering the entropy dequantizer, preventing buffer overflows and undefined memory behavior.