Binary Type Support - liuli-neko/NekoProtoTools GitHub Wiki

This page details the C++ types supported by the NekoProto::BinarySerializer and describes how they are represented in the resulting binary byte stream.

Related Concepts:

• Serializers Interface|Serializers-Interface
• Supported Types Overview|Supported-Types-Overview

To use the BinarySerializer, ensure you include the core header <nekoproto/proto/binary_serializer.hpp>.

Additionally, remember to include the necessary type support headers (<nekoproto/proto/types/...>) for the standard library types you wish to serialize/deserialize (e.g., <nekoproto/proto/types/string.hpp>, <nekoproto/proto/types/vector.hpp>), or include the general <nekoproto/proto/types/types.hpp>. Fundamental types and std::optional are typically covered by including <nekoproto/proto/binary_serializer.hpp>.

• Compact: Generally produces smaller output compared to text-based formats like JSON or XML.
• Not Human-Readable: The output is a sequence of bytes intended for machine processing.
• Network Byte Order: Multi-byte numeric types (int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, float, double, and size/length prefixes) are stored in Big Endian format (most significant byte first). This ensures consistency when sending data between systems with different native byte orders (endianness).
• Fixed/Variable Length: Basic types have fixed sizes. Strings and containers have a variable size, prefixed by their length/element count.

The following table outlines the mapping between C++ types and their binary representation:

C++ Type	Supported	Size (Bytes)	Required Type Header (nekoproto/proto/types/...)	Notes
bool	✅	1	(builtin)	0x00 for false, 0x01 for true.
int8_t, uint8_t	✅	1	(builtin)	Stored directly.
int16_t, uint16_t	✅	2	(builtin)	Network Byte Order (Big Endian).
int32_t, uint32_t	✅	4	(builtin)	Network Byte Order (Big Endian).
int64_t, uint64_t	✅	8	(builtin)	Network Byte Order (Big Endian).
float	✅	4	(builtin)	IEEE 754 format, Network Byte Order (Big Endian).
double	✅	8	(builtin)	IEEE 754 format, Network Byte Order (Big Endian).
std::string	✅	4 (length) + N (content)	string.hpp	uint32_t length prefix (Big Endian), then raw bytes.
std::u8string (C++20)	✅	4 (length) + N (content)	u8string.hpp	uint32_t length prefix (Big Endian), then raw bytes.
enum class/enum	✅	Size of underlying type	enum.hpp	Serialized as its underlying integer value (Big Endian if >1 byte).
std::optional<T>	✅	1 (flag) [+ Size of T]	(builtin)	1-byte flag (0=null, 1=present), followed by T if present.
std::vector<T>	✅	4 (count) + N * Size of T	vector.hpp	uint32_t count prefix (Big Endian), then elements.
std::list<T>	✅	4 (count) + N * Size of T	list.hpp	uint32_t count prefix (Big Endian), then elements.
std::deque<T>	✅	4 (count) + N * Size of T	deque.hpp	uint32_t count prefix (Big Endian), then elements.
std::array<T, N>	✅	N * Size of T	array.hpp	Elements stored consecutively. No size prefix needed.
std::set<T>	✅	4 (count) + N * Size of T	set.hpp	uint32_t count prefix (Big Endian), then elements.
std::multiset<T>	✅	4 (count) + N * Size of T	multiset.hpp	uint32_t count prefix (Big Endian), then elements.
std::unordered_set<T>	✅	4 (count) + N * Size of T	unordered_set.hpp	uint32_t count prefix (Big Endian), then elements.
std::unordered_multiset<T>	✅	4 (count) + N * Size of T	unordered_multiset.hpp	uint32_t count prefix (Big Endian), then elements.
std::map<K, V>	✅	4 (count) + N * (Size K + Size V)	map.hpp	uint32_t count prefix (Big Endian), then K, V pairs.
std::multimap<K, V>	✅	4 (count) + N * (Size K + Size V)	multimap.hpp	uint32_t count prefix (Big Endian), then K, V pairs.
std::unordered_map<K, V>	✅	4 (count) + N * (Size K + Size V)	unordered_map.hpp	uint32_t count prefix (Big Endian), then K, V pairs.
std::unordered_multimap<K, V>	✅	4 (count) + N * (Size K + Size V)	unordered_multimap.hpp	uint32_t count prefix (Big Endian), then K, V pairs.
std::pair<T1, T2>	✅	Size T1 + Size T2	pair.hpp	T1 followed immediately by T2.
std::tuple<T...>	✅	Sum of Sizes of T...	tuple.hpp	Elements stored consecutively in order.
std::variant<T...>	✅	4 (index) + Size of Active T	variant.hpp	uint32_t index prefix (Big Endian), then active value.
std::bitset<N>	✅	4 (size N) + ceil(N/8)	bitset.hpp	Size prefix (Big Endian), then packed bytes.
std::shared_ptr<T>	✅	1 (flag) [+ Size of T]	shared_ptr.hpp	Same as std::optional<T>.
std::unique_ptr<T>	✅	1 (flag) [+ Size of T]	unique_ptr.hpp	Same as std::optional<T>.
std::atomic<T>	✅	Size of T	atomic.hpp	Serializes the contained value T.
Custom struct/class (using macro)	✅	Sum of Sizes of Members	(user-defined)	Members serialized consecutively in order listed in macro.
Protocol class (using macros)	✅	Sum of Sizes of Members	(user-defined)	Members serialized consecutively in order listed in macro.
NamePairValue<std::string, T>	✅	4+len(name) + len(T)	(builtin)	Used internally/special cases; name + value.
std::any	❌	-	-	Not supported.

(builtin): Support provided directly by <nekoproto/proto/types/types.hpp>.

• Containers & Strings: All variable-length containers (except std::array) and strings are prefixed with a 4-byte count/length field (uint32_t in Network Byte Order) indicating the number of elements or bytes that follow.
• Custom Structs/Classes: Members declared using NEKO_SERIALIZER are serialized sequentially in the exact order they are listed in the macro. The total size is the sum of the sizes of the individual members.
• Pointers: Raw pointers are not directly supported. Use smart pointers or manage serialization explicitly.

The BinarySerializer provides a compact representation suitable for network transmission or storage where human readability is not a requirement. Careful attention to the defined order and sizes is necessary when attempting to interoperate with other systems or manually parse the binary data.