Tutorial Basic Serialization - liuli-neko/NekoProtoTools GitHub Wiki

Core Concepts: Serializers Explained

All modules of this library are implemented around this component

Serialization is the process of converting an in-memory C++ object (like a struct or class instance) into a sequence of bytes, often for storage or transmission over a network. Deserialization is the reverse process: reconstructing the C++ object from that sequence of bytes.

NekoProtoTools provides a flexible system for handling serialization and deserialization through its Serializer components.

Telling the Library What to Serialize: NEKO_SERIALIZER

Before any object can be serialized, you need to tell NekoProtoTools which of its members should be included in the process. This is done using the NEKO_SERIALIZER macro within your struct or class definition.

#include <nekoproto/proto/serializer_base.hpp> // Required for NEKO_SERIALIZER
#include <string>
#include <vector>

struct MyData {
    int         id;
    std::string name;
    double      score;
    // This field will NOT be serialized or deserialized
    bool        internal_flag = false;

    // Declare the members to be processed by serializers
    NEKO_SERIALIZER(id, name, score);
};

struct NestedData {
    std::string category;
    MyData      details; // Nested struct that also uses NEKO_SERIALIZER

    NEKO_SERIALIZER(category, details);
};
  • Place NEKO_SERIALIZER(...) inside the class/struct definition.
  • List the member variables you want to serialize/deserialize inside the parentheses, separated by commas.
  • The order you list them in can matter, especially for binary formats (it determines the order in the byte stream).
  • Members not listed in NEKO_SERIALIZER will be ignored during serialization and deserialization.
  • For serialization to work, the types of the listed members must be supported by the chosen serializer (see Supported Types).

Using Serializers Directly (Basic Usage)

For simple tasks where you just need to convert an object to bytes (e.g., JSON) and back, you can use the serializer classes directly without the full protocol management system (IProto).

Each serialization format (like JSON, Binary) provides two core classes within its namespace:

  • OutputSerializer: Takes a C++ object and writes its serialized representation to an output buffer (e.g., std::vector<char>).
  • InputSerializer: Reads data from an input buffer (e.g., const char*, std::vector<char>) and populates a C++ object.

Example (using JsonSerializer):

#include <nekoproto/proto/serializer_base.hpp>
#include <nekoproto/proto/json_serializer.hpp> // Include the specific serializer
#include <nekoproto/proto/types/string.hpp>    // Include support for std::string
// #include <nekoproto/proto/types/types.hpp>  // Or include all supported types
#include <iostream>
#include <string>
#include <vector>

// Assume MyData struct from the NEKO_SERIALIZER example above

int main() {
    using namespace NekoProto; // Optional: NekoProto namespace

    MyData data_out;
    data_out.id = 101;
    data_out.name = "Neko";
    data_out.score = 99.5;

    // --- Serialization ---
    std::vector<char> buffer;
    // 1. Create an OutputSerializer for the desired format (JSON)
    JsonSerializer::OutputSerializer serializer(buffer);
    // 2. Serialize the object using the call operator ()
    serializer(data_out);
    // 3. Finalize (optional, destructor does this too)
    serializer.end();

    std::string json_str(buffer.begin(), buffer.end());
    std::cout << "Serialized JSON: " << json_str << std::endl;
    // Output: Serialized JSON: {"id":101,"name":"Neko","score":99.5}


    // --- Deserialization ---
    MyData data_in;
    // 1. Create an InputSerializer for the same format
    JsonSerializer::InputSerializer deserializer(buffer.data(), buffer.size());
    // 2. Deserialize into the object using the call operator ()
    //    The operator returns true on success, false on failure.
    if (deserializer(data_in)) {
        std::cout << "Deserialized Data: id=" << data_in.id
                  << ", name=" << data_in.name
                  << ", score=" << data_in.score << std::endl;
        // Output: Deserialized Data: id=101, name=Neko, score=99.5
    } else {
        std::cerr << "Deserialization failed!" << std::endl;
    }

    return 0;
}

This direct usage gives you fine-grained control over which serializer format is used at the exact moment of serialization/deserialization.

Built-in Serializers

NekoProtoTools comes with support for common serialization formats:

1. JsonSerializer

  • Header: #include <nekoproto/proto/json_serializer.hpp>
  • Format: JSON (JavaScript Object Notation) - a human-readable text format.
  • Backends: Can use RapidJSON or SIMDJson (primarily for high-speed input). Select the backend using configuration flags (enable_rapidjson, enable_simdjson) in your xmake.lua. See Installation.
  • Use Cases: Web APIs, configuration files, debugging, interoperability.
  • Supported Types: See JSON Type Support.

2. BinarySerializer

  • Header: #include <nekoproto/proto/binary_serializer.hpp>
  • Format: A custom, compact binary format. Not generally human-readable.
  • Byte Order: Uses Network Byte Order (Big Endian) for multi-byte numeric types, ensuring consistency across different machine architectures.
  • Use Cases: Network protocols where bandwidth or performance is critical, saving data compactly.
  • Supported Types: See Binary Type Support.

3. XmlSerializer

  • Header: #include <nekoproto/proto/xml_serializer.hpp>
  • Format: XML (Extensible Markup Language).
  • Backend: Uses RapidXML. Enable with the enable_xml flag. See Installation.
  • Current Limitation: Currently only supports deserialization (reading XML data into C++ objects). Serialization to XML is not implemented yet.
  • Use Cases: Interacting with systems that use XML, parsing configuration files.
  • Supported Types: Similar to Binary Serializer. See XML Type Support.

Serialization with Protocol Management (IProto)

When you define your types as full-fledged protocols using NEKO_DECLARE_PROTOCOL (see Protocol Management), you associate a default serializer with that protocol type.

#include <nekoproto/proto/proto_base.hpp>
#include <nekoproto/proto/json_serializer.hpp> // Include the chosen default
#include <nekoproto/proto/types/string.hpp>
#include <vector>
#include <string>

struct UserLogin {
    std::string user;
    std::string pass;

    NEKO_SERIALIZER(user, pass);
    // Declare as a protocol, specifying JsonSerializer as the default
    NEKO_DECLARE_PROTOCOL(UserLogin, NekoProto::JsonSerializer);
};

// ... elsewhere ...
using namespace NekoProto;

// Create an instance via the protocol interface
IProto login_proto = UserLogin::emplaceProto("admin", "secret");

// Serialize using the *default* serializer (JsonSerializer)
std::vector<char> data = login_proto.toData();

// Deserialize using the *default* serializer
IProto received_proto = UserLogin::createEmptyProto(); // Create an empty shell
if (received_proto.fromData(data)) {
   // Deserialization successful
   auto* login_ptr = received_proto.cast<UserLogin>();
   // ... use login_ptr ...
}

In this case, toData() and fromData() automatically use the JsonSerializer because it was specified in NEKO_DECLARE_PROTOCOL. You don't explicitly create the JsonSerializer::OutputSerializer or InputSerializer yourself.

Key Takeaways

  • Use NEKO_SERIALIZER to mark members for serialization.
  • You can use OutputSerializer and InputSerializer directly for basic conversion tasks.
  • NekoProtoTools provides JsonSerializer, BinarySerializer, and XmlSerializer (input-only) out of the box.
  • When using IProto, a default serializer is associated with the protocol type via NEKO_DECLARE_PROTOCOL.
  • The types of members you want to serialize must be supported.

Next Steps

⚠️ **GitHub.com Fallback** ⚠️