packed_sheets - ryzom/ryzomcore GitHub Wiki

A packed sheet is a way of storing multiple Georges sheets in a binary format. The packed sheet system loads all forms of a given type into a single binary file that can be quickly deserialized at startup. It also automatically detects when source sheets have been added, modified, or removed, and updates the packed file accordingly.

This avoids the overhead of parsing XML on every startup, which is critical for games that load thousands of data sheets.

Before using packed sheets, you must initialize the sheet ID system. CSheetId maps sheet filenames to compact 32-bit identifiers using a sheet_id.bin lookup file.

// Initialize sheet ID system (must be in CPath search paths)
NLMISC::CSheetId::init(false);

The false parameter means "do not remove unknown sheets" — sheet IDs that exist in sheet_id.bin but have no corresponding file on disk will be kept. Pass true to remove them.

The sheet_id.bin file is generated by the make_sheet_id tool (nel/tools/misc/make_sheet_id/). It scans directories for sheet files and assigns each a persistent 32-bit ID. IDs are only appended, never removed, ensuring stable references for save files and network protocols.

To use the packed sheet system, you define a C++ class (or struct) that conforms to the following interface:

struct TMyLoader
{
    // Read data from a Georges form into this struct's members.
    // Called when a sheet is new or has been modified since the last pack.
    void readGeorges(const NLMISC::CSmartPtr<NLGEORGES::UForm> &form,
                     const NLMISC::CSheetId &sheetId);

    // Standard NeL serialization for binary packed sheet storage.
    // Called to save this struct into the .packed_sheets file,
    // and to load it back on subsequent startups.
    void serial(NLMISC::IStream &s);

    // Return the version of the packed sheet format.
    // Increment this whenever you change the struct layout or serial method.
    // When the version changes, all packed sheets are rebuilt from source XML.
    static uint getVersion();

    // Called when a previously packed sheet no longer exists on disk.
    // Use this to clean up any resources or reset members.
    void removed();
};

• readGeorges uses the form loader API to extract values from the XML form. For example:

  void TMyLoader::readGeorges(const NLMISC::CSmartPtr<NLGEORGES::UForm> &form,
                              const NLMISC::CSheetId &sheetId)
  {
      const NLGEORGES::UFormElm &root = form->getRootNode();
      root.getValueByName(HitPoints, "HitPoints");
      root.getValueByName(Name, "Name");
  }

• serial is standard NeL serialization. It must serialize the same members that readGeorges populates:

  void TMyLoader::serial(NLMISC::IStream &s)
  {
      s.serial(HitPoints);
      s.serial(Name);
  }

• getVersion must be incremented any time you change the struct layout or the serial method. This is very important. When the version changes, the packed sheet system discards the entire binary cache and rebuilds it from the source XML forms.

• removed is called when a sheet that was previously in the packed file no longer exists on disk. It typically resets members to default values.

A good example of a packed sheet loader in the codebase is CSoundSerializer in sound_bank.cpp.

1. Declare a container — it must be a std::map<NLMISC::CSheetId, T>:

std::map<NLMISC::CSheetId, TMyLoader> MySheets;

2. Call the loadForm template function:

NLGEORGES::loadForm("my_extension", "my_sheets.packed_sheets", MySheets, true);

This function:

• Scans search paths for all files matching *.my_extension
• Compares them against the existing packed file my_sheets.packed_sheets
• For new or modified sheets: loads the XML form via readGeorges and adds to the container
• For unchanged sheets: deserializes from the binary cache via serial
• For removed sheets: calls removed and removes from the container
• If anything changed: rewrites the packed file

The packed file is only updated when the updatePackedSheet parameter is true (the default).

If you need to load sheets with multiple file extensions into the same container, use the vector overload:

std::vector<std::string> filters;
filters.push_back("creature");
filters.push_back("player");
NLGEORGES::loadForm(filters, "creatures.packed_sheets", MySheets, true);

template <class T>
void loadForm(const std::vector<std::string> &sheetFilters,
              const std::string &packedFilename,
              std::map<NLMISC::CSheetId, T> &container,
              bool updatePackedSheet = true,
              bool errorIfPackedSheetNotGood = true);

template <class T>
void loadForm(const std::string &sheetFilter,
              const std::string &packedFilename,
              std::map<NLMISC::CSheetId, T> &container,
              bool updatePackedSheet = true,
              bool errorIfPackedSheetNotGood = true);

The packed filename must have the .packed_sheets extension. This is enforced by an assertion. {.is-warning}

Once loaded, access sheets through the map using a CSheetId:

NLMISC::CSheetId id("sample_creature.creature");
auto it = MySheets.find(id);
if (it != MySheets.end())
{
    nlinfo("HitPoints: %d", it->second.HitPoints);
}

You can also construct a CSheetId from a 32-bit integer if you stored the ID elsewhere (e.g. in a network message or save file):

NLMISC::CSheetId id(someUint32Value);

The sheet_id.bin file should be generated through the build pipeline rather than manually. The build pipeline calls make_sheet_id (nel/tools/misc/make_sheet_id/) as part of the full data build process, ensuring that all sheet types across the project are indexed consistently. Manually running make_sheet_id is discouraged as it can lead to ID conflicts or missed sheets.

• Nevrax.org : PackedSheet (archive.org)