sd2 - themeldingwars/Documentation GitHub Wiki

Static Database, clientdb.sd2 located in system/db is the database file that contains the stats and item info for the items and entities in the game.

It replaced the older SDB format. The biggest practical difference between the two is that .sdb stores table and column names, while .sd2 stores only 32 bit hashes of them, which is why a name cracker like Brutus exists.

• Reader and Writer have been implemented in FauFau: StaticDB.cs

• Brutus - A SDB table & field name cracker
• SDBrowser - Browser for Firefalls Static Database

• Magic is 0xDA7ABA5E, "DATABASE" in leetspeak. Read as a signed int32 that is -629,491,106
• The most recent file version is 12, see Version History
• The payload is zlib deflate, and is additionally obfuscated, see Obfuscation

A .sd2 is a 128 byte header followed by a single obfuscated, compressed payload. Everything else, all the tables, rows and pooled values, lives inside that one payload.

Header               128 bytes
  Payload            obfuscated with MT19937, then:
    Compression header
    zlib stream, which inflates to:
      Memory version + table count
      TableInfo   [tableCount]
      FieldInfo   [numFields] per table
      RowInfo     [tableCount]
      Pool offset
      Row data
      Pool data

Always 128 bytes long.

The value 13 seen in shipped client files is ObfuscatedPool | Compressed | Client.

When the ObfuscatedPool flag is set, the whole payload is XORed with a keystream from a Mersenne Twister seeded with the FNV hash of the patch name from the header, e.g. beta-1869.

The generator is a stock MT19937: init constant 0x6C078965, twist constant 0x9908B0DF, and the standard tempering. The part that is easy to get wrong is how the output is consumed. For a payload of length L:

• L >> 2 full 32 bit outputs are written to the keystream little endian, four bytes each.
• The remaining L & 3 bytes each take the low byte of one further output.

The seed hash is FNV-1a over the ascii patch name with two extra mixing steps:

uint FFnv32(byte[] name)
{
    uint hash = 0x811C9DC5U;
    for (var i = 0; i < name.Length; i++)
        hash = 0x1000193U * (hash ^ name[i]);

    hash = 9U * (8193U * hash ^ ((8193U * hash) >> 7));
    return 33U * (hash ^ (hash >> 17));
}

The same hash is used for the table and column ids, see Name hashing.

After deobfuscation the payload starts with a small compression header, except on file version 7 (build 1297) where the zlib stream starts immediately and the inflated size is not recorded anywhere.

Everything below is an offset into the inflated buffer.

Then tableCount TableInfo records:

Then, for every table in order, numFields FieldInfo records:

start is not always the sum of the preceding field widths. Where there is a gap, treat it as padding and skip it, it is most likely a column that was removed without repacking the table.

Then tableCount RowInfo records:

And finally a uint32 poolOffset, the offset of the pool data, followed by 33 bytes that appear to be padding.

Row n of a table starts at rowOffset + (numBytes * n), and each field within it at its own start. When nullableBitfields is non-zero, that many bytes follow the field data. A bit that is set means the corresponding field is null. The bits are used in nullableIndex order.

Each table's block of rows is padded so it ends on a 128 byte boundary.

Variable length values are not stored in the row. The row holds a key into the pool that begins at poolOffset, and the pool entry is itself XOR obfuscated. How the key is encoded depends on memoryVersion:

1002, the key is a uint32:

• address = key >> 1
• If the low bit of the key is set, seek to address and read a uint16 length, the data follows it.
• Otherwise the length is the high byte of the key, and address &= 0x7FFFFF.
• The entry data is XORed with an MT19937 keystream seeded with the key, consumed exactly as described in Obfuscation.

1000, the key is a uint64:

• address = key & 0xFFFFFFFF
• length = key >> 32
• The keystream is seeded with the row index instead of the key.

The types that live in the pool are String, Blob, ByteArray, UShortArray, UIntArray, Vector2Array, Vector3Array and Vector4Array. Everything else is stored inline in the row.

The type ids are not the same as the ones used by the older SDB format, which has a shorter list with Char and Box3 at 17 and 18.

Table ids and column ids are FFnv32 (see Obfuscation) of the name, so looking a table up by name is just hashing the name and searching the table ids. Table names look like dbitems::Weapons, i.e. the namespace and the table joined with ::.

The hash is not reversible, which is what Brutus is for, and why SDB Table Contents is worth keeping up to date.

This snippet predates the working implementation and is kept for reference only. It does not produce an inflatable stream, because it takes one byte per 32 bit generator output (r >> 1) rather than four bytes per output as described in Obfuscation. The fnv function in it is correct.

#include <iostream>
#include <fstream>
#include <vector>
#include <iterator>

unsigned int fnv(char* a1)
{
  char v1;
  int i;

  v1 = *a1;
  for(i = 0x811C9DC5; *a1; v1 = *a1)         // FNV hash function
  {
    ++a1;
    i = 0x1000193 * (i ^ (unsigned int)v1);
  }
  unsigned int res = 33 * (9 * (8193 * i ^ ((unsigned int)(8193 * i) >> 7)) ^ (9 * (8193 * i ^ ((unsigned int)(8193 * i) >> 7)) >> 17));
  return res;
}

struct MT {
  uint32_t *next;
  uint32_t items;
  uint32_t mt[624];
};

static uint8_t MT_getnext(struct MT *MT) {
  uint32_t r;

  if (!--MT->items) {
    uint32_t *mt = MT->mt;
    unsigned int i;

    MT->items = 624;
    MT->next = mt;

    for (i=0; i<227; i++)
      mt[i] = ((((mt[i] ^ mt[i+1])&0x7ffffffe)^mt[i])>>1)^((0-(mt[i+1]&1))&0x9908b0df)^mt[i+397];
    for (; i<623; i++)
      mt[i] = ((((mt[i] ^ mt[i+1])&0x7ffffffe)^mt[i])>>1)^((0-(mt[i+1]&1))&0x9908b0df)^mt[i-227];
    mt[623] = ((((mt[623] ^ mt[0])&0x7ffffffe)^mt[623])>>1)^((0-(mt[0]&1))&0x9908b0df)^mt[i-227];
  }

  r = *(MT->next++);
  r ^= (r >> 11);
  r ^= ((r & 0xff3a58ad) << 7);
  r ^= ((r & 0xffffdf8c) << 15);
  r ^= (r >> 18);
  return (uint8_t)(r >> 1);
}

static void MT_decrypt(unsigned char *buf, unsigned int size, uint32_t seed) {
  struct MT MT;
  unsigned int i;
  uint32_t *mt = MT.mt;

  *mt=seed;
  for(i=1; i<624; i++)
    mt[i] = i+0x6c078965*((mt[i-1]>>30)^mt[i-1]);
  MT.items = 1;
  MT.next = MT.mt;

  while(size--)
    *buf++ ^= MT_getnext(&MT);
}

int main()
{
    unsigned int hash = fnv("stabilization-1350");

    std::ifstream is ("db.sdb", std::ios::binary);
    if(is)
    {
        // get length of file:
        is.seekg (0, is.end);
        int length = is.tellg();
        is.seekg (0, is.beg);

        std::vector<char> buffer(length, 0);

        //std::cout << "Reading " << length << " characters... \n";
        // read data as a block:
        is.read(buffer.data(), length);

        if (!is)
            std::cout << "error: only " << is.gcount() << " could be read\n";
        is.close();

        MT_decrypt(reinterpret_cast<unsigned char*>(buffer.data()), buffer.size(), hash);

        std::ofstream os ("out.zip", std::ios::binary);

        if(os)
        {
            std::copy(buffer.begin(), buffer.end(), std::ostream_iterator<char>(os));
            os.close();
        }
        else
            std::cerr << "Couldn't open out.zip\n";
    }
}

struct FILE
{
    struct HEADER // size 128b
    {
        uint magic;        // 0xDA7ABA5E
        uint version;
        uint payloadSize;
        uint flags;        // bitmask, 13 = ObfuscatedPool | Compressed | Client
        uint64 timestamp;  // unix epoch in microseconds
        char firefallPatch[104]; // beta-XXXX, null terminated and zero padded
    } header;

    byte payload[header.payloadSize];

} file;

• vbindiff - comparing binary files
• signsrch - Search a file for compressed streams

• Latest: prod-1962
• Earliest: prod-1931

• Latest: beta-1869
• Earliest: beta-1475

• Latest: beta-1460
• Earliest: beta-1384

• Latest: beta-1297
• Earliest: beta-1297
• No compression header, the zlib stream starts at the beginning of the payload