sdb - themeldingwars/Documentation GitHub Wiki

SDB, believed to stand for Static Database, are two files, ClientDB.SDB and CommonDB.SDB, that were used before the newer SD2 format for Firefall. They live in system/db.

The two formats are not variants of each other, they are laid out completely differently. The one thing worth knowing up front is that .sdb stores names: every table carries the SQL it was exported from and every field carries its name as a string. That makes an old .sdb the most useful reference there is when trying to recover the table and column names that the newer SD2 only keeps as hashes, which is what Brutus and SDB Table Contents are for.

010 Template

sdb.bt

The template handles version 5 files, obfuscated or not, and also a raw descrambled dump with no header. It works with both clientdb.sdb and commondb.sdb from build 1189.

Format

Little endian. Strings are null terminated, not length prefixed.

Header

Field	Type	Notes
`version`	`uint32`	`5`
`magic`	`uint32`	`0xDA7ABA5E` plain, or `0xBE7AFA5E` obfuscated
`timestamp`	`int64`	Unix epoch in seconds

When magic is 0xBE7AFA5E two extra fields follow before the table index, and everything after the index is scrambled and has to be unscrambled before the rest of the file can be parsed:

Field	Type	Notes
`patch`	`uint32`
`scramblerVersion`	`uint32`

In both cases the header then ends with the table index:

Field	Type	Notes
`tableCount`	`uint32`
`tableOffsets`	`uint32[tableCount]`	Absolute file offset of each table

Because every table is reachable through this index, tables can be read in any order and do not have to sit back to back.

Table

Field	Type	Notes
`id`	`uint32`
`numFields`	`uint8`
`fields`	`Field[numFields]`	See Field
`numDataBytes`	`int32`	Bytes of field data in a row
`numDataAndNullableBytes`	`int32`	`-1` when the table has no nullable fields
`numNullableFields`	`uint32`
`sqlFrom`	`string`	The table name, shown as `table` in the client console
`sqlOrderBy`	`string`
`sqlWhere`	`string`
`sqlMore`	`string`
`unknown`	`uint32`
`rowCount`	`uint32`
`rows`	`Row[rowCount]`	See Row

The four SQL strings are the query the table was exported with. They are the reason a .sdb is worth keeping around: sqlFrom gives you the real table name, in the same namespace::Table shape that SD2 hashes.

Field

Field	Type	Notes
`type`	`uint8`	See Types
`id`	`uint32`
`name1`	`string`	Base name
`name2`	`string`	Qualified / unique / index name
`name3`	`string`
`offset`	`uint32`	Byte offset of this field inside a row
`nullableIndex`	`int32`	Bit index in the row bitfield, `-1` if not nullable

The exact distinction between the three names has not been pinned down.

Row

Field	Type	Notes
`rowId`	`uint32`
`nullableBitfield`	`uint8`	Only present when `numNullableFields` is non-zero
cells		One per field, in field order

A bit that is set in nullableBitfield means the field is present. Fields whose bit is clear are skipped entirely, no placeholder bytes are written for them. This is the opposite of what SD2 does, where a set bit means null.

The 010 template only handles a single bitfield byte, so a table with more than 8 nullable fields would need it extended.

Unlike SD2, rows are not fixed stride and there is no separate pool: variable length values are written inline, so a row has to be walked field by field.

Types

Id	Type	Encoding
0	`Unknown`
1	`Byte`	`U08`
2	`UShort`	`U16`
3	`UInt`	`U32`
4	`ULong`	`U64`
5	`SByte`	`S08`
6	`Short`	`S16`
7	`Int`	`S32`
8	`Long`	`S64`
9	`Float`	`F32`
10	`Double`	`F64`
11	`String`	`slString`, inline null terminated string
12	`Vector2`	`vec2`, 2 floats
13	`Vector3`	`vec3`, 3 floats
14	`Vector4`	`vec4`, 4 floats
15	`Matrix4x4`	`mat4`, 16 floats
16	`Blob`	`slBlob`, `uint16` length then that many bytes
17	`Char`	`char`
18	`Box3`	`box3`, 6 floats
19	`Vector2Array`	`slArray<vec2>`, `uint8` count then count * 2 floats
20	`Vector3Array`	`slArray<vec3>`, `uint8` count then count * 3 floats
21	`Vector4Array`	`slArray<vec4>`, `uint8` count then count * 4 floats

Note that these ids are not the same as the SD2 ones. Char at 17 pushes Box3 and the array types one slot later than in .sd2.

The 010 template currently lists Vector2Array and Vector3Array both as 19, which is a typo. 19, 20, 21 above follows the obvious sequence but has not been verified against a file that actually uses a Vector4Array column.

Versions

ClientDB.SDB from Build 1189

Signsrch 0.2.4

- open file "C:\Program Files (x86)\Red 5 Studios\Firefall_1189\system\db\clientdb.sdb"
- 4246484 bytes allocated
- load signatures
- open file C:\Users\X\Desktop\signsrch\signsrch.sig
- 3075 signatures in the database
- start 8 threads
- start signatures scanning:

  offset   num  description [bits.endian.size]
  --------------------------------------------
  00004ce2 3048 DMC compression [32.le.16&]

- 1 signatures found in the file in 3 seconds
- done

CommonDB.SDB from Build 1189

Signsrch 0.2.4

- open file "C:\Program Files (x86)\Red 5 Studios\Firefall_1189\system\db\commondb.sdb"
- 3173194 bytes allocated
- load signatures
- open file C:\Users\X\Desktop\signsrch\signsrch.sig
- 3075 signatures in the database
- start 8 threads
- start signatures scanning:

  offset   num  description [bits.endian.size]
  --------------------------------------------
  00015215 3048 DMC compression [32.le.16&]
  001761f0 2417 MBC2 [32.le.248&]
  00236b31 1041 SSL3 #define [32.be.176&]
  0024708b 2418 MBC2 [32.be.248&]

- 4 signatures found in the file in 3 seconds
- done

Build 1265

The descrambled 1265 database has no header at all, it is just 306 tables back to back in the format above. The 010 template detects this by checking whether the first uint32 is 5.

Open questions

The scrambling used when the magic is 0xBE7AFA5E is not documented. scramblerVersion suggests there is more than one variant of it. It is not known whether it is the same Mersenne Twister XOR that SD2 uses, which would be the first thing to try.
What id on a table and on a field are. In SD2 the equivalent values are name hashes, so checking whether these match the FFnv32 of the names stored right next to them would be a cheap and useful confirmation.