nsr - themeldingwars/Documentation GitHub Wiki

The Network Stream Replay (NSR) files are better known as replay files since they hold the streaming data which allows to replay in-game recordings.

The NSR format uses the little endian byte order, as such all the numbers are found "reversed" in the hex dump.

The streaming format inside the packets changes rather often, so a replay is only meaningful to a client that speaks the same streaming protocol version. That version is recorded in the description header, see Protocol version. The container itself (the sections described below) is a lot more stable, but it does have a few variants:

  • Version 2 replays from 2012 builds have a short 24 byte description without section offsets, see Version 2.
  • Replays up to about build 1780 have no index section, their indexOffset is 0.
  • Meta section version 3 is 8 bytes shorter than version 4, see Meta.

FauFau reads all 3'840 intact replays in Collections/NSR with these rules.

010 Editor Template

Implementations

  • Reader and Writer have been implemented in FauFau: Nsr.cs. It handles the variants and the cut off replays described here.
  • NSRViewer parses the same structures in NSR/
  • nsrinfo is a minimal C++ reader for the header and metadata

Tools

  • NSRViewer - View Firefall Network Stream Relay (NSR) files
  • nsrinfo - C++ CLI tool to extract basic NSR info

Compression

The NSR file is compressed with gzip and can for example get unpacked with 7zip. It's however important to use the gzip and not the normal zip algorithm for compressing the unpacked file again.

7z a -tgzip name.nsr name

Everything below describes the file after it has been decompressed. An uncompressed NSR starts with NSRD, a compressed one with the gzip magic 1F 8B. Only test those two bytes. Testing for 1F 8B 08 00 (the little endian uint 559903) misses gzip files that have header flags set, such as a stored file name.

Keep in mind:

  • Some replays were gzipped more than once, so unpack until the data no longer starts with 1F 8B. FauFau gives up after 4 layers.
  • Some replays are cut off, either in the middle of the gzip stream or in the middle of the last packet. A cut off gzip stream doesn't always throw, so compare the unpacked length with the size in the last 4 bytes of the gzip trailer as well. Unpack in small reads, since .NET's GZipStream throws at the cut and loses whatever the current read had decoded. A CopyTo with its 80 KB buffer lost up to 80 KB of packets that way, sometimes all of them. Everything before the cut is still a valid replay, FauFau keeps it and sets Truncated.

Layout

NSRD description   48 bytes, at offset 0
NSRI index         variable, starts right after the description
Meta               variable
Packets            until end of file

The description holds the offsets of the other three sections, so seek to them instead of reading the sections back to back. Replays up to about build 1780 have no index section at all, a reader that expects one right after the description reads their meta section 4 bytes off. The offsets should satisfy metaOffset >= 48, metaOffset + metaLength <= dataOffset and dataOffset <= file length, FauFau treats anything else as a broken file.

Format

NSRD - Description

48 bytes in version 5, which every replay in the collection outside of the 2012 builds uses. Version 2 is described below.

Field Type Notes
magic char[4] NSRD
version uint32 5
metaOffset uint32 Offset of the meta section. Equal to the combined size of the description and the index
metaLength uint32 Size of the meta section
indexOffset uint32 Offset of the index section, 48, which is also the size of the description. 0 when there is no index section
dataOffset uint32 Offset of the first packet
unknown uint32 Always 0
protocolVersion uint32 Streaming protocol version, see Protocol version
timestamp uint64 Unix epoch in microseconds, UTC. Garbage in a few replays, see below
indexInterval uint32 Milliseconds between index entries, the cvar replay.indexInterval, 5000 by default
unknown uint32 Always 0

A few clients wrote garbage into timestamp, a value that can't be a valid time. The meta section has its own copy of the time, use that one instead.

nsrinfo calls metaOffset the header size and NSRViewer calls it the header length, which is the same number seen from the other side: the meta section begins where the header ends.

Version 2

Replays of 2012 builds use version 2. The description is only 24 bytes, it has no section offsets and no index section follows. The meta section comes right after the description and the packets come right after the meta section.

Field Type Notes
magic char[4] NSRD
version uint32 2
protocolVersion uint32 See Protocol version
unknown uint32
unknown uint64 Not a Unix time, the time of a version 2 replay comes from the meta section

NSRI - Index

Field Type Notes
magic char[4] NSRI
version uint32 5
unknown uint64 Always 0
count uint32 Number of offsets that follow
indexOffset uint32 Offset of the first entry of offsets, i.e. the next byte
offsets uint32[count] Absolute file offsets of packets, used to seek in a replay. One entry every indexInterval milliseconds

Meta

Everything the client shows about a recording before you play it, plus the position the recording was made from. Strings are null terminated and not length prefixed.

Field Type Notes
version uint32 4, older replays have 3
zoneId uint32 See Zones
description cstring The recording description, e.g. (no recording description)
localDateString cstring Local time of the recorder as written by C's ctime, e.g. Fri Mar 20 17:43:28 2015 plus a newline
position float[3] Where in the zone the recording was made
rotation float[4] Quaternion
characterGuid uint64 See Firefall Guid System
characterName cstring
unknown byte[2]
zoneInstanceGuid uint64 The instance of the zone the recording was made in
clockValue uint64 Only in version 4. A clock value the playback restores
firefallVersionString cstring e.g. Firefall (v1.5.1962)
timestamp uint64 Unix epoch in microseconds, UTC
month uint32 Real world month
day uint32 Real world day
realYear uint32 Real world year
fictionalYear uint32 In-fiction year, the game is set roughly 200 years ahead
fictionalTime float Clock time as a float, e.g. 22.38 for 22:38
fictionalDateString cstring e.g. Friday, March 20.697 Zulu 2238
padding byte[] Zero padding so the string, its terminator and the padding occupy exactly 128 bytes
unknown byte[18]
ingameTime double Per the 010 template
unknown byte[5]

The last 31 bytes are read as one opaque block by FauFau. The split into 18 / double / 5 comes from the 010 template and has not been cross checked.

Keep in mind that every string can be empty, and the last string of a cut off file can be missing its terminator.

Packet

Packets run back to back from dataOffset to the end of the file. In a cut off replay the last packet can claim more bytes than are left. Drop it and keep the packets before it.

Field Type Notes
timestamp uint32 Relative recording time
length uint16 Length of data
messageId uint16 GSS message id, see Views, Updates and Keyframes
data byte[length] The message body

NSRViewer reads timestamp as two uint16 "keyframe order" values instead. Both readings consume the same four bytes.

data starts with the 8 byte entity id the message applies to. The low byte of that id is the entity type, which for networked entities is the controller or view type. That is what lets a tool sort a replay by view without decoding the message bodies:

Id View Id View
0 Generic 26 Vehicle
1 Character 27 Vehicle_BaseController
2 Character_BaseController 28 Vehicle_CombatController
3 Character_NPCController 29 Vehicle_ObserverView
4 Character_MissionAndMarkerController 30 Vehicle_CombatView
5 Character_CombatController 31 Vehicle_MovementView
6 Character_LocalEffectsController 32 Anchor
7 Character_SpectatorController 33 Anchor_AIObserverView
8 Character_ObserverView 34 Deployable
9 Character_EquipmentView 35 Deployable_ObserverView
10 Character_AIObserverView 36 Deployable_NPCObserverView
11 Character_CombatView 37 Deployable_HardpointView
12 Character_MovementView 38 Turret
13 Character_TinyObjectView 39 Turret_BaseController
14 Character_DynamicProjectileView 40 Turret_ObserverView
15 Melding 45 Outpost_ObserverView
16 Melding_ObserverView 48 ResourceNode_ObserverView
17 MeldingBubble 51 CarryableObject_ObserverView
18 MeldingBubble_ObserverView 53 LootStoreExtension_LootObjectView
19 AreaVisualData
20 AreaVisualData_ObserverView
21 AreaVisualData_ParticleEffectsView
22 AreaVisualData_MapMarkerView
23 AreaVisualData_TinyObjectView
24 AreaVisualData_LootObjectView
25 AreaVisualData_ForceShieldView

The gaps are ids we have not seen in a replay. The authoritative list for a given build is the view enum dumped from the client, for example the 1962 dump in Sift.

Protocol version

protocolVersion in the description is the GSS protocol version, not the build number. It is the same opaque uint16 value the client and server agree on for the GSS channels, stored here in a uint32. 19551 is the version used by the 1962 client, which is what AeroMessages currently targets. The raw values are not ordered chronologically, so the only way to map one to a build is a lookup table such as the one Aero generates from the Sift protocol dumps.

Writing a replay

FauFau can generate a file from scratch, which is the shortest description of what a minimal valid replay needs:

  1. Write 48 bytes of nothing, the description gets filled in last.
  2. Write the index at offset 48, remembering the offset as indexOffset.
  3. Write the meta section, remembering its offset and length.
  4. Remember the current offset as dataOffset and write the packets.
  5. Seek back to 0 and write the description now that all four values are known.
  6. gzip the whole thing.

FauFau always writes description version 5 and meta version 4, with an index section.