Debug Traces - themeldingwars/Documentation GitHub Wiki

Debug Traces

The client can export several of its debug overlays to disk and read them back later. Each overlay has an _export and an _import console command, and each writes its own little binary format:

Extension Overlay Commands
.devt Event profiler debugevent_export / debugevent_import
.dlag Lag profiler debuglag_export / debuglag_import
.dm Movement trace debugmovement_export / debugmovement_import, also debugmovement_replay
.dw Weapon trace debugweapon_export / debugweapon_import

These are the only formats in the game that let you look at what the client thought was happening rather than what was on the wire, which makes them worth having alongside a capture or an nsr. The data is also visible over the network: see DebugEventSample and the DebugLag samples and TookDebugWeaponHit for the message side of the same systems.

010 Editor Templates

Chunked arrays

All four formats use the same idiom for a variable length array, and it trips people up, so it is worth stating once. An array is written as repeated blocks of:

ubyte count
element[count]

and the array continues for as long as count is 0xFF. A count of 0 also continues the loop, so the terminator is a count that is neither 0xFF nor part of a full block. In pseudo code:

do {
    count = readByte()
    if (count > 0) readElements(count)
} while (count == 0xFF)

The total element count is usually also written somewhere ahead of the array, so it can be used as a sanity check.

.devt - Event profiler

Records what the client was doing per frame, colour coded in the overlay.

Field Type Notes
time uint32
unknown uint32
graphDuration uint32 Matches debugevent.graphDuration
prefix uint16
blob byte[] ((prefix >> 6) + 1) << 3 bytes, probably a visibility bitfield
count uint32 Number of groups, at most 4
data Group[]

Each group is a uint32 total followed by a chunked array of:

Field Type Notes
time1 uint32
time2 uint32
type uint8 See below

The four groups line up with the four debugevent.* filter commands: frames, updates, weapon inputs and weapons.

Id Type Overlay colour
0 Frame
1 Update
2 WeaponInput_FireBurst Green
3 WeaponInput_FireEnd Red
4 WeaponInput_Reload White
5 WeaponInput_Unk5 Cyan, maybe reload end
6 WeaponInput_UseScope Purple, may be swapped with 7
7 WeaponInput_SelectFireMode Purple, may be swapped with 6
8 WeaponInput_SelectWeapon Black
9 Weapon_Burst DarkOrange
10 Weapon_FireWeaponProjectile Orange

.dlag - Lag profiler

Field Type Notes
time uint32
measurements See below
graphState Chunked array, absent on a plain debuglag 1
tracerouteHost cstring Set when debuglag.traceroute is enabled
data4 Runs to the end of the file, purpose unclear

measurements is a uint32 count followed by that many measurement records, in measurement index order starting at 1. Each record is a uint32 total followed by a chunked array of float value, uint32 time, uint32 unknown.

A graphState entry is a uint16 measurement count, ((count >> 6) + 1) << 3 bytes of visibility bitfield, a float[4] offset, a float[4] scale, and two bytes.

Measurement indices

There are 72 of them in 1962 and 71 in 1946, grouped into the pages the overlay pages through.

Page Ids Measurements
0 1-8 Client Frame, Server Frame, Server Update Rate, Local Gateway, Local Gateway dropped, The Internets, The Internets dropped, Socket RTT
1 9-16 Packet Downloss, Packet Upploss, Client Application Ping, Server Application Ping, Server Player Input Delay, Server PInput Delay Sample, Gameplay Buffering, Client Lateness
2 17-24 Server Lateness, Probed MTU, Game Messages Sent, Game Messages Received, Game Bytes Sent, Game Bytes Received, Game Compression Ratio, Game Packets/Second
3 25-32 Game Bytes/Second, Game Shaped Bytes, Client Source, Matrix Upstream, GSS Received, GSS Object Processed, Matrix Downstream, Client Final
4 33-40 Simulated Objects, Update Starved Maximum, Update Starved Average, Late Update Maximum, Late Update Average, Update Requests, Rejected Update Requests, Reordered Objects
5 41-48 Priority Inversions, Dropped Priority Updates, Simulation Duration, Total Objects, Matrix Frame, Matrix Update Duration, Keyframes Requested, Keyframes Queued
6 49-56 Keyframe Delay, Matrix Bytes/Second, Batching Delay, Total Budget, Keyframes Budget, Reserved Budget, Unreliable Budget, Raia Frame
7 57-64 Raia Update Duration, Updated Objects, Total Objects, VT Used Bandwidth, Movement Confirmed Period, Movement Validations, Movement Simulation Time, Movement Correction Length
8 65-72 Remote Extrapolation Time, Remote Extrapolation Length, Clock Smoothing Delta, Clock Gameplay Delta, Last Send, Last Received, Rate, MPH
9 73 RPM

.dm - Movement trace

The whole file is one chunked array of trace records.

Field Type Notes
haveData uint8 0 means an empty slot, nothing follows
time uint32
bitfield uint32 Which of the 20 trace groups were recorded
groups One per set bit, low bit first, up to 20

The bits map onto the debugmovement cvars that select what gets recorded, so a trace only contains the groups that were enabled at the time.

Each group is:

Field Type Notes
shortTime uint16
position float[3]
orientation float[4] Quaternion
direction float[3]
state uint16 Looks like the movement state
extended Optional, see below

The extended block starts with a uint8. When it is 1 the following is present:

Field Type Notes
unknown uint16
velocity float[3]
unknown uint8[2]
jetpackEnergy uint16
groundTimePositiveAirTimeNegative int16 Positive on the ground, negative in the air
timeSinceLastJump int16
haveCollisionData uint8
collision data When set: a uint8 count of origin/direction float[3] arrows, then a uint8 count of floats
unknown float[3]
unknown float[3]
unknown uint8

.dw - Weapon trace

Field Type Notes
header byte[4] Changes with the debugweapon 0-10 mode
maxSize uint8 Matches debugweapon.maxSize
traces Until 4 bytes from the end of the file
footer uint32 Last trace counter, wraps when the max trace count is exceeded, 0 if there are no traces

Each trace starts with a uint8. When it is 1:

Field Type Notes
unknown uint8
traceId uint32 PRNG generated, uniquely identifies one bullet. The rendering code uses it to tie records together
ammoTypeId uint32
movement data 0x88 bytes, only when ammoTypeId is non-zero. Contains the ray start and end, the weapon range, the damage per round, the weapon template id and the slot index
clientDataCount uint8
client hits clientDataCount hit records
serverDataCount uint8
server hits serverDataCount hit records

Having the client's and the server's version of the same trace side by side in one file is what makes this format useful: it is a recording of exactly where the two disagreed.

Hit record

The same shape as the TookDebugWeaponHit message:

Field Type Notes
type uint8 See below
traceId uint32
time uint32
unknown uint16
position float[3]
direction float[3]
physicsMaterial uint32 For Impact, Ragdoll_Hit and Validated this matches a dbphysicsmaterials::PhysicsMaterial id. For Spawn it is int32 max, for Posefile_Hit it is 0
optional entity block uint8 + data A guid, a float[3], a quaternion and two bytes
optional ragdoll block uint8 + data A guid and a uint8 counted array of float[4][4] transforms for the pieces of the ragdoll. Only rendered on import if the matching entity exists
Id Trace type
0 ERROR
1 Spawn
2 Bounce
3 Posefile_Hit
4 Ragdoll_Hit
5 Ragdoll_Miss
6 Impact
7 Shot
8 Detonate
9 Timeout
10 Validated