DBAL Decompilation and dbal.py - krcenov/RNS510-Map-Viewer GitHub Wiki
A still later session installed a real decompiler (Ghidra, NSA's free/
open-source tool β OpenJDK 21 + Ghidra 12.1.4, both installed locally)
and ran it headless against dbal/V010_015/DBAL.OUT, producing a
complete, single-file decompilation: 1,915 functions, 0 failures,
~2.6MB of C-like pseudocode. This is a different, smaller-scale effort
than the existing Hex-Rays/capstone work on the main firmware image
(FHDD6.FLI, see Firmware Reversing) β DBAL.OUT
is a much smaller, unlinked object file with no disassembly wall, so a
full automated decompile was straightforward once a decompiler existed
to run.
Not committed to the repo (the Ghidra install + decompiled output are
570MB+; tools/ is gitignored). Reproducible: tools/ghidra_scripts/ DecompileAll.java (a GhidraScript using DecompInterface) run via
analyzeHeadless with -processor PowerPC:BE:32:default against any
dbal/V0NN_0MM/DBAL.OUT.
Before trusting the decompiler's output, db_get_minor_road_list_offset's
decompiled C was compared against this project's own earlier, independent,
by-hand capstone disassembly of the same function (see
eeu.iof's own page) β identical logic,
confirming the decompiler is trustworthy for this binary.
CORRECTION (a later session): the generic field-reader is NOT external β it's a decompiler artifact, resolvable via ELF relocations
The "DEFINITIVELY confirmed external... proven, permanent limit"
claim below (this section's original text) was wrong. It was reached
by reading Ghidra's pseudocode at face value: both eeu.iof's
unpack_intersection_offset_rec and eeu.rd's
read_current_record_from_database call every field through an
indirect pointer β (**(code **)(*(int *)(obj + 0xc) + 0x164))(...) β
which Ghidra can't statically resolve to a name (that's just what a C++
virtual/indirect call always looks like once decompiled), and that
"can't resolve" was misread as "code doesn't exist." It does.
Proven directly, this session: DBAL.OUT's .symtab has 621
DBF*-prefixed symbols β a full, real, compiled generic database
framework (DBFView, DBFViewmodel, DBFDataItem, DBFInteger,
DBFIntegerSpecial, DBFBitComponent, DBFOffset, DBFString,
DBFCounter, DBFList, DBFIndex, and more) β almost all of it
shndx=1 (i.e. real code sitting in .text, not SHN_UNDEF). The
indirect calls above are virtual dispatch through one of these classes'
own vtables (_vt$<ClassName> symbols, also real and present), and
.rela.text's relocation entries populate each vtable slot with the
REAL target function β reading them resolves any given call exactly:
unpack_intersection_offset_rec's 3 field reads, resolved via
_vt$17DBFIntegerSpecial's own relocations (vtable value=642168):
offset+0x154 -> getValueUnsignedLong__C17DBFIntegerSpecial (reloc @ +340)
offset+0x15c -> (a different class's own vtable -- NOT the same
offset-to-method mapping across classes, see caveat)
offset+0x164 -> getValueLong__C17DBFIntegerSpecial (reloc @ +356)
The actual generic integer codec is real, fully decompiled code:
setValPtr__17DBFIntegerSpecialPcT1b (@0x86a3c, 1008 bytes) is a
9-case switch implementing every integer encoding this disc uses β
fixed 1/2/3/4-byte (with 24-bit sign-extension for the 3-byte case),
and 2 distinct variable-length (base-128 "varint") schemes, one with an
extra sign bit packed into the first byte, in both byte orders. The
decoded value is cached at a fixed object offset; the various
getValueLong/getValueUnsignedLong/getValueShort/.../getValueChar
accessors just read back different widths/offsets of that same cached
value (confirmed by reading their own 1-line bodies β NOT all of them
return the full value: getValueUnsignedShort reads only its low 16
bits, for instance, so which accessor a given field actually calls
through DOES matter). setValPtr__15DBFBitComponentPcT1b/
getValueLong__C15DBFBitComponent confirm DBFBitComponent is a thin
wrapper around one of these integer fields, doing exactly
(underlying_value >> bit_offset) & ((1 << bit_width) - 1) β
independently matching eeu.mod's own
bit_width/bit_offset descriptor fields, and exactly the model this
project already used (empirically) for kd_info's bits.
Honest caveat β this doesn't mean every field is now trivially
decodable: a given call site's indirect pointer resolves through
WHICHEVER concrete DBF* class getDataItem() returned for that field
name at runtime (schema-driven, not fixed at compile time) β so
"vtable offset 0x164" means a different method on DBFInteger than on
DBFIntegerSpecial than on DBFBitComponent, and each call site needs
its own class identified before its relocations can be read. What's
now established beyond doubt: the code for ALL of it is real and
present in DBAL.OUT, nothing here is an external/missing-code wall,
and the method (vtable symbol β .rela.text/.rela.data relocations
in its byte range β real function name) is proven and repeatable.
Chased the "not yet done" item above β concretely resolved 2 real call
sites end to end, picking the right _vt$<ClassName> by matching each
call's own vtable-slot values against known candidates, not by
assumption:
-
eeu.iof'soffset_data(unpack_intersection_offset_rec):offset_o_527resolves (vtable offset0x154) togetValueUnsignedLong__C17DBFIntegerSpecialβ a plain cached 32-bit value.type_o_528/count_o_529BOTH use vtable offset0x164, but that offset means something different on each class's own vtable (DBFIntegerSpecial's own layout hasgetValueLongthere;DBFBitComponent's hasgetValueShort, confirmed by dumping its_vt$15DBFBitComponentrelocations directly) β these 2 areDBFBitComponent, matchingeeu.mod's owncountAndTypeschema EXACTLY: a single 2-byte field split intocount(15 bits, offset 0)-
type(1 bit, offset 15) β i.e.DBFBitComponent::getValueShortcomputing(countAndType >> bit_offset) & ((1<<bit_width)-1)for each, cross-confirming this project's own earlier, independently- reachedtype/countsemantics (README/eeu.iof's own page) from a 3rd, completely different angle.
-
-
eeu.rd'sroadInfo:eeu.modnames exactly 6 sub-fields βhighway(bit 0),preferred(1),explicate(2),ramp(3),name type(bits 4-6, 3 bits wide),interchange(7) β an EXACT, bit-for- bit match to this project's own already-shipped, independently- deriveddecode_roadinfo_byte()layout (research/dbal.py). SameDBFBitComponentmechanism.
The practical upshot: the "map every field to its DBF class" task
isn't actually necessary for decoding values. Scanned every one of
eeu.mod's ~1,600 field descriptors across all 36 known tables for
anything that DOESN'T fit the 2 patterns research/mod_reader.py
already extracts (byte_width N>0 = plain fixed-width int; byte_width
0 + sane bit_width/bit_offset = a DBFBitComponent-style bit
field) β zero anomalies found. Every real field on this disc is one of
those 2 shapes; none exercise the generic codec's variable-length
("varint") cases found in setValPtr__17DBFIntegerSpecialPcT1b above
β real, present, and now fully understood code, just apparently unused
by this particular disc's own schema. Concretely: eeu.mod's existing,
already-used byte_width/bit_width/bit_offset descriptors were
ALREADY a complete, correct recipe for decoding any field on this disc
β this session's DBAL.OUT work explains, at the real code level, WHY
that's true (down to the exact shift/mask/sign-extension performed),
rather than unlocking previously-inaccessible data. The one open
question from the original "not yet done" (an explicit fieldβDBF*-
class table) is answered: not needed, and dropped as a goal.
Given the above, dbal.py doesn't (and can't) port "the extraction" β
instead it ports the real logic DBAL.OUT applies on TOP of already-
extracted fields, confirmed fully self-contained (no external calls):
-
FeaCoordReceiverInternalβeeuz.fea's real point-chain decoder (get_coordinates()) and its real Douglas- Peucker-style line-simplification pass (thinout()/_thinout_recursive()), ported instruction-by-instruction. The core delta-accumulation loop was validated against a synthetic chain ((5,3),(2,-1),(4,0)β correctly cumulative-sums to(5,3),(7,2), (11,2), with the real(0,0)-after-first-point terminator rule also confirmed). The thinning path's real per-level distance-table constants were NOT successfully extracted from the binary (a quick attempt landed on an address that reads as all-zero, almost certainly a wrong file-offset mapping, not a real value) β documented as unresolved in the code rather than silently guessed. -
apply_rd_record_transform()β the realrdtype & 0x7fffmasking andmaxLong = minLong + deltaLong/maxLat = minLat + deltaLatconversionread_current_record_from_databaseperforms. -
gate_minor_road_list_offset()/gate_minor_road_count()βeeu.iof's real validity-gating decision logic (typemust be exactly0foroffsetto be valid;type==1means the on-diskcountis provisional), already documented oneeu.iof's own page from the earliercapstone-only session, now re-confirmed against the decompiled C. -
decode_roadinfo_byte()β NOT a DBAL.OUT port (that extraction is external, see above) β this project's own already-validated bit layout, centralized here as the one place every caller reads it from.
rns510_core.RoadRecord now exposes max_lon/max_lat, computed via
dbal.apply_rd_record_transform() (previously only delta_lon/
delta_lat were exposed). road_naming.RdCache's highway/
highway_names() now goes through dbal.decode_roadinfo_byte() instead
of inline bit math. Verified end-to-end against the real ISO: identical
results to the pre-refactor values (578,946/8,809,081 highway records,
42,505 distinct names) β confirming the refactor changed provenance, not
behavior.
research/dbal.py β see its own module docstring for the full honesty
accounting of what's a faithful port vs. this project's own independent
crack. tools/ghidra_scripts/DecompileAll.java (not committed) for
reproducing the decompilation itself.