Accessing Borderlands 3 Data - BLCM/BLCMods GitHub Wiki
Among the challenges of modding Borderlands 3 / Wonderlands is getting data out of the game. It's unfortunately not as straightforward as BL2 or TPS, where we had some relatively-easily-available tools to generate data dumps that are used in apps like OpenBLCMM and FT/BLCMM Explorer. This page will go over the various methods of retrieving data from BL3/WL that we've got available as of December 2020.
- Game Compatibility
- Extracting Raw Datafiles
- Basic Raw Datafile Info
- Advanced Raw Datafile Info (JohnWickParse)
- Alternate Object Access: FModel
- Ubergraph Bytecode Info (UAssetGUI)
- Console Access in Borderlands 3 Itself
- Getting Data from the Console
Game Compatibility
The methods described here basically all work equally well in both Borderlands 3 and Tiny Tina's Wonderlands. There's occasionally some slight differences in download locations, so look out for that, but the general advice is the same for both games. They even both share the same encryption keys, etc!
Extracting Raw Datafiles
Shortcut
Community member Grimm has uploaded an archive of pre-serialized BL3 data to Nexus Mods. As new patches get released (and as JohnWickParse functionality gets extended) these pre-serialized objects might lag behind the capabilities, and it's worth noting that there are some objects which we still can't serialize at all, so you may want to go through the full extraction anyway (since the un-serialized datafiles can still give you some info about the game data, even if we can't look into them too far). Still, it's a useful resource and should save you some time and hard drive space, if you don't care about extracting the data yourself.
Data Extraction
The first, and most straightforward thing you can do is unpack all the
various *.pak
files which live in OakGame/Content/Paks
(and in
AdditionalContent/*/Paks
for DLCs). These pak files are encrypted, so
you'll need to find the encryption key online (it won't be listed on this
page, but a search for borderlands 3 pakfile aes
should bring it up).
Unpacking the data can be done with the UnrealPak.exe
utility which comes
with Unreal Engine, though you need to make sure to use a correct version.
Version 4.23 should do the trick. (Earlier versions might not like the
"version 6" pakfiles which BL3 uses.)
The easiest way to extract these, by far, is using a script named
unpack_bl3.py
in Apocalyptech's "bl3data" github repo.
(For Wonderlands, the script is unpack_wl.py
in Apocalyptech's "wldata" github repo.)
The script will attempt to autodetect your BL3 install directory (though you
can specify it by hand on the commandline, or by editing a variable in
the script), and will then loop through all relevant Pakfiles, moving
them around so that their filesystem location perfectly matches their in-game
object location. The script will end up asking you for an encryption key for
the pakfiles. You should be able to find that online with that search for
borderlands 3 pakfile aes key
.
Note: Running unpack_bl3.py
from inside VSCode can lead to errors, as VSCode
can end up holding on to locks of the files that the script is working on. If
you're running it in VSCode and seeing errors, you might want to try doing it
outside that app, instead.
If you want to know which pakfiles contain which objects, you can take a look at Apocalyptech's Pakfile Contents lookup page, or this Google Drive link. (Wonderlands versions: Pakfile Lookup, Google Drive.) Keep in mind that pakfile patches might overwrite objects found in earlier pakfiles -- see the "historical info" section below for some more info on that. Note that searching the files in Google Drive might not work great; you'll be best off downloading the files and searching on your local copies.
Extraction Details (historical info)
Note: If you're using the unpack_bl3.py
/unpack_wl.py
script above, or using Grimm's
pre-serialized data, you don't have to worry about any of these details. I'm
leaving this information here for historical purposes, though. If you wanted
to extract all this data by hand, this is what you'll have to do:
First, You'll first need to create a crypto.json
file (named however you like,
actually) with contents that look like the following:
{
"$types": {
"UnrealBuildTool.EncryptionAndSigning+CryptoSettings, UnrealBuildTool, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null": "1",
"UnrealBuildTool.EncryptionAndSigning+EncryptionKey, UnrealBuildTool, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null": "2"
},
"$type": "1",
"EncryptionKey": {
"$type": "2",
"Name": null,
"Guid": null,
"Key": "<base64-encoded version of the encryption key found online>"
},
"SigningKey": null,
"bEnablePakSigning": false,
"bEnablePakIndexEncryption": true,
"bEnablePakIniEncryption": true,
"bEnablePakUAssetEncryption": false,
"bEnablePakFullAssetEncryption": false,
"bDataCryptoRequired": true,
"SecondaryEncryptionKeys": []
}
The Key
parameter in there is where to put in the encryption key for
the pakfiles -- note that it should be a base64-encoded version of the
data representation of the encryption key (not just the text hex
strings that you'll find on the internet). You can use an online
converter like this one to
do that conversion. You'll have the right string if it starts with
EV7
and ends with 6c=
.
Once that's in place, you'll unpack using the commandline:
UnrealPak.exe filename.pak -extract extractdir -cryptokeys=crypto.json
The pak file will get extracted into the extractdir
directory. The order
in which paks should be extracted is important in many cases. The base
game Pak files are all named like pakchunk4-WindowsNoEditor.pak
. A
patch later added pakchunk4-WindowsNoEditor_0_P.pak
, which should be
extracted after the original file. Then there was a
pakchunk4-WindowsNoEditor_1_P.pak
in a patch after that, and so on. So
extract the ones without _P
first, then extract all the _0_P
files
next, then _1_P
, and so on.
UE4 organizes its objects into directories, and the extracted data will be
extracted in a similar way, though the extracted directories won't exactly
match the object paths. The unpack_bl3.py
script takes care of shuffling
all this around, so you'll have to look in there for details. Without
moving the files around to match their in-game object locations, you'll have
a much harder time finding the objects you're looking for, since the
references from the game won't match your filesystem.
Basic Raw Datafile Info
Without even having to parse the datafiles, you can get a reasonable amount
of information just by using the utility
strings
to look for strings inside the .uasset
files specifically. The strings
found inside the .uasset
files will include all the object names that
that object references, so you can infer quite a bit of information about
how objects relate to each other just by that method. That's actually the
basis for the interactive Borderlands 3 Object
Refs/Wonderlands Object Refs pages,
which are very useful resources for browsing around the data.
Other useful information that can be found with strings
includes the
attribute names which can be found in the object. You can also find ingame
text by running strings
on the .uexp
files, such as item/weapon names,
card text, and the like.
Advanced Raw Datafile Info (JohnWickParse)
You can get even more info out of the datafiles by using a utility called JohnWickParse. It's an asset parser originally written for Fortnite, by SirWaddles, but it worked just fine on many UE4-based games. Apocalyptech's been maintaining a BL3-modding-focused fork of JWP, and we recommend using that instead of the original, because it includes various improvements which help with BL3 modding quite a bit.
- The latest BL3-focused JWP fork can be found here: https://github.com/apocalyptech/JohnWickParse/releases/latest
- SirWaddles' original JohnWickParse can be found here: https://github.com/SirWaddles/JohnWickParse/releases/latest
To actually run JWP to generate some human-readable data, run:
john-wick-parse serialize <object_name>
You'll need to have the .uasset
and .uexp
file in the directory where
you run it -- the object_name
should be the base filename without extension.
When you run the serialization process, it should hopefully output a new
file named object_name.json
in the same directory. For some objects it
won't work at all, unfortuantely -- for instance, you'll never be able to
serialize the BPChar
character objects, or mission objects. But for many
of them, you'll have a JSON file which contains a lot of useful info in it.
Many objects will only serialize partially, so some parts will be filled in
and others blank. So you'll end up needing to compliment these
serializations with the strings
output as well. You'll want to edit
the JSON using an editor which prettifies the output, or process it using
an app like jq or site like
jsonformatter.org, to make it easier to read.
Alternate Object Access: FModel
Like JohnWickParse, FModel is an app which can serialize Unreal objects usefully, though it primarily does so inside an interactive app, rather than being something you can automate from the commandline. FModel might be able to serialize some objects that JWP can't, as well, so it might be worth a look. Note that FModel operates directly on the pak files themselves, and you specify the encryption key right in the app's UI, so you don't have to extract the datafiles prior to using FModel. JohnWickParse is probably a bit more convenient in terms of collecting multiple object serializations, but this would be a nice point-and-click interface to the data, as well.
One other thing to note with FModel is that there are a couple of BL3 pakfiles which are literally empty of data, and FModel doesn't like those. If you have those pakfiles in FModel's list, it won't serialize anything for you. So, make sure you've omitted these two (as of March 1, 2021):
pakchunk12-WindowsNoEditor_8_P.pak
pakchunk83-WindowsNoEditor_4_P.pak
The other thing to watch out for with FModel is that even though it operates on pakfiles directly, and keeps the list of possible paks all in one unified list, it doesn't understand the pak-patching conventions used by UE4, so if an object is present in an early pakfile, and then overwritten by a later patch pakfile, you'll have to select which pakfile to look into. So keep that in mind, to make sure you're looking at the most recent version of the objects in question!
Ubergraph Bytecode Info (UAssetGUI)
One area of data which has eluded us for quite awhile, until September/October 2022 or so, was the Ubergraph Bytecode which glues a lot of the fancier object logic together in BL3/WL. This is essentially the BL3/WL version of BL2/TPS's BehaviorProviderDefinitions and Kismets. It turns out that there's at least one super-handy application which lets us get to this data, though: UAssetGUI/UAssetAPI.
UAssetGUI lets you click through all the various exports, getting very low-level and in-depth information about them. Exports which have Ubergraph Bytecode attached to them will also have an element in the tree which provides full JSON serializations of the bytecode.
The "vanilla" UAssetAPI code isn't immediately useful for BL3/WL modders, because we need to know the indexes/offsets to use in our type 7 hotfixes, but Apocalyptech's fork of UAssetAPI contains that functionality.
To use Apocalyptech's fork, download an updated UAssetAPI.dll
from its
github, and copy it over the "vanilla" one in your UAssetGUI installation.
Alternatively, the fork also includes a CLI utility to do serializations without
a GUI (very similarly to how JWP works), and another to generate graphs of the
bytecode structure. See the README of Apocalyptech's fork for more details!
Console Access in Borderlands 3 Itself
Finally, getting data out of the BL3 ingame console is possible, though it's not nearly as convenient as in BL2/TPS. First off, there isn't a simple hex-edit to enable the console; instead, you've got to inject a DLL into the running BL3 process. Most Windows users will find it most convenient to do DLL injection via Cheat Engine, though any other DLL injector should also presumably do the trick, such as Xenos, Extreme Injector, IGCS Injector, or the one which comes bundled with the Universal Unreal Engine 4 Console Unlocker (UUU).
For Linux users who run BL3 via Wine/Proton, note that most injection apps
don't seem to work, but I've had great luck with IGCS Injector.
UUU used to have an older package available with the filename
UniversalUE4Unlocker_v1016.zip
, which included both a bundled IGCS Injector,
and a simple DLL for injection which just does the console unlock and provides
some extra camera hotkeys and the like. That setup works great in Wine,
but unfortunately since then, UUU's taken down that old link. The current
UUU-provided custom injector doesn't seem to work so well in Wine, alas, and
the current DLL providecd by UUU seems to require interaction with that
injector to work properly. The commit which removed the old links
can be viewed on Github, though, which will point you to this mega.nz link,
which as of February 12, 2020, is still online. So Linux users should
check there for an injector/DLL which actually works.
Once you have an injection app, you need a DLL to inject which will unlock the console. At the moment (Feb 11, 2020), I'm only aware of one which actually works, which is the Universal Unreal Engine 4 Console Unlocker (UUU). As noted above, that comes with its own DLL injector as well, which is convenient. There was also another console-unlocker DLL available at fearlessrevolution.com, but that DLL stopped working with one of BL3's more recent patches, so as of writing, you're better off with UUU.
To inject the DLL, regardless of injector, just start up BL3, wait until you're at the main menu, then launch the injector and use it on the BL3 process. Give it a few seconds, and you should be able to hit tilde/backtick a few times to cycle through the available console modes ingame. In my experience it works just fine if you inject while ingame as well, so you probably don't have to be on the main menu specifically.
Getting Data from the Console
Once you have console access, there's unfortunately a very limited set of
commands you can use which are useful for getting data. Namely, the
mainstay of object dumping from BL2/TPS -- obj dump
-- is not available
on the BL3 console. Instead, the only command usable for data retrieval is
getall
. In BL2/TPS that was primarily just used to get lists of objects,
which you'd then obj dump
, but in BL3 it'll be your window to in-game
data.
The UE4 getall
command can be used just to list objects, like this
command to list all item pools currently loaded in memory (this is one kind
of object which JohnWickParse can serialize easily, but checking values
in-game is often helpful even then):
getall itempooldata
You can get specific and provide the exact attribute you're looking for, like so:
getall itempooldata balanceditems
When you're looking for one specific object's attribute, though, you'll
want to narrow the results down a bit, which isn't always possible,
depending on the kind of object, but for ItemPoolData objects, it is. You
can specify an extra name=foo
parameter at the end, like so:
getall itempooldata balanceditems name=itempool_artifacts
That command would show you the contents of the itempool which drops
Artifacts in-game. The name to use, to narrow down the getall
output to
something reasonable, will be found after the last slash in the object
names, so you can generally get them right from the object list itself.
(Though again, in some cases you'll find that all the objects that show up
share a common name, so you might be stuck anyway.)
Note that when you use the name=foo
construct at the end, you need to
specify an attribute name as well, otherwise it won't work. If you don't
actually care about the attributes and you just want the object list, you
can put in any bit of text for the attribute and you'll still get the
object list.
One final wrinkle that you'll have to contend with is that the BL3 console
does not wrap text to the next line, if there's enough output to reach
the righthand side of the screen. Anything beyond that point will just be
cut off and not visible ingame. Fortunately, you can get the missing
text, by using a memory viewer like Cheat Engine. You'll have to attach to
the running Borderlands 3 process and then search memory for some bit of
the getall
output that you can see, and then look at the rest of the
line in the CE memory editor. Note that the console output text looks like
it's encoded internally in memory as UTF-16 strings, so you may have to
change some dropdowns on whatever app you're using to search in memory.