Accessing Borderlands 3 Data - BLCM/BLCMods GitHub Wiki

Among the challenges of modding Borderlands 3 / Wonderlands is getting data out of the game. It's unfortunately not as straightforward as BL2 or TPS, where we had some relatively-easily-available tools to generate data dumps that are used in apps like OpenBLCMM and FT/BLCMM Explorer. This page will go over the various methods of retrieving data from BL3/WL that we've got available as of December 2020.

Game Compatibility

The methods described here basically all work equally well in both Borderlands 3 and Tiny Tina's Wonderlands. There's occasionally some slight differences in download locations, so look out for that, but the general advice is the same for both games. They even both share the same encryption keys, etc!

Extracting Raw Datafiles

Shortcut

Community member Grimm has uploaded an archive of pre-serialized BL3 data to Nexus Mods. As new patches get released (and as JohnWickParse functionality gets extended) these pre-serialized objects might lag behind the capabilities, and it's worth noting that there are some objects which we still can't serialize at all, so you may want to go through the full extraction anyway (since the un-serialized datafiles can still give you some info about the game data, even if we can't look into them too far). Still, it's a useful resource and should save you some time and hard drive space, if you don't care about extracting the data yourself.

Data Extraction

The first, and most straightforward thing you can do is unpack all the various *.pak files which live in OakGame/Content/Paks (and in AdditionalContent/*/Paks for DLCs). These pak files are encrypted, so you'll need to find the encryption key online (it won't be listed on this page, but a search for borderlands 3 pakfile aes should bring it up).

Unpacking the data can be done with the UnrealPak.exe utility which comes with Unreal Engine, though you need to make sure to use a correct version. Version 4.23 should do the trick. (Earlier versions might not like the "version 6" pakfiles which BL3 uses.)

The easiest way to extract these, by far, is using a script named unpack_bl3.py in Apocalyptech's "bl3data" github repo. (For Wonderlands, the script is unpack_wl.py in Apocalyptech's "wldata" github repo.) The script will attempt to autodetect your BL3 install directory (though you can specify it by hand on the commandline, or by editing a variable in the script), and will then loop through all relevant Pakfiles, moving them around so that their filesystem location perfectly matches their in-game object location. The script will end up asking you for an encryption key for the pakfiles. You should be able to find that online with that search for borderlands 3 pakfile aes key.

Note: Running unpack_bl3.py from inside VSCode can lead to errors, as VSCode can end up holding on to locks of the files that the script is working on. If you're running it in VSCode and seeing errors, you might want to try doing it outside that app, instead.

If you want to know which pakfiles contain which objects, you can take a look at Apocalyptech's Pakfile Contents lookup page, or this Google Drive link. (Wonderlands versions: Pakfile Lookup, Google Drive.) Keep in mind that pakfile patches might overwrite objects found in earlier pakfiles -- see the "historical info" section below for some more info on that. Note that searching the files in Google Drive might not work great; you'll be best off downloading the files and searching on your local copies.

Extraction Details (historical info)

Note: If you're using the unpack_bl3.py/unpack_wl.py script above, or using Grimm's pre-serialized data, you don't have to worry about any of these details. I'm leaving this information here for historical purposes, though. If you wanted to extract all this data by hand, this is what you'll have to do:

First, You'll first need to create a crypto.json file (named however you like, actually) with contents that look like the following:

{
    "$types": {
    "UnrealBuildTool.EncryptionAndSigning+CryptoSettings, UnrealBuildTool, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null": "1",
    "UnrealBuildTool.EncryptionAndSigning+EncryptionKey, UnrealBuildTool, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null": "2"
    },
    "$type": "1",
    "EncryptionKey": {
    "$type": "2",
    "Name": null,
    "Guid": null,
    "Key": "<base64-encoded version of the encryption key found online>"
    },
    "SigningKey": null,
    "bEnablePakSigning": false,
    "bEnablePakIndexEncryption": true,
    "bEnablePakIniEncryption": true,
    "bEnablePakUAssetEncryption": false,
    "bEnablePakFullAssetEncryption": false,
    "bDataCryptoRequired": true,
    "SecondaryEncryptionKeys": []
}

The Key parameter in there is where to put in the encryption key for the pakfiles -- note that it should be a base64-encoded version of the data representation of the encryption key (not just the text hex strings that you'll find on the internet). You can use an online converter like this one to do that conversion. You'll have the right string if it starts with EV7 and ends with 6c=.

Once that's in place, you'll unpack using the commandline:

UnrealPak.exe filename.pak -extract extractdir -cryptokeys=crypto.json

The pak file will get extracted into the extractdir directory. The order in which paks should be extracted is important in many cases. The base game Pak files are all named like pakchunk4-WindowsNoEditor.pak. A patch later added pakchunk4-WindowsNoEditor_0_P.pak, which should be extracted after the original file. Then there was a pakchunk4-WindowsNoEditor_1_P.pak in a patch after that, and so on. So extract the ones without _P first, then extract all the _0_P files next, then _1_P, and so on.

UE4 organizes its objects into directories, and the extracted data will be extracted in a similar way, though the extracted directories won't exactly match the object paths. The unpack_bl3.py script takes care of shuffling all this around, so you'll have to look in there for details. Without moving the files around to match their in-game object locations, you'll have a much harder time finding the objects you're looking for, since the references from the game won't match your filesystem.

Basic Raw Datafile Info

Without even having to parse the datafiles, you can get a reasonable amount of information just by using the utility strings to look for strings inside the .uasset files specifically. The strings found inside the .uasset files will include all the object names that that object references, so you can infer quite a bit of information about how objects relate to each other just by that method. That's actually the basis for the interactive Borderlands 3 Object Refs/Wonderlands Object Refs pages, which are very useful resources for browsing around the data.

Other useful information that can be found with strings includes the attribute names which can be found in the object. You can also find ingame text by running strings on the .uexp files, such as item/weapon names, card text, and the like.

Advanced Raw Datafile Info (JohnWickParse)

You can get even more info out of the datafiles by using a utility called JohnWickParse. It's an asset parser originally written for Fortnite, by SirWaddles, but it worked just fine on many UE4-based games. Apocalyptech's been maintaining a BL3-modding-focused fork of JWP, and we recommend using that instead of the original, because it includes various improvements which help with BL3 modding quite a bit.

To actually run JWP to generate some human-readable data, run:

john-wick-parse serialize <object_name>

You'll need to have the .uasset and .uexp file in the directory where you run it -- the object_name should be the base filename without extension. When you run the serialization process, it should hopefully output a new file named object_name.json in the same directory. For some objects it won't work at all, unfortuantely -- for instance, you'll never be able to serialize the BPChar character objects, or mission objects. But for many of them, you'll have a JSON file which contains a lot of useful info in it. Many objects will only serialize partially, so some parts will be filled in and others blank. So you'll end up needing to compliment these serializations with the strings output as well. You'll want to edit the JSON using an editor which prettifies the output, or process it using an app like jq or site like jsonformatter.org, to make it easier to read.

Alternate Object Access: FModel

Like JohnWickParse, FModel is an app which can serialize Unreal objects usefully, though it primarily does so inside an interactive app, rather than being something you can automate from the commandline. FModel might be able to serialize some objects that JWP can't, as well, so it might be worth a look. Note that FModel operates directly on the pak files themselves, and you specify the encryption key right in the app's UI, so you don't have to extract the datafiles prior to using FModel. JohnWickParse is probably a bit more convenient in terms of collecting multiple object serializations, but this would be a nice point-and-click interface to the data, as well.

One other thing to note with FModel is that there are a couple of BL3 pakfiles which are literally empty of data, and FModel doesn't like those. If you have those pakfiles in FModel's list, it won't serialize anything for you. So, make sure you've omitted these two (as of March 1, 2021):

  • pakchunk12-WindowsNoEditor_8_P.pak
  • pakchunk83-WindowsNoEditor_4_P.pak

The other thing to watch out for with FModel is that even though it operates on pakfiles directly, and keeps the list of possible paks all in one unified list, it doesn't understand the pak-patching conventions used by UE4, so if an object is present in an early pakfile, and then overwritten by a later patch pakfile, you'll have to select which pakfile to look into. So keep that in mind, to make sure you're looking at the most recent version of the objects in question!

Ubergraph Bytecode Info (UAssetGUI)

One area of data which has eluded us for quite awhile, until September/October 2022 or so, was the Ubergraph Bytecode which glues a lot of the fancier object logic together in BL3/WL. This is essentially the BL3/WL version of BL2/TPS's BehaviorProviderDefinitions and Kismets. It turns out that there's at least one super-handy application which lets us get to this data, though: UAssetGUI/UAssetAPI.

UAssetGUI lets you click through all the various exports, getting very low-level and in-depth information about them. Exports which have Ubergraph Bytecode attached to them will also have an element in the tree which provides full JSON serializations of the bytecode.

The "vanilla" UAssetAPI code isn't immediately useful for BL3/WL modders, because we need to know the indexes/offsets to use in our type 7 hotfixes, but Apocalyptech's fork of UAssetAPI contains that functionality.

To use Apocalyptech's fork, download an updated UAssetAPI.dll from its github, and copy it over the "vanilla" one in your UAssetGUI installation. Alternatively, the fork also includes a CLI utility to do serializations without a GUI (very similarly to how JWP works), and another to generate graphs of the bytecode structure. See the README of Apocalyptech's fork for more details!

Console Access in Borderlands 3 Itself

Finally, getting data out of the BL3 ingame console is possible, though it's not nearly as convenient as in BL2/TPS. First off, there isn't a simple hex-edit to enable the console; instead, you've got to inject a DLL into the running BL3 process. Most Windows users will find it most convenient to do DLL injection via Cheat Engine, though any other DLL injector should also presumably do the trick, such as Xenos, Extreme Injector, IGCS Injector, or the one which comes bundled with the Universal Unreal Engine 4 Console Unlocker (UUU).

For Linux users who run BL3 via Wine/Proton, note that most injection apps don't seem to work, but I've had great luck with IGCS Injector. UUU used to have an older package available with the filename UniversalUE4Unlocker_v1016.zip, which included both a bundled IGCS Injector, and a simple DLL for injection which just does the console unlock and provides some extra camera hotkeys and the like. That setup works great in Wine, but unfortunately since then, UUU's taken down that old link. The current UUU-provided custom injector doesn't seem to work so well in Wine, alas, and the current DLL providecd by UUU seems to require interaction with that injector to work properly. The commit which removed the old links can be viewed on Github, though, which will point you to this mega.nz link, which as of February 12, 2020, is still online. So Linux users should check there for an injector/DLL which actually works.

Once you have an injection app, you need a DLL to inject which will unlock the console. At the moment (Feb 11, 2020), I'm only aware of one which actually works, which is the Universal Unreal Engine 4 Console Unlocker (UUU). As noted above, that comes with its own DLL injector as well, which is convenient. There was also another console-unlocker DLL available at fearlessrevolution.com, but that DLL stopped working with one of BL3's more recent patches, so as of writing, you're better off with UUU.

To inject the DLL, regardless of injector, just start up BL3, wait until you're at the main menu, then launch the injector and use it on the BL3 process. Give it a few seconds, and you should be able to hit tilde/backtick a few times to cycle through the available console modes ingame. In my experience it works just fine if you inject while ingame as well, so you probably don't have to be on the main menu specifically.

Getting Data from the Console

Once you have console access, there's unfortunately a very limited set of commands you can use which are useful for getting data. Namely, the mainstay of object dumping from BL2/TPS -- obj dump -- is not available on the BL3 console. Instead, the only command usable for data retrieval is getall. In BL2/TPS that was primarily just used to get lists of objects, which you'd then obj dump, but in BL3 it'll be your window to in-game data.

The UE4 getall command can be used just to list objects, like this command to list all item pools currently loaded in memory (this is one kind of object which JohnWickParse can serialize easily, but checking values in-game is often helpful even then):

getall itempooldata

You can get specific and provide the exact attribute you're looking for, like so:

getall itempooldata balanceditems

When you're looking for one specific object's attribute, though, you'll want to narrow the results down a bit, which isn't always possible, depending on the kind of object, but for ItemPoolData objects, it is. You can specify an extra name=foo parameter at the end, like so:

getall itempooldata balanceditems name=itempool_artifacts

That command would show you the contents of the itempool which drops Artifacts in-game. The name to use, to narrow down the getall output to something reasonable, will be found after the last slash in the object names, so you can generally get them right from the object list itself. (Though again, in some cases you'll find that all the objects that show up share a common name, so you might be stuck anyway.)

Note that when you use the name=foo construct at the end, you need to specify an attribute name as well, otherwise it won't work. If you don't actually care about the attributes and you just want the object list, you can put in any bit of text for the attribute and you'll still get the object list.

One final wrinkle that you'll have to contend with is that the BL3 console does not wrap text to the next line, if there's enough output to reach the righthand side of the screen. Anything beyond that point will just be cut off and not visible ingame. Fortunately, you can get the missing text, by using a memory viewer like Cheat Engine. You'll have to attach to the running Borderlands 3 process and then search memory for some bit of the getall output that you can see, and then look at the rest of the line in the CE memory editor. Note that the console output text looks like it's encoded internally in memory as UTF-16 strings, so you may have to change some dropdowns on whatever app you're using to search in memory.