ZG SWTOR Area Tools - SWTOR-Slicers/WikiPedia GitHub Wiki

The SWTOR Area Tools comprise an Area Assembler tool (identical to the one in the stand-alone SWTOR Area Assembler Add-on), plus a couple of Blender Collections tools that ought to help deal with the results of importing massive area sets (whole SWTOR worlds, usually).



Area Assembler.

Alt text

The Area Assembler tool (co-written with fellow slicer @crunch) assembles SWTOR's in-game areas based on the information produced by Jedipedia.net's World Viewer and File Reader online tools. Those tools are able to export lists of objects in an human-readable-ish .json format, which this one uses to import the objects and position them correctly.

In this guide we will show you:

  1. How to use Jedipedia's tools to find and download the game location data in .json format.
  2. How to feed it to the Area Assembler.
  3. And its possibilities and limitations.

Requirements.

In order to work, this tool needs:

  • To have selected in this Add-on's Preferences a 'resources' folder from an assets extraction produced with Slicers GUI's 'Full' preset (a smaller, 'Static' preset-based one ought to work, too, but that hasn't been thoroughly verified yet). The Status panel will indicate if such a folder is selected. If not set, both the indicator and the tool will appear in red.
  • The latest .gr2 Importer Add-on for Blender.

And to obtain SWTOR areas' data, we'll have to:

Importing options.

These are the tool's basic importing options. The default settings are quite alright for most cases.

  • Apply Final Rotation (default=on): disable to skip final rotation. All objects will need to be rotated in the X Axis 90 Degrees by the user. Just leave it on.

    (This is more of a diagnostics tool than anything else. Due to Blender's Z-is-up coordinate system being at odds with SWTOR's Y-is-up one, which is a source of complications when using SWTOR objects transformation data and chaining those transformations when parenting objects, we delay the final 90º rotation to the very last moment. Un-ticking this checkbox would bypass that rotation)

  • Process Named Materials (default=on): it applies the Process Named Materials tool (available in the Materials Tools subpanel) on the fly to auto-texture the imported objects. As we don't cover all SWTOR materials yet, depending on the type of area (indoors, outdoors, spaceship interiors) we can get fully textured environments or ones with lots of untextured stuff showing up as white (vegetation, water bodies, etc.).

  • Apply Scene Scale (default=off): automatically scale the entire scene after import by 10x to better match Blender Units.

    SWTOR uses decameters as coordinate units, so, objects are internally sized as their real world size divided by 10. As many advanced operations in Blender (physics simulations, automatic weighting, etc.) depend on objects having real world sizes, this option will resize the imported areas x10 for us if we need it.

    Such scalings can be done with the Objects Tools subpanel's Quickscaler tool aftewards, anyway.

  • Skip dbo Objects (default=on): Don't import design blockout (DBO) objects such as blockers, portals, etc. Replace them with Blender's "Empties", instead.

    DBO are game engine objects, invisible during gameplay, used by the level designers to set barriers to movement, mob spawning points, etc. Normally, such objects are just a nuisance for the purpose of assembling a game location in Blender, so, the default setting is on.

    However: we aren't fully sure yet about whether some of those objects determine the position, rotation and/or scale values of some visible objects in the scene (that is, they act as their parent objects) or not. Because of that, at the moment we are replacing them with Blender's Empties, which are a nuisance of their own but at least they aren't rendered in the scene. Once we determine they are safe to remove, we'll have a setting do so.

  • Create Scene Lights (default=off): automatically create very basic scene lighting based on in-game lighting nodes' positions and angles (if their numbers exceed 100, they will be set as excluded from view for Blender responsiveness reasons).

  • Collect Objects by Type (default=off): if checked, the areas' objects are further arranged in Area sub-Collections. For each area (.json file), we get this Collection hierarchy:

    • Area name
      • Area name - Lights (if Create Scene Lights is on)
      • Area name - Terrains (if any)
      • Area name - Objects
  • Merge Multi-Mesh Objects (default=off): joins single .gr2 file-originated multi-meshes into single objects.

    Some of SWTOR's .gr2 objects contain several meshes at once. For example, there is an open crate object that has a separate lid and several tools spread around it, plus some invisible blocking object to keep a player from walking across it. These are the objects that slow down the importing process the most, despite some attempts at optimizing it. Then again, those are great objects for kitbashing if we keep their sub-meshes separate.

    Still, if one doesn't care for that, this option would just merge the sub-meshes into a single object, easier to pick and move around in Blender. This might marginally speed up the importing process if there are duplicates of this object listed in the .json files.

  • Full Report In Terminal (default=off): by default, the Add-on will show the progress of each of the importing job's stages in an abbreviated manner: continuously overwriting a single line except when there is some error to report. Ticking this checkbox will show all the lines, instead.

    If set to on, given the tremendous amount of lines a whole world import can mean, we recommend increasing the console app's buffer settings (or the one in a debugging environment such as VSCode's) to some ludicrous size: we are talking about 500 lines per expected .json file or so. This option doesn't make much sense otherwise, as the full report's first half or more would easily get truncated out without an ample buffer size.

Options for reducing Blender's lag when importing MASSIVE Areas.

If we are importing whole worlds because of course we are 😏:

After importing massive amounts of objects, such as a whole planet's set of .json area files, it's typical that Blender's responsiveness to operations such as a innocent Select All lags horribly. We are talking half minutes or worse just to respond to our cursor hovering over a button (if this button has to do a check for a selection of objects before making itself clickable). Even just moving through an Outliner overloaded with thousands of objects will lag considerably.

The following options mitigate or fully eliminate that problem by hiding the objects in two manners that keep Blender from getting choked, and give us a chance to save the project and organize things a little before unhiding them.

  • Hide Objects (default=off): imported Area objects are hidden in the 3D Viewport (as when turning their 'eye' icon off in Outliner or using the h shortcut) to keep Blender more responsive when having massive amounts of objects per individual Collections.

    Recommended for dealing with single .json files weighting in the MegaByte range, such as certain juicy single location/.json file areas where the game developers put ALL the decorations to test them. That means a single resulting Collection with thousands of objects.

    Lag could persist, as the Outliner still has to list the objects, but it should be fairly tolerable.

  • Hide Collections' Contents' (default=off): imported per area (per .json file) Collections are excluded from the View Layer (as when unticking their checkboxes in the Outliner or using the e shortcut) to keep Blender fully responsive and be able to manage them without any lag at all.

    Recommended when importing a massive number of areas, such as whole worlds.

    Excluded Collections won't list their objects in the Outliner, as usual, which immediately alleviates the Outliner's workload and makes Blender brisky. The idea is to activate the specific area Collection or Collections we want to tinker with and deactivate them to move on to others instead of having it all active and crush both poor Blender and our patience.

Excluding Collections resets the hide/show state of the Collections' contents. That means that, if we enable both options, Hide Objects After Importing won't produce any effect.


How to use the Area Assembler tool.

Locating Area IDs, and Downloading Area .json files.

The first step of importing an area is knowing what area we want to import. For the sake of this tutorial we're going to be focusing on the Sith Warrior's 'Fury' player ship. The steps for this area apply to all other areas as well.

  1. First, we navigate to Jedipedia's World Viewer and locate in the Player ships category the Fury (Sith Warrior) entry. Simply clicking on it will populate the AreaID in the text field below the 1. Choose an area heading, as seen below. We copy this Area ID to the clipboard or a text document for use in the next step, and we're done with the World Viewer!

    AreaID Image Here

  2. Now, we navigate to Jedipedia's File Reader (the link is available at the top left corner of the World Viewer page, too) and click the Add Assets button, which will produce a file browser.

    We have to go to the Assets Directory inside SWTOR's game files and select all the .tor files within the folder (pressing a will do it), then open them. Loading them takes very little: from less than 10 to 60 seconds (there are progress indicators).

    Location of the 'Assets' folder:

    • On a Steam Install: C:\Program Files (86)\Steam\steamapps\common\Star Wars – The Old Republic\Assets

    • On a traditional SWTOR Launcher Install: C:\Program Files (x86)\Electronic Arts\BioWare\Star Wars - The Old Republic\Assets

  3. Once all the .tor files have loaded into the File Reader, we switch to the 'Files' tab on the top left, and then, in the input search field, we paste or type in the AreaID we got earlier. In this case, the AreaID is 4611686051284477081. Doing that will narrow the displayed files down to a handful, all relating to the spc_ps_sith_warrior area: these are the .dat files, commonly referred to as area.dats.

  4. Whilst we could go through each area.dat entry and download them individually, in most cases it's quicker and easier to just import the whole area (unless you know you'll only need a certain section, and are willing to spend the time working out which area.dat is the one you require). For this tutorial, we will be importing the entire Sith Warrior's Fury.

    Continently, the .dat file of any area is called area.dat, and will encapsulate all subareas such as ps_sith_exterior.dat, ps_sith_int_barracks.dat, ps_sith_int_captain_room.dat, etc.

  5. So, let's select area.dat from the search list and scroll down until we see a heading labeled Assets with a large button labeled Extract as zipped JSON.

    After clicking it, it might take it a little bit to actually produce the download, depending on the size of the area (should be almost instantaneous for a player ship). We save the ZIP file generated by this button to somewhere easily accessable, and unzip the included directory which contains the .json files we'll require for the final step in Blender.


Processing the area .json files.

  1. Before proceeding any further: as Blender hardly offers any means for proper progress reports, and importing areas is a rather complex task (and quite lengthy if importing whole worlds), we strongly recommend to open the system console to watch the information that this tool (and others in the same Add-on) produces as it goes through it. The option is at top menu bar > Window > Toggle System Console. It gives us progress percentages of each phase of the importing and assembling, which area and objects are being processed, notes on any failures during the process…

  2. We adjust any importing options we need (all checkboxes have tooltips) and click on the Select Area's json Files button, which will open a file browser.

    Alt text

  3. Then we navigate to where ever our area.json files have been saved, which will be in the directory we created when unzipping the area.zip file in Step 1 of this section. Here we can select as many, or as few of the .json files as we like. The importer will process whatever we tell it to. Obviously, the more files we import, and the bigger they are, the longer the process will take. We recommend that, the first time you try the tool, import a single, big .json file, to get a sense of how fast it runs on your PC.

    For this guide, we will select everything by pressing A, or manually highlight all the displayed .json files used by the Sith Warrior's Fury. With these files selected, we click the Import Area .json button, and wait.

    Alt text

  4. For an area like a player ship, on a decent PC with solid state storage, it ought to take less than ten seconds. If we opened the system console we should be able to see a flurry of activity as the tool imports and processes each .gr2 object file.

    Alt text

    If we didn't, Blender will appear to be unresponsive save for its icon-cycling cursor. That's normal. If importing something as relatively small as the Fury ship takes more than one minute, that would be definitely odd and you may want to consider starting again. However, open your Blender System Console this time to see where things might have gone wrong.

    The progress of the importing that the console shows goes through these stages:

    1. A fairly quick one where all the .json files' information is being gathered.
    2. The actual importing of the objects with some additional diagnostic data: imports, duplications, discardings… with speed ups and downs (the tool tries to avoid reimporting objects that were imported already).
    3. A parenting pass where child objects are parented to parent objects, inheriting their positions, rotations and sizes accordingly. Some failure warnings are expected here, as the area data doesn't include certain animated objects that could act as parents. See the Known Limitations and To Do sections at the end of this guide.
    4. An object renaming pass.
    5. An object texturing pass.
  5. When the Add-on has finished processing the files, you should see the fully textured model of the Fury

Congratulations, you've assembled your first area. Now go forth, recreate the game in Blender and enjoy your lag 😅!

How Automatic Lighting Works.

The area data exported in the .json files contains references to light sources' positions and rotations, but nothing else about them (so far). As that information might be useful when lighting the locations in Blender, the tool can produce generic point lights out of that data as a reference (this can be enabled via the importing options'checkboxes).

At this moment, things are set so that all the lights in a "room" (a .json file) share settings (light data block) such as type, intensity, color, etc., so, changing one of them will change all the lights in the room. They can always be individualized by using the Object > Relations > Make Single User > Object + Data > Selected Object (or All if you need that) option.

Adding Terrain.

We have preliminary support for processing heightmap-based terrain objects reference data. The Add-on imports terrain objects exported as .obj files and placed in a new folder at resources\world\heightmaps by a Windows command line tool still in development by fellow Slicer UltimaKaosXIII: SWTOR Terrain Extractor. The tool is able to process all the terrains in the game in one go in a few minutes, the results occupying some 8.3 GB.

There are some caveats, such as gaps between some terrain patches, the lack of UV information to apply textures correctly, same goes for which textures and shaders to use.

General information about Objects Types and Empties.

Similar to the case of lights, many objects in the results of an import are linked duplicates sharing mesh data blocks, as that accelerates importing a whole lot. We are looking at the possibility of making them unlinked and see how that affects importing speed and Blender's responsiveness afterwwards. Meanwhile, any linked duplicate can be turned into a fully independent object through the usual means.

To facilitate debugging, the objects have some Custom Properties keeping some data from the .json files such as object ID, parent's object ID, the name of the .json file it came from, and some positional data that Jedipedia pre-calculated to check against. Those properties can be found in the Custom Properties section of both the 3D Viewport Editor sidebar's Item tab and the Properties editor's Object Properties tab.

Known Limitations.

  • We can't do successive imports inside a same Blender project: due to coordinate system issues, we do a final 90º rotation of the scene at the end of each import, and that would tumble any previously imported objects. We are seeing about some better way to solve this.

  • A nuisance about the results of an import is that everything (95%-100%) is correctly positioned, but their origins or parentings are a mess (objects' origins can be far, far from their bases or center of mass). There's a logic behind it, but still it is inconvenient if we want to manipulate them for, say, kitbashing new environments or objects. Not that it makes it impossible, not at all, but still…

    Also, the tools rely on instancing any duplicates present in the areas' .json files to save time and memory during the import, which is, again, a bother when we want to alter the results or collect stuff for kitbashing (we are seeing about adding some kind of instance-to-standalone pass, and how it impacts the already gargantuan whole worlds imports).

  • We added support for "placeables" (holoprojectors, spaceship cockpits' seats, GTN Booths, etc), but it is incomplete, as some types, such as FXspec objects, are difficult to position in the scene adequately. We are seeing about making them present at the very least, leaving its correct placement to the user.

  • Not an Area Assembler issue per se but a Named Materials Processor tool one: SWTOR has some six main materials, which we cover, and are usually enough for interiors, spaceships, and buildings' exteriors, but we still need to implement lots of simpler things like water body shaders, several vegetation types, etc. Which means that imported exterior areas will show many things as textureless, white objects.

  • Those that expect being able to export these areas to other platforms such as VRChat, Roblox, etc., via FBX, will find that our SWTOR materials for Blender don't move well through such formats: they are just too complex and don't even depend on standardized-ish shaders such as the Principled BSDF one.

    Most area-type materials could be reduced to, say, a diffuse map and a principled shader, maybe adding the gloss map too (although it's not meant to be used in that way). The normal maps would need some preprocessing, though, as they are packed with normals, opacity, and emissiveness data channels (see https://github.com/SWTOR-Slicers/WikiPedia/wiki/SWTOR-materials-and-texture-files).

    There is a tool in the Materials Tools subpanel, the Convert To Legacy Materials one, that converts our current SWTOR materials to an older version that is Principled shader-based and also better lends itself to a baking approach. That could help, if maybe not by much. We are seeing how to produce a proper SWTOR-to-FBX-friendly materials tool, but we are just swamped with work, so, no promises.

To Do.

  • Tool-side:

    • See if any standard Python module such as NumPy can be applied somehow to accelerate processing.
    • Check the implications of making the duplicates of imported objects unlinked (each having their own mesh data) on processing speed and Blender responsiveness.
    • imported objects may have Bone Bounds properties. Their duplicates don't, at the moment. Investigate (it might have to do with armature-driven elements such as doors) and see what else duplicates might be missing and how to solve it.
    • See if we can entirely disregard the Empties replacing DBOs.
  • Named Materials Processor tool-related:

    • There are lots of little materials that we need to cover: animated holograms, vegetation, glass and water, stain cards…

Group Areas in SubCollections.

Alt text

The results of the Area Assembler tool appear as Collections whose names follow a tree or directory-like logic, with words/branches separated by underline characters.

This tool groups areas/Collections inside new Collections named after those branch names, and we can choose how many levels of nesting to do. Usually one level suffices to make things more manageable.

(NOTICE: it's probable that, instead of offering multiple levels of nesting, a future version of the tool will use the level parameter to select the position of a single separator to group from, as that seems more practical, given the kind of area names we are getting from SWTOR. That, and letting us apply to all or to a selection of Collections)

The tool isn't smart at all about it, though, so, we must give the Collections' names a look to predict the results and see if we would need two levels of grouping instead, or even more of them (some locations have all their areas starting with a "space_" word, which isn't terribly useful).

(A nuisance is the fact that Collections in a project must have distinct names. If out of the grouping operation we end up with Collections with matching names, Blender will add the usual ugly ".00x"-style suffixes)

At the moment, this tool acts on all the Collections in a project. We'l see about maybe being able to act on selections of Collections, too.

Also: as new Collections are created as visible (technically, enabled in the View Layer), this operation could make a whole gigantic world become suddenly visible and turn Blender horribly slow. To avoid that, this tool will disable all Collections (unticks their checkboxes).


Enable/Disable All Collections.

Alt text

This is something we can do in the Outliner, but in the case of a whole world import in which we accidentally let all the objects become visible, trying to do it (select all Collections in the Outliner and bring up the contextual menu to disable them) could be excruciatingly slow to achieve. This tool's buttons are far faster to use.