Toolchain comparison - JayFoxRox/nxdk GitHub Wiki

This was created for the upstream wiki; but it needs formatting and links, spell-check, slimming, restructuring and so on. Some features are not finished yet either; this article is sort-of talking about the near future of nxdk.

What a toolchain for Xbox is

The original Xbox is basically an x86 based "legacy free PC". As operating-system, it runs a modified version of the Windows kernel.

It's important to realize that the operating-system is very slim when compared to many other consoles:

It can not do anything GPU related. It does not have drivers for that.
It can not do anything audio related. It does not have drivers for that.
It can not access the gamepad ports. It does not have drivers for that.
It can not access the network. It does not have drivers for that.
It can not provide the typical Windows APIs that applications normally use. It does not have code for that.

(The kernel actually has some drivers for GPU and audio to run the boot animation. However, once the animation finishes, these drivers are unloaded to save memory)

Instead, the operating-system primarily takes care of the video encoder, IDE driver (HDD / DVD) and associated filesystems. It also takes care of thread management and other CPU related tasks. The kernel also has the necessary code to load a new application XBE. The kernel functions that are exposed to applications, are not what you'd typically expect as an application developer on Windows. Instead the kernel only provides lower-level primitives that are typically exposed for interfacing with drivers.

However, this design implies another fact about the Xbox: Applications (like games) are like "Windows drivers" running in kernel mode. This design also means that the application is about as powerful as the kernel itself. This is interesting to keep in mind for the capabilities that an application can have on the original Xbox.

Each application XBE must contain the driver code for all the hardware that it will access, except for the drivers provided by the kernel (like filesystems). Therefore, the application XBE must also make assumptions about the hardware that is present in the original Xbox. For the functionality provided by the kernel, the XBE will typically contain abstraction libraries.

Therefore an Xbox toolchain typically contains these things:

Compilers for programming languages (typically C / C++).
Debugger.
APIs to expose the kernel interface.
Drivers for GPU, audio, gamepad port, networking, ..
Tools for compiling or assembling code for the GPU shaders, APU DSPs, ..
Abstraction on top of the drivers to make it easier to develop software.
Optionally sample code and documentation.

Overview of official Microsoft XDK

The original toolchain is called XDK (Xbox Development Kit), and was used by officially licensed games. The Microsoft XDK was developed by Microsoft and was only made available to licensed developers. Various versions of the Microsoft XDK were illegaly leaked on the internet. However, if you are not a registered developer, then it's almost certainly illegal to obtain or use it.

From a developers perspective using the official Microsoft XDK, all of the driver details (described above) are hidden. Instead, the XDK includes closed-source driver libraries that provide standardized Microsoft APIs (like Direct3D or the Windows API). When compiling and linking your application which uses those APIs, those drivers or kernel abstractions will be linked (/ copied) into the application XBE. The fact that the games contain the drivers (copyrighted by Microsoft) causes further legal issues.

Microsoft made the XDK as an extension for Visual Studio versions from around that time (2001 - 2006). It also features a good debugger and some useful development tools to connect to an Xbox using network or serial-port.

Issues with the official XDK

Microsoft seems to have prevented illegal sharing of XDKs and XBEs compiled with the XDK. They currently seem to tolerate this illegal sharing of such files, but this could change again in the future. Microsoft continues to work with the original Xbox brand and sells game. Legally they'd have many options.

Famous projects like XBMC (Xbox Media Center) worked around these legal issues by only distributing their source-code. This shifted the illegal step of obtaining the XDK toolchain to the user-side.

However, aside from legal issues, this design also leads to various limitations with the XDK:

The drivers are libraries that are designed for use with Microsoft tools from 2001 - 2006. This forces you to use old tools which are sometimes incompatible with modern versions of Windows. Linux and macOS support (or other platforms) is not provided at all.
These old tools don't contain many upgrades to programming languages. You can use different tools in combination with the official XDK to make this possible, but it's difficult and tedious.
The drivers also don't fully expose all hardware features of the Xbox. As we don't have the source-code, we can't easily improve this. We'd first have to figure out how the hardware works, and then make our own drivers from scratch.
The drivers expose one API, and adding another API is very difficult. This means that your application has to contain code that could normally be part of the toolchain.
If your XDK application is loaded on the original Xbox, it will do various things before jumping into the developers code. The drivers source-code was not made available by Microsoft. So it's very hard to prevent these unwanted steps without many hacks. These steps potentially consume memory, or do things that can not be undone later.

For Xbox emulators in particular, it's also problematic that nobody had to make their own drivers (because they came ready-to-use with the XDK). This means that hardly anyone understands how the Xbox hardware works (because the XDK drivers are essentially black-boxes). This is probably also why the open-source alternatives OpenXDK and nxdk have both originated from Xbox emulation projects.

Overview of OpenXDK

OpenXDK was an open-source toolchain, that was originally created by Caustik, then developed by different people. The core feature of OpenXDK was Cxbe, which converts a Windows EXE file to an original Xbox XBE file. This is the counterpart to Caustiks other project, Cxbx which converts an Xbox XBE file into a Windows EXE file (a form of Xbox emulation).

The use of Cxbe enabled OpenXDK to act like development software for Windows (namely using GCC / mingw) with OpenXDK only providing Xbox specific portions.

OpenXDK contained open-source driver libraries which also documented at least some of the hardware. However, many of the OpenXDK drivers had fundamental bugs, or were distributed separately from the toolchain. Therefore applications written in OpenXDK were often buggy or slow and some features were missing entirely. This limited usability of OpenXDK. So, aside from the widespread illegal use of XDK around that time, it is probably therefore, that very few software was written for compilation with OpenXDK.

In contrast to the Microsoft XDK, the OpenXDK also didn't feature any useful tools, and had no debugger.

Source-code written for compilation with the Microsoft XDK can not be compiled with OpenXDK. This was due to different abstractions, or lack of abstraction ontop of drivers. OpenXDK was also focusing on custom APIs over Windows APIs. Having full source-code compatibility between the Microsoft XDK and OpenXDK was a non-goal.

Unfortunately there were also other issues with OpenXDK concepts, which hindered the legal distribution of applications. OpenXDK used the GPL license for some drivers, and AFL for others. Some people consider these licenses incompatible, so using such drivers in the same program would not be allowed. The GPL also demands that the developer of the application must also make their programs source-code open-source.

However, many of the ideas in OpenXDK were still good. It used open-source drivers and did not force the user to run any code before their own.

OpenXDK has stopped development a long time ago, with isolated efforts to ressurect it.

Overview of nxdk

nxdk is the most recent open-source toolchain.

nxdk was originally created by espes, that is now maintained by the XboxDev organization. nxdk was initially used in a CTF-challenge to get people to use the XQEMU Xbox emulator, another project started by espes. However, nxdk was quickly used to create legal unit-tests and other development tools for Xbox hardware and Xbox emulation.

Due to its heritage, nxdk was fairly limited initially. nxdk also reused many drivers that were made for OpenXDK. However, nxdk was also missing some portions that were previously included with OpenXDK (like libc). For this reason, nxdk is considered a new project; not a contiunation of OpenXDK.

By now, nxdk has fixed many bugs in the OpenXDK drivers and abstracts them using more standardized APIs. New drivers have been added, and some portions of the Windows API have been added. Portions that were missing since OpenXDK (like libc) have been replaced.

A drawback of nxdk's rapid development is that some of the APIs might not be stable yet.

nxdk also continues to use tools like the OpenXDK Cxbe utility to convert Windows EXE to an original Xbox XBE. This enables nxdk to act like development software for Windows (namely using LLVM) with nxdk only providing Xbox specific portions.

Unfortunately the license issues from OpenXDK were inherited by nxdk. However, there are efforts to move nxdk code towards CC0, MIT or BSD licenses.

Source-code written for compilation with OpenXDK can not be compiled with nxdk. The build-system differs and the structure of the toolchain changed. Since nxdk is still under active development, it continues diverge from OpenXDK APIs and drivers. Having full source-code compatibility between the OpenXDK and nxdk is a non-goal.

Source-code written for compilation with the Microsoft XDK can not be compiled with nxdk. This is due to different abstractions and APIs. While nxdk provides Windows APIs, some APIs like Direct3D or DirectSound might never be supported. Xbox specific APIs from the XDK have been replaced with more cross-platform APIs like SDL2. Having full source-code compatibility between the Microsoft XDK and nxdk is a non-goal.

The only goal is to provide a feature-complete toolchain, regardless of the used APIs.

nxdk also still doesn't have a debugger. There are plans to add gdb support. Until then, the easiest way is to debug applications by using the gdb stub provided by the XQEMU Xbox emulator.

However, some tools, comparable to some Microsoft XDK tools, have been created. These tools are not only open-source, but typically more low-level. Some of these tools currently require a debug/development Xbox which came with the Microsoft XDK. In the future such software will also work with retail Xboxes.

XboxDev members have also created open-source hardware to add a serial-port (found on some debug/development Xboxes) to a retail Xbox. These are first steps to turn retail Xbox into capable development / debugging hosts with free and legal soft- and hardware.

XboxDev and nxdk are dedicated to make development easier.

Potential of nxdk

We realize that nxdk is not perfect. We realize that nxdk is not for everyone (neither should it be).

However, you should realize that nxdk has potential.

Not only has nxdk a potential for legal homebrew software, but for better homebrew software.

Here are some unfinished proof-of-concepts that would be much harder to do with the Microsoft XDK:

xbe-loader: This replaces the physical DVD drive with an emulated DVD drive, which streams Xbox Disc contents (ISO file) via network (via HTTP). As there are no file-size limits or storage limitations on the NAS, you could keep all your Xbox games on the NAS. You would no longer have to upgrade your Xbox hard disk with old IDE HDDs which get harder to obtain in the future. You could also access your stored games from different Xboxes, without having to install games. For game-modding, you could store the game on a PC and modify files without having to transfer them to your Xbox harddisk.
xbox-fps-overlay: This uses a rarely used Xbox GPU feature to create an overlay while a game is running. This could be used to have ingame menus for tunneling software like XLink Kai, dashboards, trainers, ..
ogx360 (software variant): This takes control of the gamepad ports on the original Xbox and adds a USB driver for Xbox 360 controllers. Then a virtual original Xbox controller is emulated for the running application. Mouse and keyboard support could be added easily in the future. With more work on drivers it could potentially support wiimotes to emulate a virtual Xbox Lightgun controller. In the future it could also potentially support PC steering wheels and re-enable full force-feedback for games like OutRun 2. This could also be combined with the xbox-fps-overlay concept to show an ingame overlay of a virtual controller, to display the status of a more complex controller like the Steel Battalion controller.
OpenSWE1R: An example for loading older Windows games (like Star Wars Episode 1 Racer) which never had a source-code release. The full control over the memory map in nxdk makes it easier to load these games into memory. The drivers can be adjusted to reduce the memory footprint of the game.

Aside from these applications, there are bits of hardware that were never or rarely unused in commercial games. These are only some examples where XDK prevents or discourages effective use:

There's a DSP processor in the APU which the XDK reserved for encoding of Dolby Digital 5.1 audio. It could potentially be used for computational tasks. A similar, but slower, DSP used to be used as GPU in early-3D arcade games.
There's a another DSP processor in the APU which the XDK reserved for audio effects; it had limited documentation and ran a premade framework. It's the same kind of DSP as the other DSP, and can also be repurposed.
The GPU has many features which were not exposed; nxdk provides raw access. Documentation for these features exist by now.
The GPU hardware design is very OpenGL oriented, D3D drivers actually cause CPU overhead and a less ideal design.
The gamepads and memory-units are standard USB devices. While the Xbox is only USB 1.1, many devices not meant for original Xbox can still work on these ports. Driver development and support for more devices could be simplified with nxdk.
The network driver is only supporting IPv4 and System-Link traffic depends on communication on Layer 2 (incompatible with some wireless network). nxdk supports IPv4, but also IPv6; IPv4 based protocols and existing auto-address-negotiation can be used to create something comparable to System-Link that is fully compatible with wireless networks.

By using nxdk you can experiment with these advanced features. Even if you still want to use the Microsoft XDK, you can benefit from knowledge gained by development of nxdk.

And even if you still want to use the Microsoft XDK, you should realize that certain tasks might be more suitable to be done in nxdk. You should realize that nxdk is also helpful for development of Xbox emulators. Such projects will make it easier to test and develop applications made using any toolchain.

Even if nxdk is not your choice of toolchain, you should still respect the people working on it. You might be benefitting from this work, even if you don't realize it (yet).

Which toolchain to pick for new Xbox projects

If your software must use large portions of Microsoft APIs like D3D then the Microsoft XDK is probably your only choice. Porting to use other APIs (such as graphics APIs provided by nxdk) might be a lot of work.

If you are writing new software from scratch, or porting a project that only uses a subset of APIs like D3D or OpenGL, then we'd recommend to use nxdk. nxdk is rapidly growing and improving, so stability will improve. Most issues will be resolved over time; the drivers are continously improving, too. Most of the APIs used in nxdk are also very well documented (SDL2, BSD sockets, Windows API, ..). The remaining Xbox specific APIs (Graphics / Audio) are being improved and standardized. You will also benefit from future developments in nxdk. You can also avoid legal issues with the Microsoft XDK.

The nxdk community is also very active and can provide a lot of assistence. You can usually get help within minutes.

Which toolchain to pick for existing Xbox projects

If your source-code is already working fine with the Microsoft XDK then we recommend to stick to that. It would only be a political / legal decision to move to nxdk; it would be appreciated, but it might be a lot of work.

However, if your software was written for OpenXDK, then we'd recommend to port it to nxdk. This will ensure that you can benefit from improvements in the future. The amount of required work is probably minimal.

Feature comparison

We will ignore OpenXDK because it was very incomplete.

Supported programming Languages

The Microsoft XDK provides support for C++ in the Visual C++ dialect, up to C++03.

nxdk provides support for C, up to the latest version supported by clang and pdclib. nxdk provides support for C++, up to the latest version supported by clang. pdclib is currently aiming for full C99 and C11 support, but is not complete yet. The latest version of clang usually contains experimental features from the future. Both of these languages support many dialects (such as GNU or Microsoft extensions).

nxdk uses the LLVM toolchain as foundation, so many advanced features or other languages could theoretically be implemented.

An experimental upython port that is compilable using nxdk exists. There have also been experiments with tools like IL2C for .NET (C# for example) support in nxdk. However, none of that is planned for official integration into nxdk at this point.

Operating System access

Microsoft XDK provides a subset of the Windows userspace API and extended it. The Microsoft XDK also provides functions for managing savegames or audio playlists.

nxdk also recreates some of the Windows userspace API and extends it. The implementation is complete enough for many applications. Additionally, nxdk exposes some commonly used POSIX APIs.

nxdk does not currently provide APIs for managing savegames. However, savegames are merely organized in folders with INI files. It is very easy to create a savegame that is recognized by the Microsoft Dashboard. Many libraries to create INI files exist; the format of the files is simple, and partially documented, too. Savegame thumbnails / images aren't currently supported, but creating the necessary converters is easy if necessary.

nxdk does not currently provide APIs for audio playlists.

nxdk also fully exposes the Xbox kernel interface for low-level applications.

Graphics

There is no software rendering with the Microsoft XDK.

The Microsoft XDK provides an Xbox specific variant of Direct3D8. Direct3D is normally part of DirectX. The API is rather high-level, but has extensions for low-level access to some features. The API is also state-based which means the driver sometimes has to synchronize the state with the hardware.

The Microsoft XDK provides a special variant of low-level vertex and pixel shaders. These don't map very well to hardware features. Later Microsoft XDKs also supported HLSL, a high-level shader variant. This maps even worse to the hardware.

To set up the video mode, nxdk currently uses XVideo from OpenXDK. XVideo is a wrapper around the kernel video-encoder API. It is error prone and will probably be replaced in the future.

For software rendering, nxdk has SDL2 (although no hardware acceleration is provided). Most homebrew is probably 2D, and you can use SDL2. It is probably faster and lower latency than SDL variants using the Microsoft XDK.

For hardware rendering, nxdk includes pbkit. pbkit is a very low-level driver for the GPU. It is hard to use and error prone. However, it's a good foundation for building a better driver. pbkit can still be used to make impressive 3D applications. People who are familiar with graphics development will be able to do impressive things.

In the future, nxdk will probably have a higher-level rendering library. XGU / XGUX is a concept for a rendering API / driver like that. Such an API would be similar to OpenGL 1.x + nvidia extensions from around that time. However, it would be more direct in that there is less error-checking, and no object management. These APIs would be similar to comparable toolchains for PS2 or PSP, including the official toolchain by Sony.

For low-level shaders, nxdk uses nvidia shader formats which were specifically designed for the Xbox GPU family. Name these are vertex-programs, register-combiners and texture-shaders. There is some documentation available by nvidia. Even some new features have been added to these shader languages to expose rarely used hardware features. The tooling is similar to comparable toolchains for Nintendo 3DS.

For high-level shaders nxdk supports nvidia Cg shaders which can be compiled to low-level shaders using proprietary tools. However, nvidia Cg Toolkit has been unsupported since 2012, so these tools potentially stop working in the future. The low-level tools are easy-to-use for any experienced graphics developer though.

Neither Direct3D or OpenGL support are planned for nxdk. It will probably remain an Xbox specific API.

For portability, we recommend application developers to implement subsets of the other APIs using the lower-level nxdk APIs.

Audio

The Microsoft XDK provides an Xbox specific variant of DirectSound and XAudio. DirectSound is normally part of DirectX. DirectSound on Xbox is very suitable for the hardware, and extensions and tools have been added to control the hardware audio mixer (APU).

nxdk currently has good 2D audio support; the recommended API is SDL Audio. For most homebrew applications the support is "good enough". 3D audio or effects can be mixed in software using OpenAL-soft (provided with nxdk); however, this is very CPU intense.

There is currently no driver for the APU hardware audio mixer to offload work from the CPU. In the future there might be an OpenAL driver with various Xbox extensions, to expose the APU. Unfortunately, this means that all audio computations will be done in software.

There is currently no Dolby 5.1 support, so all nxdk audio output is currently stereo.

Input

The Microsoft XDK provides an Xbox specific API called XInput (not comparable to XInput on other Microsoft platforms). Device drivers are implemented using Xbox specific callbacks; devices follow a rigid device-type design. The driver supports Memory Units, Xbox Live Communicator, and Xbox gamepads.

nxdk uses the USB stack from the Linux kernel, which had already been used in OpenXDK. Some device drivers have been ported / created; namely keyboards, mice and Xbox gamepads. However, the Linux code in nxdk is very old and there are known issues. In the future this will likely be addressed, and something like libusb will probably be provided.

We recommend to use SDL GameController as abstraction over the driver.

People who need more can implement a full driver for their device, on top of the Linux USB stack. We have successfully made a driver for Xbox 360 gamepads in the past.

Network

We will not consider Xbox Live here, because the service has been unsupported for a long time.

The Microsoft XDK provides WinSock which is a variation of BSD sockets. The traffic is typically encrypted, but it's limited to IPSec (using Layer 2) or IPv4 traffic.

There isn't much known about System-Link APIs, but it defaults to reliable UDP traffic.

nxdk uses lwip. lwip provides BSD sockets and other APIs. nxdk also features a subset of WinSock. lwip also comes with DHCP support. There is also support for IPv6.

For System-Link style applications, lwip AUTOIP (RFC 3927 / link-local addresses) can be used. For reliable UDP traffic you can use enet (provided with nxdk).

Build system

The Microsoft XDK uses Visual Studio and Visual C++. This restricts the XDK to specific versions of Windows and certain workflows. However, integration is generally very good.

nxdk does not prefer any IDE. This also means there's no deep integration. We expect the community around certain IDEs to create plugins. The standard build-system is based on GNU makefiles. It is very limited but it's "good enough" for most applications. nxdk can also be used with CMake. CMake allows more powerful features and better integration with IDEs.

Tools

FIXME: PIX vs nv2a-trace FIXME: xboxpy / nxdk-rdt vs XBDM.dll