Home - JohnMcCaffery/NuiLib GitHub Wiki

NuiLib is a utility library intended to ease integration of NUI (Natural User Input) Devices (such as the Microsoft Kinect) into applications. It provides an abstraction layer which hides the device being input from and provides easy support for common operations. Applications built using NUI lib benefit from the ability to make the logic of how the device is used clear and easy to understand. They also gain the ability to switch to different driver sets or even different devices with no code changes.

NuiLib is available for download or contribution on github.

Target Users The library is focused on three users groups. Firstly, developers creating / modifying applications to provide NUI support. Secondly, developers interested in optimising NUI device use. Thirdly, developers experimenting with new algorithms for processing NUI data. Each group can work on or with the library in different ways and their innovations automatically feed back in to the other groups. Any optimisations done on the core library will speed up any application or algorithm based around the library. Any new algorithm developed through the library becomes another tool in the application developer's arsenal. Lastly any feedback given by application developers can focus the efforts of the other two groups.

Ease of Use Making it easy to integrate NUI input is a primary concern for NuiLib. This is what should make it attractive to application developers. The following code snippet is an example of the simplicity of accessing NUI input through NuiLib. It demonstrates initialising the device, gaining access to the location information for two skeleton joints and then computing the vector between the two joints. Last it outputs this vector whenever the value changes. #include <NuiLib-API.h>
using namespace NuiLib;
...
NuiFactory()->Init();
Vector arm = joint(HAND_RIGHT) - joint(SHOULDER_RIGHT);
arm.AddListener([&arm](IObservable *s) {
    cout << "Right Arm: " << arm.X() << ',' << arm.Y() << ',' << arm.Z() << '\n';
});
NuiFactory()->SetAutoPoll(true);
...
The full source code for this example is available here.

Extensibility As well as ease of use the library focuses on extensibility. This makes it an attractive platform for experimenting with new algorithms. Developers can easily see the results of their work and make them available to other users. The library includes an extension mechanism so any physical device capable of providing cartesian coordinates for skeleton joints could be integrated. This support can be added to the main trunk or built as a seperate linkable unit. The core functionality of the system can also be extended. New algorithms can be written and linked into the system.

To aid in developing computer vision based algorithms NuiLib is built on top of OpenCV. This means depth and colour frames are available as OpenCV matrices for easy processing.

Structure The library is accessed through three concepts. Components, Component Factory Functions and the NuiFactory.

Components actually do the work, the three current component types are Vectors, Scalars and Conditions. If you access a joint in a skeleton it will be as a Vector. The mathematical and logical operators are overloaded to work with Components. The main feature of components is that they chain and automatically updated. Once the relationship between them is initialised then whenever the components at the top of the chain updated the update propogates all the way down the chain. In the basic example given above the arm Vector is chained to two joint Vectors. Whenever they update it will update. This is the same all the way through, even for much more complicated examples. This means that configuration can be done during initialisation. Once the relationships are established a main program loop can just read off the values at every iteration and input them directly to where they are needed. The Component List lists all components currently implemented.

Component Factory Functions create the components. The joint function is an example. By calling joint and specifying which joint the user gets a Vector which will automatically track the position of a joint. Using joint factories means that code can be written as regular, arithmetic, code. For instance taking the dot product of two Vectors is as simple as Scalar dot = dot(vector1, vector2). The Component List shows the names and parameters for all the Component Factory Functions.

The NuiFactory is what allows direct interaction with the device. It can be used to control general behaviour such as initalising and polling the device. It can also be used to load and save XML files. This way a configuration can be created in one application, saved as XML and then loaded into another. Alternatively multiple different configurations can be created and swapped in as necessary.

Use Case NuiLib has been used to develop Armadillo, a Second Life client that supports NUI control over movement. Other examples of use are given in the demos section.