Guide for running Unity in CAVE2 - uic-evl/omicron-unity Wiki

Last revision: 13 January 2022

getReal3D Version: 4.0.5

Currently Supported Unity Versions:

Contents

Introduction

The purpose of this guide is to provide an overview of using the Unity3D game engine on the CAVE2(TM) Hybrid-Reality Environment. This guide will provide step-by-step instructions for building a simple Unity3D environment on a desktop/laptop development machine for deployment on a CAVE2(TM) system using the Omicron-Unity package.

In general, you do not need Omicron-Unity to deploy to CAVE2. The core of this uses the getReal3D plugin by Mechdyne described in the getReal3D Overview for CAVE2. The Omicron-Unity is designed to streamline development workflow by proving a series of tools to simulate the CAVE2 environment using stock Unity3D (no pro license or platform dependences)

Using either the getReal3D plugin directly or Omicron-Unity, the main things to consider when porting an existing Unity application to CAVE2 are:

A simple Unity movement code snippet:

void Update() {
  float forward = Input.GetAxis("Vertical") * 5;
  float strafe = Input.GetAxis("Horizontal") * 5;

  transform.Translate( forward * Vector3.forward * Time.deltaTime );
  transform.Translate( strafe * Vector3.right * Time.deltaTime );

  if( Input.GetButtonDown("Fire1") )
  {
    // Fire
  }
}

The CAVE2-Omicron Equivalent:

void Update() {
  float forward = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickUD) * 5;
  float strafe = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickLR) * 5;

  transform.Translate( forward * Vector3.forward * Time.deltaTime );
  transform.Translate( strafe * Vector3.right * Time.deltaTime );

  if( CAVE2.GetButtonDown(CAVE2.Button.Button3) )
  {
    // Fire
  }
}

Overview of CAVE2

The CAVE2 Hybrid-Reality Environment consists of 6 display nodes, one master node, and a tracking computer. Each display node is connected to twelve screens. Each display node runs a separate instance of Unity which is controlled and synchronized by the Unity instance running on the master node. CAVE2 also uses a 18 camera infrared tracking system and supports 7.1 channel audio output from Unity.

CAVE2 Coordinate System also showing tracking cameras and speakers

Tracking System and Input Devices

CAVE2 using a Vicon infrared optical tracking system to track one or more tracked objects. Each object consists of a unique arrangement of retro-reflective markers. These markers allow the tracking system to determine the position and orientation of the object within CAVE2.

CAVE2 Tracked Objects

Typically there are at least two objects that are tracked in CAVE2: The head and a wand controller. The head tracker is used to calculate the viewing perspective in CAVE2 allowing for an immersive virtual reality experience. Objects within the virtual space will be drawn at 1:1 scale (1 unit in Unity = 1 meter) and displayed based on the position of the tracked user.

The wand consists of a similar tracked pattern to allow the game to know the position and orientation of the user's hand. The wand can also has a series of buttons and analog controls for user input. Currently CAVE2 uses Xbox 360, PS3 Six-axis, or the PS3 Navigation controllers. CAVE2 uses the Omicron Input Abstraction Library to combine mocap and controller data into a single 'wand' event. Additional details on Omicron's XInputService which manages the button mappings.

CAVE2 PS3 Navigation Controller Button Mappings

A full list of input mappings is available under Basic Interaction in the getReal3D overview for CAVE2.

CAVE2 Unity Tutorial

Getting your application to display in CAVE2 is relatively easy. Adding basic navigation controls require some code modifications or use of the provided prefabs. The more complex interactions in your project, the more potential synchronization issues that may occur.

By the end of this tutorial you should be able to:

As of January 21, 2020, Unity 2017.4.10f1 and 2019.2.11f1 are supported by getReal3D (v4.0.5) and tested for use in CAVE2. Older versions of Unity are available at https://unity3d.com/get-unity/download/archive.

It is strongly recommended that you start developing on a supported version of Unity rather than downgrading to 2019.2.11. While in most cases you may only need to replace scripting function changes, a newer .scene file may not properly convert to an older version of Unity. For example scenes and prefabs in Unity 5.5 will not open at all in Unity 5.4.

The following tutorial can be downloaded from Omicron-Unity GitHub repository which contains a complete Unity project and all the steps as separate scenes in the Tutorial folder.

Configure a Unity project for use with the CAVE2 Simulator

  1. Create new project or open existing one. If you downloaded the full omicron-unity example project, skip to step 3

  2. Download a release binary or git clone of module-omicron from https://github.com/arthurnishimoto/module-omicron

    • Place the module-omicron folder into Assets
  3. Install the Unity Multiplayer HLAPI from the Package Manager if necessary

    • If you see the following error messages after importing module-omicron:
      • error CS0246: The type or namespace name 'MessageBase' could not be found
      • error CS0246: The type or namespace name 'NetworkLobbyPlayer' could not be found
      • error CS0246: The type or namespace name 'NetworkMessageDelegate' could not be found
      • error CS0103: The name 'Channels' does not exist in the current context
      1. From the Unity Editor, go to Window/Package Manager
      2. Change the view from packages 'In Project' to 'All Packages' (may take a few seconds to load package list)
      3. Scroll down and select Multiplayer HLAPI
      4. Click Install
  4. Remove some of the default Unity input key bindings (Edit/Project Settings/Input)

    • If using Unity 2018+, an Input Preset has been provided in module-omicron releases v3.7+.
      • Manual configuration
        • Under Axes/Horizontal and Axis/Vertical:
        • Set the Negative and Positive Buttons to blank as the CAVE2 simulator uses the arrow keys for the DPad
  5. Delete the main camera from the scene

  6. Add the CAVE2-Manager prefab (module-omicron/CAVE2/Prefabs/CAVE2-Manager.prefab) to the scene

  7. Create a CAVE2 Player controller as described below or use the prefab provided (module-omicron/CAVE2/Prefabs/CAVE2-PlayerController.prefab)

Create a CAVE2 Player Controller

The purpose of this exercise is to get you familiar with the various components of the CAVE2-Unity interface: How the camera works, how to map CAVE2 input controls, and how to map tracked objects to virtual objects.

If you are more interested in quickly dropping a player controller and building to CAVE2, you can skip this section and instead use the provided CAVE2-PlayerController prefab instead.

Create a CAVE2 Camera Controller

To setup a Unity application to display correctly in CAVE2, you will need to configure your camera setup:

  1. Create a new scene
  2. Add the CAVE2-Manager prefab to the scene by dragging and dropping the prefab into the Hierarchy (module-omicron/CAVE2/Prefabs)
  3. Build a test environment. Try adding the Ground Plane and PhysicsCube prefabs (module-omicron/CAVE2/Tutorial/Prefabs)
  4. Drag and drop the prefabs into the Hierarchy
  5. Create a new empty GameObject (GameObject/3D Object), set the transform position to 0, 0, 0 and rename it CameraController.
  6. Drag the Main Camera object to make it a child of the CameraController. Set the position of the camera to 0, 0, 0.
  7. Add the CAVE2CameraController.cs script (module-omicron/CAVE2/Scripts/Util) to your CameraController GameObject

  1. Keep in mind that the CameraController position now represents the CAVE2 tracking origin (Center of the floor in CAVE2). So when inside CAVE2, the local position of the camera will be an offset of the CameraController's position.
  2. Try running the scene. Notice that the Camera child object jumps to 0, 1.6, 0. This is the simulated head tracking position of the user in the center of CAVE2 with a height of 1.5 meters. This can be changed on the CAVE2Manager script on the CAVE2-Manager object under Head Emulated Position.
  1. A completed scene of this tutorial can be found under (module-omicron/CAVE2/Tutorial/CAVE2 Tutorial 1 - Camera Controller.scene)

Build a simple head-tracked player controller for CAVE2

Next we will setup a green marker to represent the head position of the player. Note that this is separate from the camera position as there is no one single camera when Unity is running in CAVE2.

  1. Create a new empty GameObject. Rename it PlayerController. Set the transform position to 0, 0, 0
  2. Set the CameraController as a child of PlayerController
  1. Create a new 3D Sphere. Rename it 'Head'. Set the scale to 0.1, 0.1, 0.1 and make it a child of PlayerController
  2. Set the local position of the Head to 0, 0.9, 0
  3. Change the Material of the Head (Head/MeshRenderer/Materials/Element 0) from Default-Material to Default-Green
  4. Add the CAVE2MocapUpdater.cs script (module-omicron/CAVE2/Scripts/Updaters) to the Head
  5. Set the sourceID to 0 (Primary head tracker)
  6. Run the Unity scene
  1. A completed scene of this tutorial can be found under (module-omicron/CAVE2/Tutorial/CAVE2 Tutorial 2 - Head Tracking.scene)

SourceIDs used to identify all tracked objects registered with CAVE2. These can be accessed in your scripts by using the following:

// From CAVE2MocapUpdater.cs ------------------------------------------
public void Update()
{
	// Mocap source IDs can correspond to any tracked object such as
	// common head/wand or custom objects
	transform.localPosition = CAVE2.GetMocapPosition(sourceID);
	transform.localRotation = CAVE2.GetMocapRotation(sourceID);
	
	// For convience, you can use GetHead(), GetWand() to get the
	// default head and wand trackers
	// Primary head tracker (Head 1 == sourceID 0)
	transform.localPosition = CAVE2.GetHeadPosition(1);
	transform.localRotation = CAVE2.GetHeadRotation(1);
	
	// Primary wand tracker (Wand 1 == sourceID 1)
	transform.localPosition = CAVE2.GetWandPosition(1);
	transform.localRotation = CAVE2.GetWandRotation(1);
	
	// Secondary head tracker (Head 2 == sourceID 2)
	transform.localPosition = CAVE2.GetHeadPosition(2);
	transform.localRotation = CAVE2.GetHeadRotation(2);
	
	// Secondary wand tracker (Wand 2 == sourceID 3)
	transform.localPosition = CAVE2.GetWandPosition(2);
	transform.localRotation = CAVE2.GetWandRotation(2);
	
	// The Head/Wand IDs are mapped to mocap sourceIDs
	// in CAVE2Manager.cs
}
// ----------------------------------------------------------------------------

Add basic movement to the player controller using the CAVE2 simulator

At this point you probably want to move in the virtual world beyond the physical size of CAVE2. As previously mentioned, you might be familiar with basic movement in Unity using the following:

void Update() {
  float forward = Input.GetAxis("Vertical") * 5;
  float strafe = Input.GetAxis("Horizontal") * 5;

  transform.Translate( forward * Vector3.forward * Time.deltaTime );
  transform.Translate( strafe * Vector3.right * Time.deltaTime );

  if( Input.GetButtonDown("Fire1") )
  {
    // Fire
  }
}

This works great on with a keyboard or a gamepad on a single machine, but to run on CAVE2, you'll need to use some special calls to make sure the entire cluster receives the movement command from the wand controller.

// From SimpleWalkController.cs (module-omicron/CAVE2/Scripts/Examples)

void Update() {
  float forward = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickUD) * 5;
  float strafe = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickLR) * 5;

  transform.Translate( forward * Vector3.forward * Time.deltaTime );
  transform.Translate( strafe * Vector3.right * Time.deltaTime );

  if( CAVE2.GetButtonDown(CAVE2.Button.Button3) )
  {
    // Fire
  }
}

If you're working with multiple CAVE2 wand controllers, your can specify the wand ID after the Axis/Button parameter:

void Update() {
	
  int wandID = 1; // (Typically 1 or 2. Default = 1)
  
  float forward = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickUD, wandID) * 5;
  float strafe = CAVE2.GetAxis(CAVE2.Axis.LeftAnalogStickLR, wandID) * 5;

  transform.Translate( forward * Vector3.forward * Time.deltaTime );
  transform.Translate( strafe * Vector3.right * Time.deltaTime );

  if( CAVE2.GetButtonDown(CAVE2.Button.Button3, wandID) )
  {
    // Fire
  }
}

Add a basic wand controller to have a virtual object appear where the wand is

If you want to have a GameObject mapped to a tracked marker like the wand, the most basic way to do that would be similar to how we added a sphere for the head:

  1. Create a new 3D Sphere. Rename it 'Wand'. Set the scale to 0.2, 0.2, 0.2 and make it a child of PlayerController
  2. Set the local position of the Wand to 0, 0.9, 0
  3. Change the Material of the Wand (Head/MeshRenderer/Materials/Element 0) from Default-Material to Default-Blue
  4. Add the CAVE2MocapUpdater.cs script (module-omicron/CAVE2/Scripts/Updaters) to the Wand
  5. Set the sourceID to 1 (Primary wand tracker)
  6. A completed scene of this tutorial and the basic movement one can be found under (module-omicron/CAVE2/Tutorial/CAVE2 Tutorial 3 - Simple Walk Controller.scene)

Testing this in CAVE2, you should now have a blue sphere floating at the position of the wand controller (more specifically the center of the markers). We'll apply an offset so we can accurately place an object; say a model of the wand in the exact position of the wand so that it looks like you're gripping the handle of the virtual object.

Sphere at wand position

The CAVE2 simulator also provides a wand prefab designed to match the size and shape of CAVE2's wand. Wand.prefab can be found in (module-omicron/CAVE2/Prefabs) and can be added as a child of the PlayerController. Recalling how the head was constructed, the Wand prefab is designed as follows:

Wand object with Mocap Updater

Navigation child with Visual Updater

The convenient part of this setup is that attaching an object as a child of Navigation which has its handle at its local origin will appear to be in the user's hand.

A scene with the wand prefab which will let you point and drag the cubes can be found under (module-omicron/CAVE2/Tutorial/CAVE2 Tutorial 3A - Simple Walk Controller (Wand prefab).scene)

CAVE2 Player Controller

We've developed our own player controller with similar functionality to the getRealPlayerController. This was done so we could more easily integrate custom interactions with our player controller as well as provide some useful tools for CAVE2 development that does not require Unity Pro or platform dependent functions until building on CAVE2.

CAVE2PlayerController

The CAVE2 Player Controller also includes the CAVE2 screen geometry, allowing users to preview what parts of the scene will be viewable on the CAVE2 system.

CAVE2ScreenSimulator

One of the child objects of the CAVE2 Player Controller is the CAVE2ScreenMask. Clicking on this in the Hierarchy and looking at the inspector will show the CAVE2 Screen Mask Renderer script. This controls how the screen mask, representing the area of the scene not visible on the CAVE2 displays. There are 3 options for rendering in the editor:

CAVE2ScreenMask CAVE2 Screen Mask (Left to Right): None, Background, Overlay

Building on CAVE2

Once you're ready to build your project on CAVE2 you have to copy your project to the CAVE2 master node.

All project files should be placed in the following location in the DATA drive (D:): D:/[Your name or Course/Team Number]/

You should add your project into a subfolder of that directory with your name. Students in a class using CAVE2 should add their projects into the directory with their course number and team/group number (i.e. 'CS 426 Spring 2019/Team1')

Spatialized Audio

By default, Unity projects output Stereo (2-channel) audio. CAVE2 is capable of spatialized audio using Unity's 7.1 surround output. To enable this in your project:

  1. From the Unity editor's main menu, select Edit/Project Settings
  2. Select Audio
  3. Click on 'Default Speaker Mode' and select '7.1 Surround'

Additional details and troubleshooting are described in CAVE2 Unity Tips and Examples.

Adding the getReal3D Plugin

The getReal3D plugin has to be installed before creating a build for CAVE2. In preparing for this you must enable the USING_GETREAL3D define flags present in the omicron-unity code. This is accomplished through the following steps:

Setting the DEFINE Symbols

  1. From the Unity editor, go to Omicron/Configure for CAVE2 Build
  1. You should now see some Errors related to 'getReal3D' not being found.

Adding the getReal3D unitypackage

  1. From the Unity editor, go to Assets/Import Package/Custom Package
  2. Navigate to D:/Unity Packages
  3. Select the latest getReal3D plugin. As of Feb. 2020: 'getReal3D_Plugin_4.0.5.1382.unitypackage'

Warning: If you already have a Standard Assets folder with Character Controllers, getReal3D contains its own CharacterMotor.js (under Standard Assets/Character Controllers/Sources/Scripts) so if you made modifications to that script, be sure you have a backup and uncheck that file from the import.

At this point those 'getReal3D not found' errors should be gone. You may have some warnings, but that is normal.

You are now ready to build!

Building

Building the executable for CAVE2 is no different than any other Desktop build.

Unity 2018 and later

Versions of Unity after 2017 changed the way Unity builds binaries, instead of specifying the application/exe name on build, the user is now prompted for a destination folder and the exe name is determined by the 'Product Name' found in Build Settings/Player Settings. This is the name that will appear in the getReal3D launcher game list so be sure to make this a unique name.

Unity Project Name

  1. From the Unity editor, go to File/Build Settings
  2. Select Player Settings
  3. Set the name of your exe/project in 'Product Name'
  4. Select Build
  5. Save your build in the following location D:/Unity Binaries/[Name/CourseNumber]

Running on CAVE2

After your executable is built, open up the getReal3D 4.x Launcher (1). This is usually always open on the desktop, but if not check the task bar of the getReal3D icon (2) or open the shortcut on the desktop or in Start by pressing the Windows key (3).

Also make sure trackD is running (4). If you find head tracking isn't working, it may not be running. In this case open the Start shortcut for trackD (5).

CAVE2 Master Desktop

There are three tabs in the getReal3D launcher: Configuration, Status, and Cluster Games List.

Status

The status window should show green checkboxes for all 6 nodes. If all of the screens are showing the Windows desktop in the CAVE and some checkboxes are red, wait a minute for the network to reinitialize. Closing and restarting the Launcher may help the status window refresh.

getReal3D Launcher

After all nodes are green you will be able to modify the Cluster Games List.

Configuration

Before running your application, make sure the correct configuration file is loaded. In general there are three different config files that could be used - all dealing with which mocap marker is used as the primary controller.

Make sure the controller you want is selected before moving on to the Cluster Games List.

getReal3D Launcher

Cluster Games List

This windows lists all the currently loaded games on the CAVE2 cluster. A green thumbs up denotes that the game is up to date and synchronized on the cluster. Red thumbs down shows that the version of the plugin on that game is either out of date or not current across the cluster.

getReal3D Launcher

To add you game to the list:

  1. Click on the blue plus at the bottom of the launcher and select your game in the explorer window.
  1. Once the game is loaded, you game should be listed with a green thumbs up and the Launch button in the lower left should be active
  2. Click launch to start your game in CAVE2!

Specific Examples

See CAVE2 Unity Tips and Examples for useful tips for working with getReal3D / Unity on the CAVE2 cluster. Covers topics like correcting particle rendering, custom menus, loading different scenes, and other special cases that have come up over the years.

Appendix A: Omicron-Unity package

OmicronManager and CAVE2Manager

The OmicronManager script handles connection with an Omicron input server, parses events and then broadcast those events to registered Omicron clients (OmicronEventClient.cs). The current Omicron-Unity package can handle the following Omicron inputs:

The Omicron Manager also works with the CAVE2Manager to help simplify event handling for head and wand inputs. The CAVE2Manager also provides some basic keyboard emulation of tracking and wand inputs for development systems.

InputManager

Both OmicronManager and CAVE2Manager are packaged into the CAVE2-InputManager prefab for easy integration into a CAVE2 Unity project. Other than toggling CAVE2 simulator mode no other setup is required when using this prefab.

Appendix B: Synchronizing Unity Components (getReal3D v2.x)

CAVE2 is essentially running 37 different instances of Unity. A major function of getReal3D is synchronizing anything in the game that could potentially behave differently between machines. This includes user inputs, objects that move, anything that uses a random function, physics collision detection, and particle systems.

getReal3D provides synchronization through two methods using the ClusterView script: Serialization and Remote Procedure Calls (RPC).

Serialization

// Standard getReal3D Code Block ----------------------------------------------
getReal3D.ClusterView clusterView;
public void Awake()
{
	// Adds a cluster view to this script
	clusterView = gameObject.AddComponent<getReal3D.ClusterView>();
	clusterView.observed = this;
}
public void OnSerializeClusterView(getReal3D.ClusterStream stream)
{
	// These variables are constantly synced across the cluster
	// and are based on input events handled in Update()
	stream.Serialize( ref myVariable);
}
// ----------------------------------------------------------------------------

Above is my generic cut-and-paste code block for adding scripting synchronization. In Awake() the getReal3D ClusterView object is added to this gameobject and observed is set to this (i.e. the script). If this object also moves in the world, you may also want to add an additional ClusterView observing this object's transform.

gameObject.AddComponent<getReal3D.ClusterView>().observed = transform;

Note: If you have multiple scripts that need to be synchronized in the same gameObject, it may be better to manually add a ClusterView to the gameObject from the Inspector.

OnSerializeClusterView() streams the state of serialized variables from the master node to the client nodes. This is the easiest and quickest way to make sure variables in your scripts are synchronized across CAVE2. The drawback of this is that these variables are updated multiple times per frame and can reduce performance. The alternative is using RPC calls to send messages from the master to the clients as needed.

The following variable types and be serialized:

RPC Calls

Note: The following code example is using Omicron input rather than getReal3D, but the syntax is similar.

void Update()
{
	if( getReal3D.Cluster.isMaster )
	{
		// Checking inputs should only be done on master node
		// The getReal3D equivalent would look like:
		// if( getReal3D.Input.GetButton("Fire")	)
		if( cave2Manager.getWand(wandID).GetButtonDown(CAVE2Manager.Button.Button3) )
		{
			// Check if getReal3D is enabled (i.e. Running on a getReal3D supported platform)
			if( Application.HasProLicense() && Application.platform == RuntimePlatform.WindowsPlayer )
				clusterView.RPC ("Fire",0);
			else
				Fire(0);
		}
	}
}

[getReal3D.RPC]
void Fire( int weaponType )
{
	...
}

The previous code block shows how you should organize your scripts for CAVE2. In Update functions and Physics callbacks (i.e. OnCollisionEnter or OnTriggerEnter), you should have two sections of code. One that only runs on the master node (contained in the isMaster if) and the other than runs on all nodes.

In that example on Update() we're checking the input of the wand on the master node. If the button 3 is pressed, we call the Fire() function one of two ways. The first one checks if Unity is currently running with a Unity Pro license and on Windows. As getReal3D is a Windows-based native library, clusterView.RPC() will only be called on a Windows platform using Unity Pro. The else allows you to still test your scripts on a non-Windows machine or the free version of Unity.

RPC allow the following types to be passed as parameters:

Appendix C: CAVE2 prior to 2019

Switching the CAVE2 Cluster to Windows

CAVE2 is typically running in Linux. Before you can run your Unity application, you will have to switch the inputs on the displays to Windows. This is done by the iPad interface on the left of the CAVE2 entrance.

  1. If not already on the Home page, click Home

  2. Click Linux_Windows at the bottom of the Home page Sabi Home

  3. Click Switch to Windows

Sabi LinuxWindows

getReal3D Launcher

CAVE2 Master Desktop

Appendix D: Unity Hub 3.0.0 and Unity 2019.2.11

Following the release of Unity Hub 3.0.0 in January 2022, project templates in some versions of Unity no longer work and will throw an error when creating a new project:

The current workaround as of Jaunary 13, 2022 is to create a project using the 2D project template and re-adjust the scene view to use 3D mode:

  1. Create a New Project from Unity Hub

  1. Change Editor Version

  2. Select '2019.2.11f1' from the dropdown menu

  3. Select '2D Template'

  4. Give a unique name to your project (Probably including course number and team/group number)

  5. Create project

  6. Click Continue if given a 'Non-Matching Editor' message

  1. Un-select '2D' from the Scene view menu

  2. From GameObject/Light add a 'Directional Light' to the scene

  3. Enable 'Scene Lighting' on the Scene view menu

  1. Change the 'Main Camera' Projection from 'Orthographic' to 'Perspective'

  1. On the Lighting tab (Window/Rendering/Light Settings) set the 'Skybox Material' to 'Default-Skybox'