Rokoko Studio Live Plugin for Unreal Engine

You can find README.MD for the version you are interested, in the code

Rokoko Studio is a powerful and intuitive software for recording, visualizing and exporting motion capture.

This plugin let's you stream animation data from Rokoko Studio into Unreal Engine to preview and work with all your motion capture data easily and intuitively.

Plugin version

Requirements

Unreal Engine 4.23-5.x (please see notes below)
Rokoko Studio or Rokoko Studio Legacy (starting from 1.19.0)

Note about plugin versions

4.23-4.25 are Windows-tested only. (4.23 and 4.24 won't be supported after 1.11.1)
4.26-4.27 are both Windows and Mac-tested.
5.x is also only Windows-tested.

Features

Live stream data:
- Up to five actors that can all include both body, face (52 blendshapes) and finger data at the same time.
- Character data
- Camera data
- Props data
Control Rokoko Studio from within Unreal Engine

Unreal Engine Demo project including a character already set up for live streaming from Rokoko Studio: https://github.com/Rokoko/Unreal-Demo-Project

Getting Started for Streaming

To learn more about how to use Virtual Production in Rokoko Studio, read our guide here. Or contact our Support!

Setting up Studio Live for Unreal Engine streaming

Make sure the model is ready for Studio Live

Before importing your model into Unreal Engine. Make sure that the character's hands and fingers are modeled as close to the Newton model as possible, to get the best possible retargeting of finger animation. The fingers should be straight and the thumb pointing 45 degrees away from the other fingers.

Studio Live supports up to a 4 finger joint setup: Metacarpal, Proximal, Medial and Distal finger joint.

Installing the plugin

Uninstall ANY previous Rokoko plugin that you may have installed manually. (It should be in C:\Program Files\Epic Games\UE_4.27\Engine\Plugins as “Smartsuit” folder, delete the file)
Install to your current engine version(4.26-4.27) the Rokoko Plugin from Marketplace

Launch a project. Go to Edit>Plugins and search for “Rokoko Studio Live'' and in that section make sure the “Enabled” checkbox is checked. Then, restart your project.

Setting up the character (UE 4.26-4.27)

Import your character. A pop-up window will appear. Expand the “Mesh” tab and make sure “Use T0 as Ref Pose” and “Import Morph Targets” checkboxes are checked.

(In this documentation, we imported our Mime character. Download it here. )

Right click on your Skeletal Mesh and click Create>Anim Blueprint. Rename it to something like “character_AnimBP”
Open the “character_AnimBP” and in the anim graph create a “Rokoko Body Pose” and a “Rokoko Face Pose” node. Connect them as shown below. (The “Component to Local” node will appear automatically) Note: You can skip any step regarding the “Rokoko Face Pose”, if you want to livestream just the body/hands.

Create a variable, set it as “Name” and rename it something like “Rokoko Actor Name”.

Drag the “Rokoko Actor Name” variable into the graph, select “get Rokoko Actor name” and connect it to “Rokoko Body Pose” and “Rokoko Face Pose” inputs. Then, compile and save.

In the content browser, right-click, create a Blueprint class, expand the “All Classes” tab and create a “RokokoBodyMapData” and “RokokoFaceMapData” blueprint. Name them something like “character_boneMap” and “character_faceMap”.

Open the “character_boneMap” blueprint and write the names of the corresponding joints in the hierarchy.

Compile and save the blueprint.(If you use the same joint naming on more characters you can reuse this asset). 8. Open the “character_faceMap” blueprint and write the names of the blendshape names you'd like to override(first, try not making any changes). (If you use the same blendshape names on more characters you can reuse this asset). When done, compile and save. 9. Back in the “character_AnimBP” graph, create a variable, set it at “Rokoko Body Map Data” as class reference and name it something like “BodyMapRetargetAsset”.

Compile and set its default value to the bone map you just created.

Drag the “BodyMapRetargetAsset” variable to the graph, select “get BodyMapRetarget Asset” and connect it to the “Retarget Asset” input of “Rokoko Body Pose” node.

Create one more variable, set it at “Rokoko Face Map Data” as class reference and name it something like “FaceMapRetargetAsset”.

Compile and set its default value to the face map you just created.

Drag the “FaceMapRetargetAsset” variable to the graph, select “get FaceMapRetarget Asset” and connect it to the “Retarget Asset” input of “Rokoko Face Pose” node.
Finally, your animation graph will look like this:

Note: Ignore the warnings!! 14. Right-click on the content browser and create an “Actor” blueprint class. Name it something like “character_BP”. Open it and on-top left, click on Add Component and select skeletal mesh.

To this skeletal mesh, apply your character’s skeleton and the Animation Blueprint you just created.

Hit compile and save. 16. On the event graph of the “character_BP” drag the skeletal mesh to the graph. From its pin, drag and create a “Get Anim Instance” node. From its pin,drag and create a “Cast to character_AnimBP” node. From “Cast to character_AnimBP” blue pin drag and create a “Set Rokoko Actor Name”. Set the name as it is on your Actor Profile in Rokoko Studio. Connect the rest pins, Compile and Save the Blueprint.

Drag the Actor Blueprint “character_BP” into the scene.
Under Place Actors(top-left of Unreal Editor) search for “Rokoko Receiver” and drag it into the scene. Set the Rokoko Port Number as 14045, in the Details Panel.

Setting up Rokoko studio for Studio Live

Open Rokoko Studio, open a scene and make sure your Smartsuit is connected and paired with your Profile. Click on Start Livestream, navigate to Unreal Engine and click on the cogwheel. Make sure the port number is 14045 and enable the Livestream.